Fix deadlock when piping to shell associated file extension #19940

SeeminglyScience · 2023-07-12T03:52:24Z

Fixes #19876

PR Summary

Marking the process as initialized was done inside a try that can be part of normal operation if shell32!FindExecutableW finds a valid console subsystem executable. This change moves it after the try so that scenario will be caught as well.

~~Also reduced some allocations in IsExecutable. That change is separated into it's own commit for easier review, but I'm also fine with reverting that commit if desired.~~

I don't think there's an non-interactive way to set a file extension these days so I'm not sure how this could be tested. If you know of a way (that doesn't use undocumented APIs) please comment.

PR Context

PR Checklist

If the initial process creation failed and `shell32!FindExecutableW` was used, the process initialization flag was getting skipped. Moving the handle release until after the try/catch fixed a hang.

iSazonov · 2023-07-12T04:07:39Z

I don't think there's an non-interactive way to set a file extension these days so I'm not sure how this could be tested. If you know of a way (that doesn't use undocumented APIs) please comment.

Search cmd /c assoc in our tests for examples.

src/System.Management.Automation/engine/NativeCommandProcessor.cs

This reverts commit 4052701.

Must have some automatic formatting set that I'm not aware of. Didn't mean for these changes to be added.

SeeminglyScience · 2023-07-12T14:08:48Z

I don't think there's an non-interactive way to set a file extension these days so I'm not sure how this could be tested. If you know of a way (that doesn't use undocumented APIs) please comment.

Search cmd /c assoc in our tests for examples.

Hmm does that work outside of CI? I wasn't able to get assoc to do anything meaningful on Windows 11. Maybe that still works in nano/Windows Server.

Thanks I'll look into it

Also remove redundant lock

pull-request-quantifier-deprecated · 2023-07-12T17:43:54Z

This PR has 9 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details

Label      : Extra Small
Size       : +3 -6
Percentile : 3.6%

Total files changed: 1

Change summary by file extension:
.cs : +3 -6

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

Fast and predictable releases to production:
- Optimal size changes are more likely to be reviewed faster with fewer
  iterations.
- Similarity in low PR complexity drives similar review times.
Review quality is likely higher as complexity is lower:
- Bugs are more likely to be detected.
- Code inconsistencies are more likely to be detected.
Knowledge sharing is improved within the participants:
- Small portions can be assimilated better.
Better engineering practices are exercised:
- Solving big problems by dividing them in well contained, smaller problems.
- Exercising separation of concerns within the code changes.

What can I do to optimize my changes

Use the PullRequestQuantifier to quantify your PR accurately
- Create a context profile for your repo using the context generator
- Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
- Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
- Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
Change your engineering behaviors
- For PRs that fall outside of the desired spectrum, review the details and check if:
  - Your PR could be split in smaller, self-contained PRs instead
  - Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

One line was added: +1 -0
One line was deleted: +0 -1
One line was modified: +1 -1 (git diff doesn't know about modified, it will
interpret that line like one addition plus one deletion)
Change percentiles: Change characteristics (addition, deletion, modification)
of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

SeeminglyScience · 2023-07-12T18:04:16Z

I don't think there's an non-interactive way to set a file extension these days so I'm not sure how this could be tested. If you know of a way (that doesn't use undocumented APIs) please comment.

Search cmd /c assoc in our tests for examples.

Hmm does that work outside of CI? I wasn't able to get assoc to do anything meaningful on Windows 11. Maybe that still works in nano/Windows Server.

Thanks I'll look into it

So even if assoc works in CI, it doesn't actually test what I'm fixing here. The deadlock only occurred once kernel32!CreateProcess has already failed and we then call shell32!FindExecutable to get the shell association. But if assoc works, then kernel32!CreateProcess will work, and the deadlock wouldn't get hit anyway.

SeeminglyScience · 2023-07-12T18:18:38Z

src/System.Management.Automation/engine/NativeCommandProcessor.cs

+                    {
+                        _processInitialized ??= new SemaphoreSlim(0, 1);
+                        _processInitialized.Release();
+                    }


btw I removed the lock because we're already in a lock for that object and I just didn't notice (line 596)

iSazonov · 2023-07-13T03:24:43Z

I don't think there's an non-interactive way to set a file extension these days so I'm not sure how this could be tested. If you know of a way (that doesn't use undocumented APIs) please comment.

Search cmd /c assoc in our tests for examples.

Hmm does that work outside of CI? I wasn't able to get assoc to do anything meaningful on Windows 11. Maybe that still works in nano/Windows Server.
Thanks I'll look into it

So even if assoc works in CI, it doesn't actually test what I'm fixing here. The deadlock only occurred once kernel32!CreateProcess has already failed and we then call shell32!FindExecutable to get the shell association. But if assoc works, then kernel32!CreateProcess will work, and the deadlock wouldn't get hit anyway.

Interesting.... if it is possible to reproduce the issue with Python can we emulate the same behavior with cmd.exe and .bat and use this in tests?

SeeminglyScience · 2023-07-13T16:35:09Z

Interesting.... if it is possible to reproduce the issue with Python can we emulate the same behavior with cmd.exe and .bat and use this in tests?

.bat just straight up works as if it were an executable file, handling for that is hard coded. Even if it wasn't though, the shell (explorer) already has an association for .bat so in theory that would work anyway.

The main issue is that in recent history it's been made explicitly unsupported to set shell associations outside of the UI. I've heard of some ways, but it involves reverse engineered formats that change between Windows versions? Something like that, but either way not something we'd want to use in a test.

daxian-dbw

LGTM

rmccampbell · 2023-08-12T21:59:02Z

Hi, do you know if there is going to be a preview and/or release build with this fix any time soon?

SeeminglyScience added 2 commits July 11, 2023 16:29

Reduce some allocations in IsExecutable

4052701

Move initialized flag set until after creation

159b389

If the initial process creation failed and `shell32!FindExecutableW` was used, the process initialization flag was getting skipped. Moving the handle release until after the try/catch fixed a hang.

pull-request-quantifier-deprecated bot added the Extra Small label Jul 12, 2023

SeeminglyScience requested a review from daxian-dbw July 12, 2023 03:52

iSazonov reviewed Jul 12, 2023

View reviewed changes

SeeminglyScience added 2 commits July 12, 2023 09:56

Revert "Reduce some allocations in IsExecutable"

395e076

This reverts commit 4052701.

Revert some unintended formatting changes

bdc8fc9

Must have some automatic formatting set that I'm not aware of. Didn't mean for these changes to be added.

Move the flag set back into the lock

43febac

Also remove redundant lock

SeeminglyScience commented Jul 12, 2023

View reviewed changes

StevenBucher98 added the PowerShell-Docs not needed The PR was reviewed and doesn't appear to require a PowerShell Docs update label Jul 17, 2023

daxian-dbw approved these changes Jul 17, 2023

View reviewed changes

daxian-dbw merged commit 153feb2 into PowerShell:master Jul 17, 2023

daxian-dbw added the CL-Engine Indicates that a PR should be marked as an engine change in the Change Log label Jul 17, 2023

SeeminglyScience deleted the fix-native-stdout-hang branch July 17, 2023 18:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix deadlock when piping to shell associated file extension #19940

Fix deadlock when piping to shell associated file extension #19940

Uh oh!

SeeminglyScience commented Jul 12, 2023 •

edited

Loading

Uh oh!

iSazonov commented Jul 12, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SeeminglyScience commented Jul 12, 2023

Uh oh!

pull-request-quantifier-deprecated bot commented Jul 12, 2023

What can I do to optimize my changes

How to interpret the change counts in git diff output

Uh oh!

SeeminglyScience commented Jul 12, 2023

Uh oh!

SeeminglyScience Jul 12, 2023

Uh oh!

iSazonov commented Jul 13, 2023

Uh oh!

SeeminglyScience commented Jul 13, 2023

Uh oh!

daxian-dbw left a comment

Uh oh!

rmccampbell commented Aug 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Fix deadlock when piping to shell associated file extension #19940

Fix deadlock when piping to shell associated file extension #19940

Uh oh!

Conversation

SeeminglyScience commented Jul 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

iSazonov commented Jul 12, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SeeminglyScience commented Jul 12, 2023

Uh oh!

pull-request-quantifier-deprecated bot commented Jul 12, 2023

What can I do to optimize my changes

How to interpret the change counts in git diff output

Uh oh!

SeeminglyScience commented Jul 12, 2023

Uh oh!

SeeminglyScience Jul 12, 2023

Choose a reason for hiding this comment

Uh oh!

iSazonov commented Jul 13, 2023

Uh oh!

SeeminglyScience commented Jul 13, 2023

Uh oh!

daxian-dbw left a comment

Choose a reason for hiding this comment

Uh oh!

rmccampbell commented Aug 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SeeminglyScience commented Jul 12, 2023 •

edited

Loading