Skip to content

Fix race condition to access powershell.config.json - #8249

Merged
Aditya Patwardhan (adityapatwardhan) merged 5 commits into
PowerShell:masterfrom
iSazonov:fix-race-cond
Nov 28, 2018
Merged

Aditya Patwardhan (adityapatwardhan) merged 5 commits into
PowerShell:masterfrom
iSazonov:fix-race-cond

Conversation

@iSazonov

@iSazonov Ilya (iSazonov) commented Nov 13, 2018 •

Copy link
Copy Markdown
Collaborator

PR Summary

See #8237 (comment)

(Also there can be a race condition if some processes access the config file. I don't now how to fix.)

PR Checklist

@anmenaga

Copy link
Copy Markdown

Restarted builds because of "Get-PSGalleryApiAvailability : PowerShell Gallery is currently unavailable. Please try again later."

@TravisEz13 Travis Plunk (TravisEz13) added WG-Engine core PowerShell engine, interpreter, and runtime CL-Engine Indicates that a PR should be marked as an engine change in the Change Log labels Nov 14, 2018

@rjmholt Rob Holt (rjmholt) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

To prevent other processes from accessing, is it possible to take out an OS-level write lock on the json file? After searching for a bit on the web, it seems like new FileStream() with the right arguments already does this, no?

@iSazonov

Copy link
Copy Markdown
Collaborator Author

Rob Holt (@rjmholt) A problem I indicated is that we don't process an exception if file is locked for read. Scenario: run two pwsh-s, one lock the config file, second is trying to access the file and - what? Wait and stop whole process? Currently we'll crash. On a system where many PowerShell processes run, this can cause unpredictable (rarely) crashes. (At minimum, we could not put an exclusive write lock.)

@rjmholt

Copy link
Copy Markdown
Collaborator

Hmmmm. Well given that the config file is changed at runtime by Set-ExecutionPolicy (and possibly others), maybe it should crash (more gracefully) -- given that config files may be changed by the PowerShell session, maybe they are session specific?

Otherwise we are forced to use some OS-level lock to get an atomic interleaving of PowerShell config writes?

@iSazonov

Copy link
Copy Markdown
Collaborator Author

I added a simple file unlock wait. I think it will resolve the inter process race condition in most of cases. Of cause an admin user can still exclusively open the file with read lock and crash pwsh processes. Seems it is an edge case.

@iSazonov

Copy link
Copy Markdown
Collaborator Author

MSFT team please review new commit.

@iSazonov Ilya (iSazonov) mentioned this pull request Nov 14, 2018
5 of 11 tasks
@iSazonov

Copy link
Copy Markdown
Collaborator Author

@anmenaga

Copy link
Copy Markdown

I'm not a fan of this WaitForFile addition (and looped wait timers in general unless they are absolutely necessary).
Original problem was intermittently sharing read access to the file between PS processes. FileShare.ReadWrite is good enough for this. I would remove WaitForFile addition (especially considering that it only partially addresses the issue that it's trying to address).

@rjmholt

Copy link
Copy Markdown
Collaborator

Yes, sorry, only meant to discuss rather than request changes (couldn't find a good issue to discuss in)

@iSazonov

Ilya (iSazonov) commented Nov 16, 2018 •

Copy link
Copy Markdown
Collaborator Author

Original problem was intermittently sharing read access to the file between PS processes.

Original problem is a bug that is in-process race condition. It is hosting scenarios. Fixed by first commit.
Second commit is for inter-process race condition. It is scenarios with many powershell processes (like background scheduled tasks). Fixed by second commit.

I think both commits is important because pwsh crashed.

looped wait timers in general unless they are absolutely necessary

The solution looks good for me because it is very rare race condition and we will never block a process for long especially since we cache powershell.config.json.

@rjmholt

Copy link
Copy Markdown
Collaborator

The looped wait timer approach I think looks good in comparison to the alternative: a corrupt powershell.config.json which ideally leads to a crash, but otherwise leads to one of the PowerShell processes incorrectly picking up a new configuration.

I get the impression though that perhaps a shared mutable configuration between processes is something we should reconsider as an idea?

@iSazonov

Ilya (iSazonov) commented Nov 16, 2018 •

Copy link
Copy Markdown
Collaborator Author

I get the impression though that perhaps a shared mutable configuration between processes is something we should reconsider as an idea?

This could be addressed by PowerShell/PowerShell-RFC#111 too.

@adityapatwardhan

Copy link
Copy Markdown
Member

Dongbo Wang (@daxian-dbw) Can you have a look at this?

@iSazonov

Copy link
Copy Markdown
Collaborator Author

Aditya Patwardhan (@adityapatwardhan) Dongbo Wang (@daxian-dbw) The PR block sequence of other PRs. Could you please continue?

@daxian-dbw Dongbo Wang (daxian-dbw) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look mostly good to me. Please address the 3 comments.

using (var streamReader = new StreamReader(fileName))
// The config file can be locked by another process
// so we wait some milliseconds in 'WaitForFile()' for recovery before stop current process.
using (var readerStream = WaitForFile(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By having FileShare.ReadWrite, we allow the file to be opened for writing by another process. FileShare.ReadWrite could only possibly work in a logging scenario, where new content is always appended to the end of a file and you read the file through seek. I think it should be changed to FileShare.Read.

@iSazonov Ilya (iSazonov) Nov 23, 2018 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With FileShare.Read the PowerShell Core process will terminate if anybody open the file for writing. We aim to avoid this and open the file with minimal lock level. Also we don't keep the file opened: every time we create new StreamReader on very short time. There still can be a race condition if we read partially changed file but it is another issue - what PowerShell Core should do if the config file is broken.

@daxian-dbw Dongbo Wang (daxian-dbw) Nov 26, 2018 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this permission raises more confusion than the potential race condition it might mitigate. I think the right approach would be to return a default setting when the file cannot be opened after the retry, or when the file is corrupted (schema validation would be good but that would impact the startup time).

The current design is to only read the particular setting that is requested, which will result in opening the files multiple times but actually reading the whole file every time. I don't think this is the right thing to do. We probably should be very explicit that the setting file is read only one time at startup, and we process all settings that is present in the file. If the file cannot be opened after retry or the content is corrupted, we use the default settings and somehow convey the error to user (yes, we need to define the default settings first). In this way, the file is read only once, which means less race conditions, and less file access cost. Anyways, this should be a separate PR, not a change request for this PR. Here we only focus on the share permission.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I change anything in the PR?

@daxian-dbw Dongbo Wang (daxian-dbw) Nov 26, 2018 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this permission raises more confusion than the potential race condition it might mitigate. I think the right approach would be to return a default setting when the file cannot be opened after the retry, or when the file is corrupted (schema validation would be good but that would impact the startup time).

Yes, I think changing FileShare.ReadWrite to FileShare.Read :)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that this resolve the race condition in our xUnit tests - I started the PR to get this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your xUnit tests actively modify the config file (write lock?) and this lead to crash other tests .
I agree that here we can only temporary resolve the problem and we should work on better solution. (Address in my RFC or make PR?)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dongbo Wang (@daxian-dbw) returning a default settings file could result in security settings in the config file being bypassed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your xUnit tests actively modify the config file (write lock?) and this lead to crash other tests .

Ilya (@iSazonov) that's unfortunate to hear :( Then let's go with FileShare.ReadWrite for now.

Travis Plunk (@TravisEz13) I agree. It sounds to me crashing pwsh when config file parsing fails is more securer :)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dongbo Wang (@daxian-dbw) Travis Plunk (@TravisEz13) Could you please add more feedback about "security settings" to my RFC? We could discuss this there. I don't like the "crashing pwsh" because it could be a DoS attack even by admin user.

@iSazonov

Copy link
Copy Markdown
Collaborator Author

Aditya Patwardhan (@adityapatwardhan) Please merge the PR.

@adityapatwardhan

Copy link
Copy Markdown
Member

Ilya (@iSazonov) Can you fix the two CodeFactor issues? Then the PR can be merged.

@iSazonov

Copy link
Copy Markdown
Collaborator Author

Aditya Patwardhan (@adityapatwardhan) CodeFactor issues was fixed.

@adityapatwardhan
Aditya Patwardhan (adityapatwardhan) merged commit 9381d03 into PowerShell:master Nov 28, 2018
@iSazonov
Ilya (iSazonov) deleted the fix-race-cond branch November 29, 2018 03:06
Ilya (iSazonov) added a commit to iSazonov/PowerShell that referenced this pull request Nov 29, 2018
Remove unneeded dotnet restore

Add  --no-restore

Add extraParams for MacOS

Fix TestRunspaceWithPowerShellAndInitialSessionState

Fix race condition to access powershell.config.json

Reduce an impact of the inter process race condition

Fix race condition to access powershell.config.json (PowerShell#8249)
Ilya (iSazonov) added a commit to iSazonov/PowerShell that referenced this pull request Nov 29, 2018
Remove unneeded dotnet restore

Add  --no-restore

Add extraParams for MacOS

Fix TestRunspaceWithPowerShellAndInitialSessionState

Fix race condition to access powershell.config.json

Reduce an impact of the inter process race condition

Fix race condition to access powershell.config.json (PowerShell#8249)
Travis Plunk (TravisEz13) pushed a commit to iSazonov/PowerShell that referenced this pull request Dec 12, 2018
Remove unneeded dotnet restore

Add  --no-restore

Add extraParams for MacOS

Fix TestRunspaceWithPowerShellAndInitialSessionState

Fix race condition to access powershell.config.json

Reduce an impact of the inter process race condition

Fix race condition to access powershell.config.json (PowerShell#8249)
Thatgfsj (Thatgfsj) pushed a commit to Thatgfsj-contribs/PowerShell that referenced this pull request Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-Engine Indicates that a PR should be marked as an engine change in the Change Log WG-Engine core PowerShell engine, interpreter, and runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants