Change detection of home directory to fallback to host if SMA.dll location is not found #18674

SteveL-MSFT · 2022-11-28T22:05:55Z

PR Summary

The path to the system wide powershell.config.json file depends on the location of SMA.dll. However, for some uses like hosting in a Windows Service, the .NET call to get the path to SMA.dll returns null which then subsequently fails a Path.Join() call. The change is to use AppContext.BaseDirectory instead of location of SMA.dll. In addition, the internal method takes a parameter that hasn't been used and has been removed.

I originally tried making the members nullable, however, this brought in a cascade of other changes needed to make it work across multiple files and also adds more regression risk since the behavior changes being nullable. Instead, I opened for this current approach which is more straight forward.

Also, this change is needed to experiment with building a single exe pwsh where SMA.dll won't be found.

PR Context

Fix #18109

PR Checklist

iSazonov · 2022-11-29T03:25:55Z

Perhaps we could use RuntimeType instead of Type in

PowerShell/src/System.Management.Automation/engine/Utils.cs

Lines 483 to 488 in 8686612

    
           internal static string GetApplicationBase(string shellId) 
        
           { 
        
               // Use the location of SMA.dll as the application base. 
        
               Assembly assembly = typeof(PSObject).Assembly; 
        
               return Path.GetDirectoryName(assembly.Location); 
        
           }

SteveL-MSFT · 2022-11-29T22:37:06Z

@iSazonov RuntimeType is internal, can you explain how you think it could be used here?

iSazonov · 2022-11-30T03:54:34Z

@iSazonov RuntimeType is internal, can you explain how you think it could be used here?

I mean something like "123".GetType().Assembly.

SteveL-MSFT · 2022-11-30T06:20:17Z

In this case, I don't think the result will be any different. The fundamental problem seems to be that in some hosted scenarios, SMA.dll's location is empty.

iSazonov · 2022-11-30T13:14:23Z

Maybe @jborean93 and @SeeminglyScience have ideas.

daxian-dbw · 2022-11-30T19:55:16Z

I think we need to fix GetApplicationBase instead.
This value of Utils.DefaultPowerShellAppBase is assumed to be always available and is depended at many other places, where A null value may cause problems that are unhandled.

We need to try reproducing this in a Windows Background Service to better understand the problem, and then contact the .NET team to see how to get the assembly location in that environment.

iSazonov · 2022-12-02T12:23:16Z

Perhaps it is more reliable to use AppContext.BaseDirectory. See implementation https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/AppContext.cs,22

From https://learn.microsoft.com/en-us/dotnet/api/system.appcontext.basedirectory?view=net-7.0#remarks

In .NET 5 and later versions, for bundled assemblies, the value returned is the containing directory of the host executable.

Source Browser

AppContext.BaseDirectory Property (System)
Gets the file path of the base directory that the assembly resolver uses to probe for assemblies.

SteveL-MSFT · 2022-12-02T17:28:43Z

@daxian-dbw I don't think we can rely on SMA.dll location as if we can get singleexe working, there isn't a SMA.dll as it's part of the exe. Perhaps @iSazonov's suggestion on base exe location is a better all around solution as long as hosted scenarios keep PowerShell assemblies in the same location as the host exe (which might not always be the case).

iSazonov · 2022-12-02T17:43:42Z

Perhaps @iSazonov's suggestion on base exe location is a better all around solution as long as hosted scenarios keep PowerShell assemblies in the same location as the host exe (which might not always be the case).

To be clear - AppContext.BaseDirectory defaults to host exe location but it can be configured by the host to point a place where managed dll-s are.

daxian-dbw · 2022-12-02T19:47:41Z

For an application that hosts PowerShell and build using dotnet publish, the PowerShell assemblies are under the publish\runtimes\win\lib\netx.x folder, while AppContext.BaseDirectory points to publish folder.

In the singleexe scenario, does typeof(PSObject).Assembly returns S.M.A.dll or the single exe binary that is produced from the build? If it's the latter, GetApplicationBase should continue to work unless the Assembly.Location always returns null for singleexe.

Also, does AppContext.BaseDirectory work in seingleexe? It's also using Assembly.Location, but just the entry assembly.

At any rates, simply handling the null case in PSConfiguration won't work, and we would need a more fundamental change to get the $PSHome value.

SteveL-MSFT · 2022-12-02T21:16:09Z

I have a branch that can build singleexe I can test this on. Note that singleexe currently is impossible to debug other than writing traces to the console.

Here's the results:

Assembly.FullName = System.Management.Automation, Version=7.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
Assembly.Location = 
AppContext.BaseDirectory = /Users/steve/repos/PowerShell/src/powershell-unix/bin/Debug/net7.0/osx-arm64/publish/

So it seems that AppContext.BaseDirectory may be the better fix here and we can document for host developers what they need to do if they don't have SMA.dll in the same folder as their host.

daxian-dbw · 2022-12-02T21:34:52Z

Looks good. Let's change to use AppContext.BaseDirectory then, and I can submit a PR to update the SDK NuGet package to binplace the module folder under publish instead of publish\runtimes\win\lib\netx.x as of today.

The only thing is we cannot verify whether AppContext.BaseDirectory works in Windows Background Service.

SteveL-MSFT · 2022-12-02T22:02:10Z

So SingleExe build does work as long as you don't use trimming. But the result is a 112MB executable.

I think for a Windows Service, we'll have to wait for user feedback.

daxian-dbw

LGTM

daxian-dbw

@SteveL-MSFT Can you please verify what AppContext.BaseDirectory returns in a remote session?

I found the following comments in https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/AddType.cs#LL628-L630C61

   // However, 'Assembly.GetEntryAssembly()' returns null when the managed code is called from
   // unmanaged code (PowerShell WSMan remoting scenario), so in that case, we continue to use
   // the location of 'System.Management.Automation.dll'.

Since AppContext.BaseDirectory uses Assembly.GetEntryAssembly(), I'm afraid it may not work a remote session.

SteveL-MSFT · 2022-12-02T23:01:22Z

@daxian-dbw unfortunately I don't have a system where I can do WinRM remoting (it doesn't work on win-arm64 due to the msix install sandbox)

SteveL-MSFT · 2022-12-02T23:43:47Z

@daxian-dbw I think we have our answer: https://dev.azure.com/powershell/PowerShell/_build/results?buildId=115792&view=logs&j=99798060-e5bf-55b9-b3b5-109e6acacce3&t=0050f0b1-14ee-5c4d-b4b0-da6405e3eaf0&l=413. So it won't work with remoting or hosted in unmanaged host.

SteveL-MSFT · 2022-12-03T00:10:47Z

What if we use AppContext.BaseDirectory only as a fallback?

src/System.Management.Automation/help/HelpProvider.cs

pull-request-quantifier-deprecated · 2023-03-13T23:20:34Z

This PR has 28 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details

Label      : Extra Small
Size       : +15 -13
Percentile : 11.2%

Total files changed: 7

Change summary by file extension:
.cs : +15 -13

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

Fast and predictable releases to production:
- Optimal size changes are more likely to be reviewed faster with fewer
  iterations.
- Similarity in low PR complexity drives similar review times.
Review quality is likely higher as complexity is lower:
- Bugs are more likely to be detected.
- Code inconsistencies are more likely to be detected.
Knowledge sharing is improved within the participants:
- Small portions can be assimilated better.
Better engineering practices are exercised:
- Solving big problems by dividing them in well contained, smaller problems.
- Exercising separation of concerns within the code changes.

What can I do to optimize my changes

Use the PullRequestQuantifier to quantify your PR accurately
- Create a context profile for your repo using the context generator
- Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
- Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
- Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
Change your engineering behaviors
- For PRs that fall outside of the desired spectrum, review the details and check if:
  - Your PR could be split in smaller, self-contained PRs instead
  - Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

One line was added: +1 -0
One line was deleted: +0 -1
One line was modified: +1 -1 (git diff doesn't know about modified, it will
interpret that line like one addition plus one deletion)
Change percentiles: Change characteristics (addition, deletion, modification)
of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍 :ok_hand: :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

daxian-dbw · 2023-03-20T17:21:43Z

src/System.Management.Automation/engine/Utils.cs

-            Assembly assembly = typeof(PSObject).Assembly;
-            return Path.GetDirectoryName(assembly.Location);
+            // AppContext is needed where PS7 may be compiled as a single exe
+            // Environment.ProcessPath covers the general case


What is the general case that Environment.ProcessPath is going to cover?
AppContext.BaseDirectory calls Environment.ProcessPath in case of NativeAOT, so I think that already covers it, isn't it?

daxian-dbw · 2023-03-20T17:24:47Z

src/System.Management.Automation/engine/Utils.cs

+            if (!string.IsNullOrEmpty(AppContext.BaseDirectory))
+            {
+                return Path.TrimEndingDirectorySeparator(AppContext.BaseDirectory);
+            }
+            else if (!string.IsNullOrEmpty(Path.GetDirectoryName(typeof(PSObject).Assembly.Location)))
+            {
+                return Path.GetDirectoryName(typeof(PSObject).Assembly.Location);
+            }


We should use a static variable to hold the resulted application base path, instead of running this every time.
Also, users can change the application path by AppContext.SetData("APP_CONTEXT_BASE_DIRECTORY", <new-path>), which may completely break PowerShell, so we have to cache the original value on the first call to GetApplicationBase.

daxian-dbw · 2023-03-20T17:28:22Z

src/System.Management.Automation/engine/Utils.cs

 #endif

-        internal static string DefaultPowerShellAppBase => GetApplicationBase(DefaultPowerShellShellID);
+        internal static string DefaultPowerShellAppBase => GetApplicationBase();


The AddType.cs needs to be updated too:

PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/AddType.cs

Lines 634 to 640 in 8f0cd65

private static readonly string s_netcoreAppRefFolder = PathType.Combine(

PathType.GetDirectoryName(

(Assembly.GetEntryAssembly() ?? typeof(PSObject).Assembly).Location),

"ref");

// Path to the folder where .NET Core runtime assemblies are located.

private static readonly string s_frameworkFolder = PathType.GetDirectoryName(typeof(object).Assembly.Location);

s_netcoreAppRefFolder should be just using GetApplicationBase(). As for s_frameworkFolder, the reference of this variable can be replaced by GetApplicationBase() directly I think.

daxian-dbw · 2023-03-20T17:29:43Z

src/System.Management.Automation/engine/Utils.cs

        }
 #endif

-        internal static string DefaultPowerShellAppBase => GetApplicationBase(DefaultPowerShellShellID);


I guess we can remove all the reference to DefaultPowerShellAppBase now, and just GetApplicationBase() directly instead.

daxian-dbw · 2023-03-20T17:30:09Z

src/System.Management.Automation/engine/CommandCompletion/CompletionCompleters.cs

        {
            var results = new List<CompletionResult>();
            string userHelpDir = HelpUtils.GetUserHomeHelpSearchPath();
-            string appHelpDir = Utils.GetApplicationBase(Utils.DefaultPowerShellShellID);


Can we remove Utils.DefaultPowerShellID now?

daxian-dbw · 2023-03-20T18:10:57Z

This change will cause a problem to the Microsoft.PowerShell.SDK package., so adding @adityapatwardhan for input.

Today, $PSHome points to the folder where S.M.A.dll is located, so for a self-contained application that hosts PowerShell, the $PSHome points to different locations when it's running on Windows vs. on Unix (e.g. publish\runtimes\win\lib\net7.0 vs. publish\runtimes\unix\lib\net7.0).

The way we ship the built-in modules in Microsoft.PowerShell.SDK leveraged that -- when running dotnet publish, the Windows version built-in modules will be bin-placed to runtimes\win\lib\net7.0\Modules, and the Unix version of built-in modules will be bin-placed to runtimes\unix\lib\net7.0\Modules. As a result, the application will have the expected modules in its $PSHome\Modules module path.

After this change, the $PSHome will points to the same location for the application (i.e. the publish folder), but we cannot bin-place both the Windows version and Unix version of built-in modules to the same location publish\Modules. This will break the PowerShell NuGet package, which is depended by Azure Function and .NET Interactive.

ghost · 2023-04-04T20:00:53Z

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 15 days. It will be closed if no further activity occurs within 10 days of this comment.

daxian-dbw · 2023-04-27T18:11:28Z

@adityapatwardhan Please take a look at #18674 (comment). We may need to brainstorm on this NuGet packaging issue.

ghost · 2023-05-16T23:00:46Z

This pull request has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 15 days. It will be closed if no further activity occurs within 10 days of this comment.

SCLDGit · 2023-09-27T13:57:38Z

This is a huge thorn in our side. As long as there are vital configuration functions which are only available via PowerShell (such as the enabling/disabling of server roles and features), which our application must be able to call via C#, this is going to continue to be a headache for us and our deployment requirements. The .NET Core 3.1 version of single file deployment (extract all files to /temp) is simply not acceptable to our customer base for security reasons.

pull-request-quantifier-deprecated bot added the Extra Small label Nov 28, 2022

SteveL-MSFT requested a review from daxian-dbw November 28, 2022 22:05

ghost assigned TravisEz13 Nov 28, 2022

SteveL-MSFT requested a review from adityapatwardhan as a code owner December 2, 2022 22:10

SteveL-MSFT changed the title ~~Handle case where system wide config path cannot be found~~ Change detection of home directory to the host instead of SMA.dll location Dec 2, 2022

SteveL-MSFT force-pushed the null-systemconfig branch from 4946e69 to 38e1c0c Compare December 2, 2022 22:15

daxian-dbw approved these changes Dec 2, 2022

View reviewed changes

daxian-dbw requested changes Dec 2, 2022

View reviewed changes

ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Dec 2, 2022

SteveL-MSFT force-pushed the null-systemconfig branch from 38e1c0c to 958ed20 Compare December 2, 2022 23:00

ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Dec 2, 2022

SteveL-MSFT changed the title ~~Change detection of home directory to the host instead of SMA.dll location~~ Change detection of home directory to fallback to host if SMA.dll location is not found Dec 3, 2022

iSazonov reviewed Dec 3, 2022

View reviewed changes

src/System.Management.Automation/help/HelpProvider.cs Outdated Show resolved Hide resolved

iSazonov reviewed Dec 5, 2022

View reviewed changes

src/System.Management.Automation/help/HelpProvider.cs Outdated Show resolved Hide resolved

SteveL-MSFT added 4 commits March 7, 2023 21:21

trim trailing separator from appcontext.basedirectory

e0b6632

fix use of trimming empty basedirectory

1da2cc0

fallback to original code for hosted remoting scenario

b13b699

remove use of null coalescing operator since it only works against null

5c56e31

SteveL-MSFT force-pushed the null-systemconfig branch from 5e8aa23 to 5c56e31 Compare March 8, 2023 05:21

SteveL-MSFT closed this Mar 13, 2023

SteveL-MSFT reopened this Mar 13, 2023

ghost removed the Review - Needed The PR is being reviewed label Mar 13, 2023

add back original source

ff8f539

daxian-dbw requested changes Mar 20, 2023

View reviewed changes

ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 20, 2023

ghost added the Stale label Apr 4, 2023

ghost closed this Apr 15, 2023

daxian-dbw reopened this Apr 17, 2023

ghost closed this Apr 27, 2023

daxian-dbw reopened this Apr 27, 2023

daxian-dbw removed the Stale label Apr 27, 2023

daxian-dbw added WG-Engine core PowerShell engine, interpreter, and runtime CL-Engine Indicates that a PR should be marked as an engine change in the Change Log Needs-Triage The issue is new and needs to be triaged by a work group. labels May 1, 2023

ghost added the Stale label May 16, 2023

ghost closed this May 27, 2023

microsoft-github-policy-service bot removed the Stale label Sep 27, 2023

daxian-dbw mentioned this pull request Aug 5, 2024

If SMA.dll location is not found, use host exe to determine $PSHOME location #24072

Merged

21 tasks

This pull request was closed.

	private static readonly string s_netcoreAppRefFolder = PathType.Combine(
	PathType.GetDirectoryName(
	(Assembly.GetEntryAssembly() ?? typeof(PSObject).Assembly).Location),
	"ref");

	// Path to the folder where .NET Core runtime assemblies are located.
	private static readonly string s_frameworkFolder = PathType.GetDirectoryName(typeof(object).Assembly.Location);

Change detection of home directory to fallback to host if SMA.dll location is not found #18674

Change detection of home directory to fallback to host if SMA.dll location is not found #18674

Uh oh!

Conversation

SteveL-MSFT commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

iSazonov commented Nov 29, 2022

Uh oh!

SteveL-MSFT commented Nov 29, 2022

Uh oh!

iSazonov commented Nov 30, 2022

Uh oh!

SteveL-MSFT commented Nov 30, 2022

Uh oh!

iSazonov commented Nov 30, 2022

Uh oh!

daxian-dbw commented Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov commented Dec 2, 2022 • edited by unfurl-links bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SteveL-MSFT commented Dec 2, 2022

Uh oh!

iSazonov commented Dec 2, 2022

Uh oh!

daxian-dbw commented Dec 2, 2022

Uh oh!

SteveL-MSFT commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daxian-dbw commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SteveL-MSFT commented Dec 2, 2022

Uh oh!

daxian-dbw left a comment

Choose a reason for hiding this comment

Uh oh!

daxian-dbw left a comment

Choose a reason for hiding this comment

Uh oh!

SteveL-MSFT commented Dec 2, 2022

Uh oh!

SteveL-MSFT commented Dec 2, 2022

Uh oh!

SteveL-MSFT commented Dec 3, 2022

Uh oh!

Uh oh!

Uh oh!

pull-request-quantifier-deprecated bot commented Mar 13, 2023

What can I do to optimize my changes

How to interpret the change counts in git diff output

Uh oh!

daxian-dbw Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

daxian-dbw Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

daxian-dbw Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

daxian-dbw Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

daxian-dbw Mar 20, 2023

Choose a reason for hiding this comment

Uh oh!

daxian-dbw commented Mar 20, 2023

Uh oh!

ghost commented Apr 4, 2023

Uh oh!

daxian-dbw commented Apr 27, 2023

Uh oh!

ghost commented May 16, 2023

SteveL-MSFT commented Nov 28, 2022 •

edited

Loading

daxian-dbw commented Nov 30, 2022 •

edited

Loading

iSazonov commented Dec 2, 2022 •

edited by unfurl-links bot

Loading

SteveL-MSFT commented Dec 2, 2022 •

edited

Loading

daxian-dbw commented Dec 2, 2022 •

edited

Loading