Skip to content

Conversation

@iSazonov
Copy link
Collaborator

@iSazonov iSazonov commented Oct 18, 2021

PR Summary

Latest PerfView doesn't load PDB files for PowerShell assemblies (SMA and other) without the fix.

After adding PublishReadyToRunEmitSymbols I had to exclude NewtonSoft.JSON.dll from processing by crossgen2.exe because this broke PDB loading.

PR Context

PR Checklist

@iSazonov iSazonov added the CL-BuildPackaging Indicates that a PR should be marked as a build or packaging change in the Change Log label Oct 18, 2021
@daxian-dbw
Copy link
Member

This PR feels more like a workaround for yourself, rather than a complete fix for an issue:

  1. Did it work on previous older version of PerfView?
  2. What's the root cause? Could it be a problem with the latest PerfView?
  3. Removing Newtonsoft.Json.dll means it will not be pre-jitted, and that will impact the startup.

Given that, I will turn this into a draft for now.

@daxian-dbw daxian-dbw marked this pull request as draft October 18, 2021 19:59
@iSazonov
Copy link
Collaborator Author

iSazonov commented Oct 19, 2021

This PR feels more like a workaround for yourself, rather than a complete fix for an issue:

  1. Did it work on previous older version of PerfView?

No, I started with old version (six months old).

Also there was a problem with .Net dotnet/runtime#53743

  1. What's the root cause? Could it be a problem with the latest PerfView?

For .Net 5.0 and crossgen 1.0, PerfView generated ni.pdb files for PowerShell assemblies on the fly.
Now it doesn't. When I try to load SMA.pdb, I see "Out of memory" message in the Perfview log.
After I manually processed the SMA with crossgen2.exe I get SMA.ni.dll (with exactly the same size as original one) and SMA.ni.pdb. After renaming SMA.ni.dll to SMA.dll PerfView loads SMA.ni.pdb and show all callsites in trace.

I can not point a root of the issue since crossgen2.exe is not well documented. But I found a comment that now PDB files contain a reference to ni.pdb. This force me think that our crossgen process is out of date.

  1. Removing Newtonsoft.Json.dll means it will not be pre-jitted, and that will impact the startup.

I measured and don't see noticeable difference (81 vs 80 ms for reading/parsing pwsh config on my notebook). I guess Newtonsoft.Json implementation is not optimized for .Net at all.

Given that, I will turn this into a draft for now.

I don't know how fix right the issue with Newtonsoft.Json.dll but I am sure we should enable PublishReadyToRunEmitSymbols.
I think it is a time to migrate from Newtonsoft.Json.dll to System.Text.Json in the config R/W scenario. 10% to read json config at startup is too many/slow. I hope System.Text.Json will be a bit faster.

Update: I have refreshed my early experiment with using System.Text.Json from .Net 5 to .Net 6.0 RC2 and now I see good perf win for ReadValueFromFile() ~4x.

@iSazonov
Copy link
Collaborator Author

iSazonov commented Oct 21, 2021

@daxian-dbw After thinking more I am sure the PublishReadyToRunEmitSymbols must be added to 7.2 release otherwise we will distribute "broken" PDB files.
I guess we can not fix the issue with NewtonSoft.Json because it is not compiled for new .Net 6.0 format - in any case we have broken either pdb or ni.pdb scenario. I think to remove the NewtonSoft.Json workaround from the PR - it is for MSFT team conclusion (should we still crossgen the dll for Add-Type as comment in Build.psm1 say?).

@daxian-dbw
Copy link
Member

I will take a look tomorrow. The 7.2-rc.1 release build already started, but if this is proven to be really needed, we may still have time to include it in the GA release.

Copy link
Member

@daxian-dbw daxian-dbw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, profiler needs the symbols generated by <PublishReadyToRunEmitSymbols> to examine the R2R files. However, those symbols are only useful for profiling the startup scenario of PowerShell because PowerShell enables tiered compilation which will overwrite the R2R code.

Given that it's a very specific dev scenario, I don't think we should always generate the native symbols when build and publish R2R images. Instead, there should be a switch parameter to Start-PSBuild -CreateR2RSymbols to turn it on when needed.

<TargetFramework>net6.0</TargetFramework>
<LangVersion>10.0</LangVersion>
<PublishReadyToRun Condition=" '$(Configuration)' != 'Debug' ">true</PublishReadyToRun>
<PublishReadyToRunEmitSymbols>true</PublishReadyToRunEmitSymbols>
Copy link
Member

@daxian-dbw daxian-dbw Oct 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PublishReadyToRunEmitSymbols>true</PublishReadyToRunEmitSymbols>
<PublishReadyToRunEmitSymbols Condition=" '$(Configuration)' != 'Debug' And '$(CreateR2RSymbols)' == '1'">true</PublishReadyToRunEmitSymbols>

@daxian-dbw
Copy link
Member

This doesn't need to be included in 7.2 GA. PowerShell packages don't ship pdb files anyways.

@iSazonov
Copy link
Collaborator Author

iSazonov commented Oct 22, 2021

You are right, profiler needs the symbols generated by to examine the R2R files. However, those symbols are only useful for profiling the startup scenario of PowerShell because PowerShell enables tiered compilation which will overwrite the R2R code.

It is not only for startup scenario. It is for all scenarios including hosting. (User should disable TC)
Anyone who is going to measure PowerShell or their .Net 6.0 hosting application will encounter this problem.
Next version is LTS and it is not god to have broken pdbs for next 3 years.

PowerShell packages don't ship pdb files anyways.

I think they distributed by Microsoft Symbol servers. If no it must be.

Also I guess .Net are migrating to portable pdb format where possible dotnet/runtime#45492
We use portable pdb for Unix but full for Windows. I think it is a time to use portable on Windows too.
<DebugType>portable</DebugType> in PowerShell.Common.props

Official intention

We run still crossgen for standard .Net dll-s. I think it should be removed.

PowerShell/build.psm1

Lines 2498 to 2499 in 612aade

# Common assemblies used by Add-Type or assemblies with high JIT and no pdbs to crossgen
$commonAssembliesForAddType = @(

@daxian-dbw
Copy link
Member

daxian-dbw commented Oct 22, 2021

It is not only for startup scenario. It is for all scenarios including hosting. (User should disable TC)

PowerShell nuget packages use IL assemblies, not R2R images. So applications that hosting powershell using NuGet packages don't have this problem. They can choose to publish R2R images for their applications though, but that's nothing to do with the powershell build.

Today PowerShell symbols are not distributed by any symbol servers. It's in our backlog.
For portable pdb on Windows, I think our build requires Windows pdb on Windows for APIScan and other compliance jobs, @TravisEz13 can share more insights regarding this.

We run still crossgen for standard .Net dll-s. I think it should be removed.

Does <PublishReadyToRun> generate R2R images for them all? Or, are those .NET runtime assemblies already in R2R image format?

@iSazonov
Copy link
Collaborator Author

For portable pdb on Windows, I think our build requires Windows pdb on Windows for APIScan and other compliance jobs, @TravisEz13 can share more insights regarding this.

It is already in PowerShell.Common.props - for compliance and release build there are different conditions.

@iSazonov
Copy link
Collaborator Author

Does <PublishReadyToRun> generate R2R images for them all? Or, are those .NET runtime assemblies already in R2R image format?

Yes, .Net 6.0 dll-s are already in R2R format.

@daxian-dbw
Copy link
Member

Yes, .Net 6.0 dll-s are already in R2R format.

Can you share the link to the documentation/issue/blog that confirms this? I have been assuming this but couldn't find an official confirmation.

As for portable pdb, I will leave it to @TravisEz13 to comment.

@iSazonov
Copy link
Collaborator Author

Can you share the link to the documentation/issue/blog that confirms this?

I can not find explicit post but I saw :-) I could open issue-question in Runtime repo if you want.

Now I opened one dll in ILSpy and it show R2R
image

@daxian-dbw
Copy link
Member

That's perfect! I have been looking for ways to deterministically know if an assembly is R2R image!

It seems the following 3 assemblies (at least) are not R2R image:
System.Runtime.Extensions.dll
System.IO.FileSystem.dll
System.IO.FileSystem.Primitives.dll

Can you verify with ILSpy?

@iSazonov
Copy link
Collaborator Author

Only System.IO.FileSystem.Primitives.dll is not crossgen2 as ILSpy shows.

Perhaps PublishReadyToRun does crossgen for PowerShell. Will ask in Runtime.

@daxian-dbw
Copy link
Member

How come it doesn't show Crossgen2 message to me? I'm using ILSpy version 7.1.0.6543. How to get the Crossgen2 ... message shown up?

image

@iSazonov
Copy link
Collaborator Author

You need switch from C# to ReadyToRun in menu

@iSazonov
Copy link
Collaborator Author

iSazonov commented Oct 22, 2021

Issue in .Net Runtime dotnet/runtime#60779

@iSazonov
Copy link
Collaborator Author

System.IO.FileSystem.Primitives.dll is compiled for AnyCPU. So it can not be in R2R :-)

@daxian-dbw
Copy link
Member

daxian-dbw commented Oct 22, 2021

You need switch from C# to ReadyToRun in menu

Thanks! You are using 7.2-preview.1. I see the ReadyToRun option after switching to that version.

System.IO.FileSystem.Primitives.dll is compiled for AnyCPU. So it can not be in R2R :-)

Yeah. both System.Runtime.Extensions.dll and System.IO.FileSystem.dll are also targeting AnyCPU. If you see that differently, you must be using the assembly that are further processed by Start-Crossgen in powershell build.

The .NET team has confirmed that ALL .NET 6 runtime assemblies should be R2R images.
Based on that, I opened dotnet/runtime#60782 to report the findings.
[Update] It turns out this is by design because those assemblies as pure façade assemblies.

I have also verified that <PublishReadyToRun>true</PublishReadyToRun> actually crossgen'ed all the package assemblies pwsh depends on, including all that in the $commonAssembliesForAddType array today. The following is the pdb generated for those crossgen'ed assemblies.

InputObject                                           SideIndicator
-----------                                           -------------
Markdig.Signed.ni.pdb                                 =>
Microsoft.ApplicationInsights.ni.pdb                  =>
Microsoft.CodeAnalysis.CSharp.ni.pdb                  =>
Microsoft.CodeAnalysis.ni.pdb                         =>
Microsoft.Extensions.ObjectPool.ni.pdb                =>
Microsoft.Management.Infrastructure.CimCmdlets.ni.pdb =>
Microsoft.Management.Infrastructure.Native.ni.pdb     =>
Microsoft.Management.Infrastructure.ni.pdb            =>
Microsoft.PowerShell.Commands.Diagnostics.ni.pdb      =>
Microsoft.PowerShell.Commands.Management.ni.pdb       =>
Microsoft.PowerShell.Commands.Utility.ni.pdb          =>
Microsoft.PowerShell.ConsoleHost.ni.pdb               =>
Microsoft.PowerShell.CoreCLR.Eventing.ni.pdb          =>
Microsoft.PowerShell.GraphicalHost.ni.pdb             =>
Microsoft.PowerShell.MarkdownRender.ni.pdb            =>
Microsoft.PowerShell.Security.ni.pdb                  =>
Microsoft.WSMan.Management.ni.pdb                     =>
Microsoft.WSMan.Runtime.ni.pdb                        =>
Namotion.Reflection.ni.pdb                            =>
Newtonsoft.Json.ni.pdb                                =>
NJsonSchema.ni.pdb                                    =>
pwsh.ni.pdb                                           =>
System.ComponentModel.Composition.ni.pdb              =>
System.ComponentModel.Composition.Registration.ni.pdb =>
System.Data.Odbc.ni.pdb                               =>
System.Data.OleDb.ni.pdb                              =>
System.Data.SqlClient.ni.pdb                          =>
System.DirectoryServices.AccountManagement.ni.pdb     =>
System.DirectoryServices.Protocols.ni.pdb             =>
System.IO.Ports.ni.pdb                                =>
System.Management.Automation.ni.pdb                   =>
System.Management.ni.pdb                              =>
System.Net.Http.WinHttpHandler.ni.pdb                 =>
System.Private.ServiceModel.ni.pdb                    =>
System.Reflection.Context.ni.pdb                      =>
System.Runtime.Caching.ni.pdb                         =>
System.ServiceModel.Syndication.ni.pdb                =>
System.ServiceProcess.ServiceController.ni.pdb        =>
System.Speech.ni.pdb                                  =>

Given all above, I think we consider removing the Start-CrossGen code now. I will try working on that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-BuildPackaging Indicates that a PR should be marked as a build or packaging change in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants