Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 #6958

SteveL-MSFT · 2018-05-30T16:30:05Z

PR Summary

To support PowerShell Modules built with .Net Windows Compatibility Pack, we decided that it was best to ship the WCP assemblies with PSCore6. This also adds many new .Net apis be default while only adding ~3.5 MB additional disk footprint (to a ~137 MB install currently).

Added ref to WCP 2.0.0 to SDK csproj.

Updated all refs of 4.5.0-RC1 to 4.5.0 final.

Update dotnetcore2.1 to final build

Update WebListener to aspnetcore.app 2.1.0 final
(Microsoft.VisualStudio.Web.CodeGeneration.Tools is not final yet, so no change)

PR Checklist

daxian-dbw · 2018-05-30T16:43:11Z

src/powershell-win-core/powershell-win-core.csproj

    <PackageReference Include="PSDesiredStateConfiguration" Version="6.0.0-beta.8" />
    <PackageReference Include="PowerShellHelpFiles" Version="1.0.0-*" />
    <PackageReference Include="psrp.windows" Version="6.1.0-*" />
+    <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" />


I think this reference should be put in SDK.csproj with a Condition attribute, like <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" Condition=" '$(IsWindows)' != 'true' " />. In this way, application that refernce to SDK nuget package will get those extended pack too.

IsWindows is defined in https://github.com/PowerShell/PowerShell/blob/master/PowerShell.Common.props#L113

/cc @adityapatwardhan, in case I'm wrong about the SDK reference part.

Actually, I don't know if we should even have the condition part. Some of the compat assemblies work on Unix platforms too.

Yes, this should be part of SDK, so apps hosting PowerShell can use the modules with WCP.

Perhaps we don't need the condition as @daxian-dbw is correct that some of the assemblies from the WCP meta-package work on non-Windows so we should make those available. I'll make the change.

updated all refs of 4.5.0-RC1 to 4.5.0 final

daxian-dbw · 2018-05-30T17:59:45Z

Ha, it looks dotnet core 2.1 is officially out: https://www.nuget.org/packages/Microsoft.NETCore.App/2.1.0
Shall we update our build to use the official 2.1?

markekraus · 2018-05-30T19:11:48Z

@SteveL-MSFT Can you push a commit with [feature] to ensure that the version bump didn't cause any regressions?

bergmeister · 2018-05-30T19:12:31Z

@SteveL-MSFT I believe you also need to update the installer for .Net Core 2.1 RTM and cleanup comments, see my commit here

update cache names for CI update files.wxs to reflect new versions

iSazonov · 2018-05-31T06:50:39Z

We have a issue with System.Text.Encoding.CodePages. Maybe not still published.

iSazonov · 2018-05-31T06:52:51Z

We could use Version="4.5.*" for packages.

iSazonov · 2018-05-31T15:31:44Z

we decided that it was best to ship the WCP assemblies with PSCore6

Can we use it directly - to access registry, perfcounters and so on?

SteveL-MSFT · 2018-05-31T16:37:31Z

@iSazonov System.Text.Encoding.CodePages 4.5.0 is published: https://www.nuget.org/packages/System.Text.Encoding.CodePages/

update pwrshplugin.dll trusted assembly list

daxian-dbw · 2018-05-31T22:01:19Z

src/powershell-native/nativemsh/pwrshcommon/pwrshcommon.cpp

-        "Microsoft.CodeAnalysis",
        "Microsoft.CodeAnalysis.CSharp",
+        "Microsoft.CodeAnalysis",
+        "Microsoft.CodeAnalysis.VisualBasic",


Do we want to include this one? I think we will remove VB support before 6.1 RC. If we include it here now, we will have to update the windows.psrp package again when removing the VB support.

Will remove. We can always add it back later if needed.

daxian-dbw · 2018-05-31T22:05:13Z

src/Microsoft.PowerShell.SDK/Microsoft.PowerShell.SDK.csproj

  </ItemGroup>

  <ItemGroup>
+    <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" />


If we don't know the source location of this package, maybe put it at the end, in the section with the comment the source could not be found for the following package(s), along with Microsoft.NETCore.Windows.ApiSets

address Dongbo's feedback

SteveL-MSFT · 2018-06-01T14:34:07Z

Can't tell why System.Text.Encoding.CodePages is failing to load as the signature matches the assembly.

iSazonov · 2018-06-01T15:02:00Z

I suspect this is a publishing error. Could you ask .Net team?

SteveL-MSFT · 2018-06-01T16:35:44Z

@iSazonov that xunit test passes fine on my desktop machine, just not in AppVeyor though

daxian-dbw · 2018-06-01T17:36:27Z

I see the following from Start-PSxUnit:

            $requiredDependencies = @(
                $nativeLib,
                "$Content/Microsoft.Management.Infrastructure.dll",
                "$Content/System.Text.Encoding.CodePages.dll"
            )

The codepages.dll line is suspicious to me. But I haven't spent time digging further yet.

Removed unncessary dep ref for psxunit on System.Text.Encoding.CodePages.dll Updated packaging nupkg dep assembly versions

iSazonov · 2018-06-01T18:05:45Z

Errors in Appveyor tests:

[00:21:47]   Describing CmsMessage cmdlets thorough tests
[00:21:48] +++++++++++++++++++?+
[00:21:48] 
[00:21:48] New-Item : The directory specified, 'Modules', is not a subdirectory of 'C:\'.
[00:21:48] Parameter name: path
[00:21:48] At C:\projects\powershell\test\powershell\Modules\Microsoft.PowerShell.Security\ConstrainedLanguageRestriction.Tests.ps1:87 char:21
[00:21:48] + ...          $null = New-Item -ItemType Directory $moduleDirectory -Force
[00:21:48] +                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[00:21:48] + CategoryInfo          : InvalidArgument: (C:\Modules:String) [New-Item], ArgumentException
[00:21:48] + FullyQualifiedErrorId : CreateDirectoryArgumentError,Microsoft.PowerShell.Commands.NewItemCommand

[00:22:22]   Describing Invoke-Item basic tests
[00:22:22] 
[00:22:22]     Context Invoke a text file on Unix
[00:22:22] !!
[00:22:22] Get-Process : Cannot find a process with the name "notepad". Verify the process name and call the cmdlet again.
[00:22:22] At C:\projects\powershell\test\powershell\Modules\Microsoft.PowerShell.Utility\Invoke-Item.Tests.ps1:70 char:17
[00:22:22] +                 Get-Process -Name $notepadProcessName | Stop-Process  ...
[00:22:22] +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[00:22:22] + CategoryInfo          : ObjectNotFound: (notepad:String) [Get-Process], ProcessCommandException
[00:22:22] + FullyQualifiedErrorId : NoProcessFoundForGivenName,Microsoft.PowerShell.Commands.GetProcessCommand

[00:37:57] new file {$(env.ProductSourcePath)\Microsoft.Win32.SystemEvents.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.CodeDom.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.ComponentModel.Composition.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Configuration.ConfigurationManager.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Data.DataSetExtensions.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Data.Odbc.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Diagnostics.PerformanceCounter.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.AccountManagement.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.Protocols.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Drawing.Common.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.IO.Ports.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Management.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Runtime.Caching.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Security.Cryptography.ProtectedData.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Security.Cryptography.Xml.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.ServiceModel.Syndication.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57]

SteveL-MSFT · 2018-06-01T20:33:08Z

Ok, figured out the problem with System.Text.Encoding.CodePages.dll. The nupkg doesn't contain unix runtime only Windows. Older version has both. Created https://github.com/dotnet/corefx/issues/30059

Moving back to v4.3.0 fixes that issue. Will update files.wxs and submit new commit. (v4.4.0 also doesn't have unix runtime)

revert ref to System.Text.Encoding.CodePages to 4.3.0 as newer versions don't have Unix runtime updated files.wxs

revert CodePages back to 4.5.0-rc1

daxian-dbw · 2018-06-02T00:05:45Z

PSXunit tests passed on my machine (consistently failing previously) after I update the xunit packages. So push a change to see if that also fixes the issue in CI.

SteveL-MSFT · 2018-06-02T01:16:42Z

AppVeyor failure is due to recent PR to migrate the install docs, the aka.ms link now returned 404. I updated the aka.ms link so restarting AppVeyor.

daxian-dbw

LGTM

iSazonov

Leave a comment

iSazonov · 2018-06-02T09:19:30Z

src/System.Management.Automation/System.Management.Automation.csproj

+    <PackageReference Include="System.Text.Encoding.CodePages" Version="4.5.0" />
    <!-- the following package(s) are from the powershell org -->
    <PackageReference Include="Microsoft.Management.Infrastructure" Version="1.0.0-alpha*" />
    <PackageReference Include="PowerShell.Core.Instrumentation" Version="6.0.0-beta*" />


Have we this APIs in WCP?

These are pkgs PowerShell Team publishes

Should we migrate to WCP assemblies if we have them on long time?

You mean include them in WCP? Probably not.

iSazonov · 2018-06-02T09:24:50Z

src/powershell-native/nativemsh/pwrshcommon/pwrshcommon.cpp

        "Microsoft.ApplicationInsights",
-        "Microsoft.CodeAnalysis",
        "Microsoft.CodeAnalysis.CSharp",
+        "Microsoft.CodeAnalysis",


Why was the order changed?

The list was generated rather than trying to insert a large number of new assemblies into the list. Sort-Object seems to prefer the period over end of string.

iSazonov · 2018-06-02T09:26:11Z

test/csharp/csharp.tests.csproj

    <!-- DotNetCliToolReference element specifies the CLI tool that the user wants to restore in the context of the project. -->
-    <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.1" />
-    <PackageReference Include="Xunit.SkippableFact" Version="1.3.3" />
+    <DotNetCliToolReference Include="dotnet-xunit" Version="2.4.0-beta.1.build3958" />


I think we should open new tracking issue to update the beta to release.
And Microsoft.VisualStudio.Web.CodeGeneration.Tools too.

Created #6977

iSazonov · 2018-06-02T09:29:32Z

tools/packaging/packaging.psm1

                'Microsoft.PowerShell.Commands.Management' {
                    $deps.Add([tuple]::Create([tuple]::Create('id', 'Microsoft.PowerShell.Security'), [tuple]::Create('version', $PackageVersion))) > $null
-                    $deps.Add([tuple]::Create([tuple]::Create('id', 'System.ServiceProcess.ServiceController'), [tuple]::Create('version', '4.4.1'))) > $null
+                    $deps.Add([tuple]::Create([tuple]::Create('id', 'System.ServiceProcess.ServiceController'), [tuple]::Create('version', '4.5.0'))) > $null


Not related the PR - why we set the versions manually and do not take from csproj files?

iSazonov · 2018-06-02T09:32:33Z

assets/files.wxs

        <DirectoryRef Id="$(var.ProductDirectoryName)">
+            <Component Id="cmpCBE3857586454EB3B634B118E8EBD1D1" Guid="{CBE38575-8645-4EB3-B634-B118E8EBD1D1}">
+                <File Id="filCBE3857586454EB3B634B118E8EBD1D1" KeyPath="yes" Source="$(env.ProductSourcePath)\Microsoft.Win32.SystemEvents.dll" />
+            </Component>


I lost understanding - why do we update the file manually? Can we generate it automatically?

The WiX tools will break patching. When you build, I added errors to tell you what to add or remove, but didn't fully automate the update. I updated the issue with the requirements of the update.

iSazonov · 2018-06-02T18:10:37Z

Newtonsoft.Json latest version is 11.0.1
https://github.com/JamesNK/Newtonsoft.Json/releases

NJsonSchema latest version is 9.10.50
https://www.nuget.org/packages/NJsonSchema/

update Newtonsoft.Json and NJsonSchema to latest versions

add Windows Compat Pack 2.0.0 to PSCore6

7bb7c0b

SteveL-MSFT requested review from TravisEz13, adityapatwardhan and daxian-dbw as code owners May 30, 2018 16:30

SteveL-MSFT assigned daxian-dbw May 30, 2018

daxian-dbw reviewed May 30, 2018

View reviewed changes

moved ref of WCP to SDK

f4b146f

updated all refs of 4.5.0-RC1 to 4.5.0 final

update dotnetcore to 2.1.300 RTM

24c5bc8

SteveL-MSFT changed the title ~~Add Windows Compat Pack 2.0.0 to PSCore6~~ Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 May 30, 2018

[feature]

625f106

update cache names for CI update files.wxs to reflect new versions

[feature]

ce61eb9

update pwrshplugin.dll trusted assembly list

SteveL-MSFT force-pushed the add-wcp branch from 5bfdb3c to ce61eb9 Compare May 31, 2018 21:25

daxian-dbw reviewed May 31, 2018

View reviewed changes

[feature]

0ee8d62

address Dongbo's feedback

[feature]

fdf87e4

Removed unncessary dep ref for psxunit on System.Text.Encoding.CodePages.dll Updated packaging nupkg dep assembly versions

[feature]

6c0fa8b

revert ref to System.Text.Encoding.CodePages to 4.3.0 as newer versions don't have Unix runtime updated files.wxs

SteveL-MSFT force-pushed the add-wcp branch from e25f2f2 to 6c0fa8b Compare June 1, 2018 21:07

Steve Lee (POWERSHELL) and others added 2 commits June 1, 2018 15:13

[feature]

39628b7

revert CodePages back to 4.5.0-rc1

[Feature] Update xunit packages

40d3425

daxian-dbw approved these changes Jun 2, 2018

View reviewed changes

iSazonov approved these changes Jun 2, 2018

View reviewed changes

SteveL-MSFT mentioned this pull request Jun 2, 2018

Update PackageReferences to GA releases #6977

Closed

[feature]

ceb25f0

update Newtonsoft.Json and NJsonSchema to latest versions

SteveL-MSFT force-pushed the add-wcp branch from f5e1552 to ceb25f0 Compare June 2, 2018 20:20

daxian-dbw merged commit 5dc7a6c into PowerShell:master Jun 4, 2018

SteveL-MSFT deleted the add-wcp branch June 5, 2018 01:15

mi-hol mentioned this pull request Jun 20, 2018

Question: has it been tested with PowerShell Core 6.0.0? tomohulk/WinSCP#65

Closed

Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 #6958

Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 #6958

Uh oh!

Conversation

SteveL-MSFT commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daxian-dbw commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markekraus commented May 30, 2018

Uh oh!

bergmeister commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov commented May 31, 2018

Uh oh!

iSazonov commented May 31, 2018

Uh oh!

iSazonov commented May 31, 2018

Uh oh!

SteveL-MSFT commented May 31, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteveL-MSFT commented Jun 1, 2018

Uh oh!

iSazonov commented Jun 1, 2018

Uh oh!

SteveL-MSFT commented Jun 1, 2018

Uh oh!

daxian-dbw commented Jun 1, 2018

Uh oh!

iSazonov commented Jun 1, 2018

Uh oh!

SteveL-MSFT commented Jun 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daxian-dbw commented Jun 2, 2018

Uh oh!

SteveL-MSFT commented Jun 2, 2018

Uh oh!

daxian-dbw left a comment

Choose a reason for hiding this comment

Uh oh!

iSazonov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iSazonov Jun 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteveL-MSFT commented May 30, 2018 •

edited

Loading

daxian-dbw commented May 30, 2018 •

edited

Loading

bergmeister commented May 30, 2018 •

edited

Loading

SteveL-MSFT commented Jun 1, 2018 •

edited

Loading

iSazonov Jun 3, 2018 •

edited

Loading