Skip to content

Allow packaging from a zip package to allow for signing #5418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daxian-dbw merged 8 commits into from
Nov 13, 2017
Merged

Allow packaging from a zip package to allow for signing #5418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7d7321e
ignore the expanded signed build
TravisEz13 Nov 10, 2017
bba7d86
Refactor PSModuleRestore so it can be run for a signed build
TravisEz13 Nov 10, 2017
0c1e2a3
update to support packaging signed build
TravisEz13 Nov 10, 2017
2bf41fb
Add ability to expand signed build for packaging
TravisEz13 Nov 10, 2017
b3df229
add signed build definition
TravisEz13 Nov 10, 2017
bf2340b
update to support building a signed build
TravisEz13 Nov 10, 2017
94ca4c3
export New-PSSignedBuildZip
TravisEz13 Nov 10, 2017
63ea065
address PR feedback
TravisEz13 Nov 12, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 41 additions & 21 deletions build.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,23 +578,35 @@ Fix steps:

# download modules from powershell gallery.
# - PowerShellGet, PackageManagement, Microsoft.PowerShell.Archive
if ($PSModuleRestore) {
$ProgressPreference = "SilentlyContinue"
log "Restore PowerShell modules to $publishPath"
if($PSModuleRestore)
{
Restore-PSModuleToBuild -PublishPath $publishPath
}
}

$modulesDir = Join-Path -Path $publishPath -ChildPath "Modules"
function Restore-PSModuleToBuild
{
param(
[Parameter(Mandatory)]
[string]
$PublishPath
)

# Restore modules from myget feed
Restore-PSModule -Destination $modulesDir -Name @(
# PowerShellGet depends on PackageManagement module, so PackageManagement module will be installed with the PowerShellGet module.
'PowerShellGet'
)
$ProgressPreference = "SilentlyContinue"
log "Restore PowerShell modules to $publishPath"

# Restore modules from powershellgallery feed
Restore-PSModule -Destination $modulesDir -Name @(
'Microsoft.PowerShell.Archive'
) -SourceLocation "https://www.powershellgallery.com/api/v2/"
}
$modulesDir = Join-Path -Path $publishPath -ChildPath "Modules"

# Restore modules from myget feed
Restore-PSModule -Destination $modulesDir -Name @(
# PowerShellGet depends on PackageManagement module, so PackageManagement module will be installed with the PowerShellGet module.
'PowerShellGet'
)

# Restore modules from powershellgallery feed
Restore-PSModule -Destination $modulesDir -Name @(
'Microsoft.PowerShell.Archive'
) -SourceLocation "https://www.powershellgallery.com/api/v2/"
}

function Compress-TestContent {
Expand Down Expand Up @@ -745,20 +757,28 @@ function New-PSOptions {
}

return @{ RootInfo = [PSCustomObject]$RootInfo
Top = $Top;
Configuration = $Configuration;
Framework = $Framework;
Runtime = $Runtime;
Output = $Output;
CrossGen = $CrossGen
PSModuleRestore = $PSModuleRestore }
Top = $Top
Configuration = $Configuration
Framework = $Framework
Runtime = $Runtime
Output = $Output
CrossGen = $CrossGen.IsPresent
PSModuleRestore = $PSModuleRestore.IsPresent }
}

# Get the Options of the last build
function Get-PSOptions {
return $script:Options
}

function Set-PSOptions {
param(
[PSObject]
$Options
)

$script:Options = $Options
}

function Get-PSOutput {
[CmdletBinding()]param(
Expand Down
1 change: 1 addition & 0 deletions tools/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
dotnet-install.sh
dotnet-uninstall-debian-packages.sh
ExpandedBuild/
2 changes: 1 addition & 1 deletion tools/packaging/packaging.psd1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ CompanyName="Microsoft Corporation"
Copyright="Copyright (c) Microsoft Corporation. All rights reserved."
ModuleVersion="1.0.0"
PowerShellVersion="5.0"
CmdletsToExport="Start-PSPackage"
CmdletsToExport=@("Start-PSPackage",'New-PSSignedBuildZip')
RootModule="packaging.psm1"
RequiredModules = @("build")
}
70 changes: 70 additions & 0 deletions tools/packaging/packaging.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ function Start-PSPackage {
# Zip symbols.zip to the root package
$zipSource = Join-Path $symbolsSource -ChildPath '*'
$zipPath = Join-Path -Path $Source -ChildPath 'symbols.zip'
$Script:Options | ConvertTo-Json -Depth 3 | Out-File -Encoding utf8 -FilePath (Join-Path -Path $source -ChildPath 'psoptions.json')
Copy link
Member

@daxian-dbw daxian-dbw Nov 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Be careful about the entries PSModuleRestore and CrossGen when convert to/from json. Currnetly the values of these 2 are SwitchParameter values, which works fined with the comparison to bool. However, after convert from json, you will get a synthetic property named 'IsPresent' with a bool value, and the comparison to bool would behave differently. See the example:

> $p = New-PSOptions
> $s=  $p | ConvertTo-Json | ConvertFrom-Json
> if (-not $p.PSModuleRestore) { echo "ff" }
ff
> if (-not $s.PSModuleRestore) { echo "ff" } ## print nothing.

I suggest to not use the SwitchParameter value in the first place. Instead, use $CrossGen.IsPresent and $PSModuleRestore.IsPresent when setting the initial $options

Copy link
Member Author

@TravisEz13 TravisEz13 Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to suggested implementation

Compress-Archive -Path $zipSource -DestinationPath $zipPath
}
finally
Expand Down Expand Up @@ -425,6 +426,75 @@ function New-TempFolder

return $tempFolder
}
function New-PSSignedBuildZip
{
param(
[Parameter(Mandatory)]
[string]$BuildPath,
[Parameter(Mandatory)]
[string]$SignedFilesPath,
[Parameter(Mandatory)]
[string]$DestinationFolder,
[parameter(HelpMessage='VSTS variable to set for path to zip')]
[string]$VstsVariableName
)

# Replace unsigned binaries with signed
$signedFilesFilter = Join-Path -Path $signedFilesPath -ChildPath '*'
Get-ChildItem -path $signedFilesFilter -Recurse | Select-Object -ExpandProperty FullName | Foreach-Object -Process {
$relativePath = $_.Replace($signedFilesPath,'')
$destination = Join-Path -Path $buildPath -ChildPath $relativePath
log "replacing $destination with $_"
Copy-Item -Path $_ -Destination $destination -force
}

$name = split-path -Path $BuildPath -Leaf
$zipLocationPath = Join-Path -Path $DestinationFolder -ChildPath "$name-signed.zip"
Compress-Archive -Path $BuildPath\* -DestinationPath $zipLocationPath
Write-Host "##vso[artifact.upload containerfolder=results;artifactname=$name]$zipLocationPath"
if ($VstsVariableName)
{
# set VSTS variable with path to package files
log "Setting $VstsVariableName to $zipLocationPath"
Write-Host "##vso[task.setvariable variable=$VstsVariableName]$zipLocationPath"
}
else
{
return $zipLocationPath
}
}

function Expand-PSSignedBuild
{
param(
[Parameter(Mandatory)]
[string]$BuildZip
)

$psModulePath = Split-Path -path $PSScriptRoot
# Expand unsigned build
$buildPath = Join-Path -path $psModulePath -childpath 'ExpandedBuild'
New-Item -path $buildPath -itemtype Directory -force
Expand-Archive -path $BuildZip -destinationpath $buildPath -Force
remove-item -Path (Join-Path -Path $buildPath -ChildPath '*.zip') -Recurse

$windowsExecutablePath = (Join-Path $buildPath -ChildPath 'pwsh.exe')

Restore-PSModuleToBuild -PublishPath $buildPath

$options = Get-Content -Path (Join-Path $buildPath -ChildPath 'psoptions.json') | ConvertFrom-Json

if(Test-Path -Path $windowsExecutablePath)
{
$options.Output = $windowsExecutablePath
}
else
{
throw 'Could not find pwsh'
}

Set-PSOptions -Options $options
}

function New-UnixPackage {
[CmdletBinding(SupportsShouldProcess=$true)]
Expand Down
41 changes: 31 additions & 10 deletions tools/releaseBuild/Images/microsoft_powershell_windowsservercore/PowerShellPackage.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,29 @@
[cmdletbinding()]
[cmdletbinding(DefaultParameterSetName='default')]
# PowerShell Script to clone, build and package PowerShell from specified fork and branch
param (
[string] $fork = 'powershell',

[string] $branch = 'master',

[string] $location = "$pwd\powershell",

[string] $destination = "$env:WORKSPACE",
[ValidateSet("win7-x64", "win81-x64", "win10-x64", "win7-x86")]

[ValidateSet("win7-x64", "win7-x86")]
[string]$Runtime = 'win10-x64',

[switch] $Wait,

[ValidatePattern("^v\d+\.\d+\.\d+(-\w+\.\d+)?$")]
[ValidateNotNullOrEmpty()]
[string]$ReleaseTag,
[switch] $Symbols

[Parameter(Mandatory,ParameterSetName='IncludeSymbols')]
[switch] $Symbols,

[Parameter(Mandatory,ParameterSetName='packageSigned')]
[ValidatePattern("-signed.zip$")]
[string]$BuildZip
)

$releaseTagParam = @{}
Expand Down Expand Up @@ -59,15 +71,24 @@ try{
Write-Verbose "Bootstrapping powershell build..." -verbose
Start-PSBootstrap -Force -Package

Write-Verbose "Starting powershell build for RID: $Runtime and ReleaseTag: $ReleaseTag ..." -verbose
$buildParams = @{}
$buildParams['CrossGen'] = $true
if(!$Symbols.IsPresent)
if($PSCmdlet.ParameterSetName -eq 'packageSigned')
{
$buildParams['PSModuleRestore'] = $true
Write-Verbose "Expanding signed build..." -verbose
Expand-PSSignedBuild -BuildZip $BuildZip
Remove-Item -Path $BuildZip
}
else
{
Write-Verbose "Starting powershell build for RID: $Runtime and ReleaseTag: $ReleaseTag ..." -verbose
$buildParams = @{}
$buildParams['CrossGen'] = $true
if(!$Symbols.IsPresent)
{
$buildParams['PSModuleRestore'] = $true
}

Start-PSBuild -Clean -Runtime $Runtime -Configuration Release @releaseTagParam @buildParams
Start-PSBuild -Clean -Runtime $Runtime -Configuration Release @releaseTagParam @buildParams
}

$pspackageParams = @{'Type'='msi'}
if ($Runtime -ne 'win10-x64')
Expand All @@ -80,7 +101,7 @@ try{
Write-Verbose "Starting powershell packaging(msi)..." -verbose
Start-PSPackage @pspackageParams @releaseTagParam
}
else
else
{
$pspackageParams += @{'IncludeSymbols' = $true}
}
Expand Down
28 changes: 28 additions & 0 deletions tools/releaseBuild/build.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,34 @@
"BinaryBucket": "symbols",
"BinariesExpected": 1,
"VariableForExtractedBinariesPath": "Symbols_x86"
},
{
"Name": "win7-x64-package",
"RepoDestinationPath": "C:\\PowerShell",
"BuildCommand": "C:\\PowerShellPackage.ps1 -BuildZip _RepoDestinationPath_\\_BuildPackageName_ -location _RepoDestinationPath_ -destination _DockerVolume_ -Runtime win7-x64 -ReleaseTag _ReleaseTag_",
"BuildDockerOptions": [
"-m",
"3968m"
Copy link
Member

@daxian-dbw daxian-dbw Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why do we want to limit the memory for the container?

Copy link
Member Author

@TravisEz13 TravisEz13 Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is increasing the memory in the container on many machines. I cannot add comments in JSON. but the packaging script verifies that we have increased the memory to at least 2GB.

Copy link
Member Author

@TravisEz13 TravisEz13 Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say a minimum of this value if I could but they don't have this option.

Copy link
Member

@daxian-dbw daxian-dbw Nov 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks!

],
"DockerFile": ".\\tools\\releaseBuild\\Images\\microsoft_powershell_windowsservercore\\DockerFile",
"AdditionalContextFiles" :[ ".\\tools\\releaseBuild\\Images\\microsoft_powershell_windowsservercore\\PowerShellPackage.ps1"],
"DockerImageName": "ps-winsrvcore",
"BinaryBucket": "signed",
"BinariesExpected": 2
},
{
"Name": "win7-x86-package",
"RepoDestinationPath": "C:\\PowerShell",
"BuildCommand": "C:\\PowerShellPackage.ps1 -BuildZip _RepoDestinationPath_\\_BuildPackageName_ -location _RepoDestinationPath_ -destination _DockerVolume_ -Runtime win7-x86 -ReleaseTag _ReleaseTag_",
"BuildDockerOptions": [
"-m",
"3968m"
],
"DockerFile": ".\\tools\\releaseBuild\\Images\\microsoft_powershell_windowsservercore\\DockerFile",
"AdditionalContextFiles" :[ ".\\tools\\releaseBuild\\Images\\microsoft_powershell_windowsservercore\\PowerShellPackage.ps1"],
"DockerImageName": "ps-winsrvcore",
"BinaryBucket": "signed",
"BinariesExpected": 2
}
],
"Linux": [
Expand Down
43 changes: 41 additions & 2 deletions tools/releaseBuild/vstsbuild.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
[cmdletbinding(DefaultParameterSetName='Build')]
param(
[Parameter(ParameterSetName='packageSigned')]
[Parameter(ParameterSetName='Build')]
[ValidatePattern("^v\d+\.\d+\.\d+(-\w+\.\d+)?$")]
[string]$ReleaseTag
[string]$ReleaseTag,

# full paths to files to add to container to run the build
[Parameter(Mandatory,ParameterSetName='packageSigned')]
[string]
$BuildPath,

[Parameter(Mandatory,ParameterSetName='packageSigned')]
[string]
$SignedFilesPath
)

DynamicParam {
Expand Down Expand Up @@ -37,6 +48,29 @@ Begin {
End {
$ErrorActionPreference = 'Stop'

$additionalFiles = @()
$buildPackageName = $null
# If specified, Add package file to container
if ($BuildPath)
{
Import-Module (Join-Path -path $PSScriptRoot -childpath '..\..\build.psm1')
Import-Module (Join-Path -path $PSScriptRoot -childpath '..\packaging')

# Use temp as destination if not running in VSTS
$destFolder = $env:temp
if($env:Build_ArtifactStagingDirectory)
{
# Use artifact staging if running in VSTS
$destFolder = $env:Build_ArtifactStagingDirectory
}

$BuildPackagePath = New-PSSignedBuildZip -BuildPath $BuildPath -SignedFilesPath $SignedFilesPath -DestinationFolder $destFolder
Write-Host "##vso[artifact.upload containerfolder=results;artifactname=$name-singed.zip]$BuildPackagePath"
$buildPackageName = Split-Path -Path $BuildPackagePath -Leaf
$additionalFiles += $BuildPackagePath
}


$psReleaseBranch = 'master'
$psReleaseFork = 'PowerShell'
$location = Join-Path -Path $PSScriptRoot -ChildPath 'PSRelease'
Expand Down Expand Up @@ -74,7 +108,12 @@ End {
Import-Module "$location/dockerBasedBuild" -Force
Clear-VstsTaskState

Invoke-Build -RepoPath $resolvedRepoRoot -BuildJsonPath './tools/releaseBuild/build.json' -Name $Name -Parameters $PSBoundParameters
$buildParameters = @{
ReleaseTag = $ReleaseTag
Copy link
Member

@daxian-dbw daxian-dbw Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously, you pass in $PSBoundParameters, so if -ReleaseTag is not specified, it won't be passed to Invoke-Build.
Now you always pass in ReleaseTag, and the value will be $null in case -ReleaseTag is not specified when calling this script. Will this cause any problems?

Copy link
Member Author

@TravisEz13 TravisEz13 Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new hashtable I have has ReleaseTag

Copy link
Member Author

@TravisEz13 TravisEz13 Nov 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this won't cause any issues. I've verified this previously.

BuildPackageName = $buildPackageName
}

Invoke-Build -RepoPath $resolvedRepoRoot -BuildJsonPath './tools/releaseBuild/build.json' -Name $Name -Parameters $buildParameters -AdditionalFiles $AdditionalFiles
}
catch
{
Expand Down