Skip to content

For reliability, start using pester PassThru object to verify tests passed #4644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daxian-dbw merged 4 commits into from
Aug 26, 2017
Merged

For reliability, start using pester PassThru object to verify tests passed #4644

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d0ddf5d
Add ability to verify a test run using the -passthru object from pester
TravisEz13 Aug 22, 2017
3773283
update travis-ci to use -passthru object to verify test results
TravisEz13 Aug 22, 2017
cc04525
Address PR feedback
TravisEz13 Aug 22, 2017
996c4af
call show-pspestererror with a named parameter
TravisEz13 Aug 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 95 additions & 25 deletions build.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -802,7 +802,7 @@ function Publish-PSTestTools {
}

function Start-PSPester {
[CmdletBinding()]
[CmdletBinding(DefaultParameterSetName='default')]
param(
[string]$OutputFormat = "NUnitXml",
[string]$OutputFile = "pester-tests.xml",
Expand All @@ -813,8 +813,10 @@ function Start-PSPester {
[string]$binDir = (Split-Path (New-PSOptions).Output),
[string]$powershell = (Join-Path $binDir 'powershell'),
[string]$Pester = ([IO.Path]::Combine($binDir, "Modules", "Pester")),
[Parameter(ParameterSetName='Unelevate',Mandatory=$true)]
Copy link

@anmenaga anmenaga Aug 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this added?

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 22, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unelevate cannot be used with PassThru. Just making sure no one tries. PassThru didn't even work anymore.

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll file an issue to move the Unelevated tests to use the same mechanism to verify that the tests passed, which should get rid of this.

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File #4648

[switch]$Unelevate,
[switch]$Quiet,
[Parameter(ParameterSetName='PassThru',Mandatory=$true)]
[switch]$PassThru
)

Expand Down Expand Up @@ -845,7 +847,7 @@ function Start-PSPester {
}

Write-Verbose "Running pester tests at '$path' with tag '$($Tag -join ''', ''')' and ExcludeTag '$($ExcludeTag -join ''', ''')'" -Verbose
Publish-PSTestTools
Publish-PSTestTools | ForEach-Object {Write-Host $_}

# All concatenated commands/arguments are suffixed with the delimiter (space)
$Command = ""
Expand Down Expand Up @@ -920,7 +922,24 @@ function Start-PSPester {
}
else
{
& $powershell -noprofile -c $Command
if ($PassThru.IsPresent)
{
$passThruFile = [System.IO.Path]::GetTempFileName()
try
{
$Command += "|Export-Clixml -Path '$passThruFile' -Force"
Start-NativeExecution -sb {& $powershell -noprofile -c $Command} | ForEach-Object { Write-Host $_}
Copy link
Member

@daxian-dbw daxian-dbw Aug 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since all output from Invoke-Pester will be captured by Export-Clixml, what will be passed to Foreach-Object?

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 23, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the PassThru object is captured by Export-Clixml... everything else is done by a write host, but when we run another PowerShell process that gets turned into an output. Thus, the complexity ( I have to turn it back into a write-host)

Import-Clixml -Path $passThruFile | Where-Object {$_.TotalCount -is [Int32]}
}
finally
{
Remove-Item $passThruFile -ErrorAction SilentlyContinue
}
}
else
{
Start-NativeExecution -sb {& $powershell -noprofile -c $Command}
}
}
} finally {
$env:PSModulePath = $originalModulePath
Expand Down Expand Up @@ -952,46 +971,97 @@ function script:Start-UnelevatedProcess

function Show-PSPesterError
{
param ( [Xml.XmlElement]$testFailure )
logerror ("Description: " + $testFailure.description)
logerror ("Name: " + $testFailure.name)
[CmdletBinding(DefaultParameterSetName='xml')]
param (
[Parameter(ParameterSetName='xml',Mandatory)]
[Xml.XmlElement]$testFailure,
[Parameter(ParameterSetName='object',Mandatory)]
[PSCustomObject]$testFailureObject
)

if ($PSCmdLet.ParameterSetName -eq 'xml')
{
$description = $testFailure.description
$name = $testFailure.name
$message = $testFailure.failure.message
$stackTrace = $testFailure.failure."stack-trace"
}
elseif ($PSCmdLet.ParameterSetName -eq 'object')
{
$description = $testFailureObject.Describe + '/' + $testFailureObject.Context
$name = $testFailureObject.Name
$message = $testFailureObject.FailureMessage
$stackTrace = $testFailureObject.StackTrace
}
else
{
throw 'Unknown Show-PSPester parameter set'
}

logerror ("Description: " + $description)
logerror ("Name: " + $name)
logerror "message:"
logerror $testFailure.failure.message
logerror $message
logerror "stack-trace:"
logerror $testFailure.failure."stack-trace"
logerror $stackTrace

}

#
# Read the test result file and
# Throw if a test failed
function Test-PSPesterResults
{
[CmdletBinding(DefaultParameterSetName='file')]
param(
[Parameter(ParameterSetName='file')]
[string]$TestResultsFile = "pester-tests.xml",
[string]$TestArea = 'test/powershell'
)
[Parameter(ParameterSetName='file')]
[string]$TestArea = 'test/powershell',
[Parameter(ParameterSetName='PesterPassThruObject',Mandatory)]
[pscustomobject] $ResultObject
)

if(!(Test-Path $TestResultsFile))
if($PSCmdLet.ParameterSetName -eq 'file')
{
throw "Test result file '$testResultsFile' not found for $TestArea."
}
if(!(Test-Path $TestResultsFile))
{
throw "Test result file '$testResultsFile' not found for $TestArea."
}

$x = [xml](Get-Content -raw $testResultsFile)
if ([int]$x.'test-results'.failures -gt 0)
{
logerror "TEST FAILURES"
# switch between methods, SelectNode is not available on dotnet core
if ( "System.Xml.XmlDocumentXPathExtensions" -as [Type] ) {
$failures = [System.Xml.XmlDocumentXPathExtensions]::SelectNodes($x."test-results",'.//test-case[@result = "Failure"]')
$x = [xml](Get-Content -raw $testResultsFile)
if ([int]$x.'test-results'.failures -gt 0)
{
logerror "TEST FAILURES"
# switch between methods, SelectNode is not available on dotnet core
if ( "System.Xml.XmlDocumentXPathExtensions" -as [Type] )
{
$failures = [System.Xml.XmlDocumentXPathExtensions]::SelectNodes($x."test-results",'.//test-case[@result = "Failure"]')
}
else
{
$failures = $x.SelectNodes('.//test-case[@result = "Failure"]')
}
foreach ( $testfail in $failures )
{
Show-PSPesterError -testFailure $testfail
}
throw "$($x.'test-results'.failures) tests in $TestArea failed"
}
else {
$failures = $x.SelectNodes('.//test-case[@result = "Failure"]')
}
elseif ($PSCmdLet.ParameterSetName -eq 'PesterPassThruObject')
{
if ($ResultObject.TotalCount -le 0)
{
logerror 'NO TESTS RUN'
}
foreach ( $testfail in $failures )
elseif ($ResultObject.FailedCount -gt 0)
{
Show-PSPesterError $testfail
logerror 'TEST FAILURES'
$ResultObject.TestResult | Where-Object {$_.Passed -eq $false} | ForEach-Object {
Show-PSPesterError -testFailureObject $_
}
}
throw "$($x.'test-results'.failures) tests in $TestArea failed"
}
}

Expand Down
9 changes: 6 additions & 3 deletions tools/travis.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,10 @@ else
$ProgressPreference = $originalProgressPreference
}

$pesterParam = @{ 'binDir' = $output }
$pesterParam = @{
'binDir' = $output
'PassThru' = $true
}
Copy link
Member

@daxian-dbw daxian-dbw Aug 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this issue only happen to Travis CI? I assume it doesn't happen in AppVeyor as AppVeyor.psm1 is not chnaged to use the new parameter -PassThru.

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the use of -Unelevate

Copy link
Member Author

@TravisEz13 TravisEz13 Aug 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anmenaga already asked a question, why I marked these two options as mutually exclusive. I plan on making the code paths more similar after this gets merged.


if ($isFullBuild) {
$pesterParam['Tag'] = @('CI','Feature','Scenario')
Expand All @@ -165,7 +168,7 @@ else
Remove-Item -force ${telemetrySemaphoreFilepath}
}

Start-PSPester @pesterParam
$pesterPassThruObject = Start-PSPester @pesterParam

if (-not $isPr) {
# Run 'CrossGen' for push commit, so that we can generate package.
Expand Down Expand Up @@ -197,7 +200,7 @@ else

try {
# this throws if there was an error
Test-PSPesterResults
Test-PSPesterResults -ResultObject $pesterPassThruObject
$result = "PASS"
}
catch {
Expand Down