Skip to content

Make xUnit test run sequentially to rule out race conditions around 'powershell.config.json' #8945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daxian-dbw merged 4 commits into from
Feb 22, 2019
Merged

Make xUnit test run sequentially to rule out race conditions around 'powershell.config.json' #8945

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d3715b0
Make xUnit test run sequentially to rule out race conditions around '…
daxian-dbw Feb 21, 2019
abde6d2
Address Aditya's comment
daxian-dbw Feb 21, 2019
82f44a3
Fix the parameter name in ci.psm1
daxian-dbw Feb 21, 2019
7a2833f
Update windows.yml to use the new unit test result file name
daxian-dbw Feb 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .vsts-ci/windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ jobs:

- powershell: |
Import-Module .\tools\ci.psm1
$ParallelXUnitTestResultsFile = "$(System.ArtifactsDirectory)\xunit\ParallelXUnitTestResults.xml"
$xUnitTestResultsFile = "$(System.ArtifactsDirectory)\xunit\xUnitTestResults.xml"
Copy link
Member

@TravisEz13 TravisEz13 Feb 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do something smarter here. Like:

  1. Get all items in the xunit folder
  2. run Test-XUnitTestResults on each item found
  3. or throw an exception if no items were found

Copy link
Member

@TravisEz13 TravisEz13 Feb 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although, this is not changed frequently. So, this is not blocking this PR.


Test-XUnitTestResults -TestResultsFile $ParallelXUnitTestResultsFile
Test-XUnitTestResults -TestResultsFile $xUnitTestResultsFile
displayName: Test
condition: succeeded()
22 changes: 7 additions & 15 deletions build.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1374,7 +1374,7 @@ function Test-PSPesterResults

function Start-PSxUnit {
[CmdletBinding()]param(
[string] $ParallelTestResultsFile = "ParallelXUnitResults.xml"
[string] $xUnitTestResultsFile = "xUnitResults.xml"
)

# Add .NET CLI tools to PATH
Expand Down Expand Up @@ -1420,23 +1420,15 @@ function Start-PSxUnit {
}
}

dotnet build --configuration $Options.configuration

if (Test-Path $ParallelTestResultsFile) {
Remove-Item $ParallelTestResultsFile -Force -ErrorAction SilentlyContinue
if (Test-Path $xUnitTestResultsFile) {
Remove-Item $xUnitTestResultsFile -Force -ErrorAction SilentlyContinue
}

# we are having intermittent issues on macOS with these tests failing.
# VSTS has suggested forcing them to be sequential
if($env:TF_BUILD -and $IsMacOS)
{
Write-Log 'Forcing parallel xunit tests to run sequentially.'
dotnet test -p:ParallelizeTestCollections=false --configuration $Options.configuration --no-restore --no-build --test-adapter-path:. "--logger:xunit;LogFilePath=$ParallelTestResultsFile"
} else {
dotnet test --configuration $Options.configuration --no-restore --no-build --test-adapter-path:. "--logger:xunit;LogFilePath=$ParallelTestResultsFile"
}
# We run the xUnit tests sequentially to avoid race conditions caused by manipulating the config.json file.
# xUnit tests run in parallel by default. To make them run sequentially, we need to define the 'xunit.runner.json' file.
dotnet test --configuration $Options.configuration --test-adapter-path:. "--logger:xunit;LogFilePath=$xUnitTestResultsFile"

Publish-TestResults -Path $ParallelTestResultsFile -Type 'XUnit' -Title 'Xunit Parallel'
Publish-TestResults -Path $xUnitTestResultsFile -Type 'XUnit' -Title 'Xunit Sequential'
}
finally {
Pop-Location
Expand Down
8 changes: 8 additions & 0 deletions test/xUnit/xUnit.tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,12 @@
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />
<PackageReference Include="XunitXml.TestLogger" Version="2.1.26" />
</ItemGroup>

<ItemGroup>
<Content Include="xunit.runner.json">
<Link>xunit.runner.json</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
</Content>
</ItemGroup>
</Project>
3 changes: 3 additions & 0 deletions test/xUnit/xunit.runner.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"parallelizeTestCollections": false
}
16 changes: 8 additions & 8 deletions tools/ci.psm1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,17 +288,17 @@ function Invoke-CIxUnit
throw "CoreCLR pwsh.exe was not built"
}

$ParallelXUnitTestResultsFile = "$pwd\ParallelXUnitTestResults.xml"
$xUnitTestResultsFile = "$pwd\xUnitTestResults.xml"

Start-PSxUnit -ParallelTestResultsFile $ParallelXUnitTestResultsFile
Start-PSxUnit -xUnitTestResultsFile $xUnitTestResultsFile
Write-Host -ForegroundColor Green 'Uploading PSxUnit test results'
Update-TestResults -resultsFile $ParallelXUnitTestResultsFile
Push-Artifact -Path $ParallelXUnitTestResultsFile -name xunit
Update-TestResults -resultsFile $xUnitTestResultsFile
Push-Artifact -Path $xUnitTestResultsFile -name xunit

if(!$SkipFailing.IsPresent)
{
# Fail the build, if tests failed
Test-XUnitTestResults -TestResultsFile $ParallelXUnitTestResultsFile
Test-XUnitTestResults -TestResultsFile $xUnitTestResultsFile
}
}

Expand Down Expand Up @@ -768,10 +768,10 @@ function Invoke-LinuxTests
}

try {
$ParallelXUnitTestResultsFile = "$pwd/ParallelXUnitTestResults.xml"
Start-PSxUnit -ParallelTestResultsFile $ParallelXUnitTestResultsFile
$xUnitTestResultsFile = "$pwd/xUnitTestResults.xml"
Start-PSxUnit -xUnitTestResultsFile $xUnitTestResultsFile
# If there are failures, Test-XUnitTestResults throws
Test-XUnitTestResults -TestResultsFile $ParallelXUnitTestResultsFile
Test-XUnitTestResults -TestResultsFile $xUnitTestResultsFile
} catch {
$result = "FAIL"
if (!$resultError)
Expand Down