Skip to content

Reactivate CI regression tests for Windows + improve CI #1117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hsutter merged 5 commits into from
Jun 18, 2024
Merged

Reactivate CI regression tests for Windows + improve CI #1117

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
412b7b4
CI Aggreate generated patches
jarzec Jun 15, 2024
40b77bd
CI Reactivate Windows regression tests
jarzec Jun 15, 2024
29a268d
Ci Update C++20 test on Windows
jarzec Jun 16, 2024
ed7d5e8
CI exclude lines with Windows paths from git
jarzec Jun 17, 2024
6647266
CI Update mixed-default-arguments.cpp2 test
jarzec Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions .github/workflows/regression-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ jobs:
bash run-tests.sh -c ${{ matrix.compiler }} -s ${{ matrix.cxx_std }} -d ${{ matrix.stdlib }} -l ${{ matrix.os }}

- name: Run regression tests - Windows version
if: matrix.os == 'windows-latest'
if: startsWith(matrix.os, 'windows')
Copy link
Contributor Author

@jarzec jarzec Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Explanation] This line was not updated in a change of this workflow. I've noticed that somewhat accidentally looking at a change that should have caused the test run to fail (e.g. in here) while in fact it showed as successful.

run: |
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" && ^
git config --local core.autocrlf false && ^
Expand All @@ -101,9 +101,36 @@ jobs:
shell: cmd

- name: Upload patch
if: ${{ !cancelled() }}
if: success() || failure()
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.cxx_std }}-${{ matrix.stdlib }}.patch
path: regression-tests/${{ matrix.os }}-${{ matrix.compiler }}-${{ matrix.cxx_std }}-${{ matrix.stdlib }}.patch
if-no-files-found: ignore

aggregate-results:
needs: regression-tests
if: success() || failure()
runs-on: ubuntu-latest
steps:
- name: Download all patches
uses: actions/download-artifact@v4
with:
path: downloaded-results

- name: Prepare result files
id: prepare_files
run: |
mkdir aggregated-results
echo "Flattening file hierarchy"
find . -type f -wholename "./downloaded-results*" -exec mv {} aggregated-results \;
patch_count=$(ls aggregated-results 2>/dev/null | wc -l)
echo "patch_count=${patch_count}" >> $GITHUB_OUTPUT

- name: Upload aggregated results
if: steps.prepare_files.outputs.patch_count != '0'
uses: actions/upload-artifact@v4
with:
name: aggregated-results
path: aggregated-results
if-no-files-found: ignore
22 changes: 14 additions & 8 deletions regression-tests/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,14 @@ usage() {
# If the diff is not empty print it with the provided message
report_diff () {
file="$1"
diff_opt="$2"
error_msg="$3"
patch_file="$4"
error_msg="$2"
patch_file="$3"
# The remaining arguments will be passed to git
shift 3
diff_opts="$@"

# Compare the content with the reference value checked in git
diff_output=$(git diff "$diff_opt" -- "$file")
diff_output=$(git diff $diff_opts -- "$file")
if [[ -n "$diff_output" ]]; then
echo " $error_msg:"
echo " $file"
Expand Down Expand Up @@ -55,17 +57,21 @@ check_file () {
git add "$file"
# ... report the diff ...
report_diff "$file" \
"HEAD" \
"The $description is not tracked by git" \
"$patch_file"
"$patch_file" \
"HEAD"
# ... and remove the file from the index
git rm --cached -- "$file" > /dev/null 2>&1
else
# Compare the content with the reference value checked in git
# Lines includng Windows paths are excluded from diff
# This is necessary due to characters in those paths on GitHub runners
# that cause git diff to spuriously flag them
report_diff "$file" \
"--ignore-cr-at-eol" \
"Non-matching $description" \
"$patch_file"
"$patch_file" \
--ignore-cr-at-eol \
--ignore-matching-lines="C:\\\\"
Copy link
Contributor Author

@jarzec jarzec Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Explanation] This option is excluding lines with Windows paths. On the GitHub runners those paths may include characters that seem to cause trouble with git diff.
This is hopefully a temporary solution.

fi
}

Expand Down
4 changes: 4 additions & 0 deletions regression-tests/test-results/msvc-2022-c++20/mixed-default-arguments.cpp.execution
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
1 "test"
0 ""
1 "test"
0 ""
1 change: 1 addition & 0 deletions regression-tests/test-results/msvc-2022-c++20/mixed-default-arguments.cpp.output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mixed-default-arguments.cpp
14 changes: 8 additions & 6 deletions regression-tests/test-results/msvc-2022-c++20/pure2-assert-expected-not-null.cpp.output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
pure2-assert-expected-not-null.cpp
pure2-assert-expected-not-null.cpp2(7): error C2039: 'expected': is not a member of 'std'
predefined C++ types (compiler internal)(348): note: see declaration of 'std'
predefined C++ types (compiler internal)(347): note: see declaration of 'std'
pure2-assert-expected-not-null.cpp2(7): error C2062: type 'int' unexpected
pure2-assert-expected-not-null.cpp2(7): error C2143: syntax error: missing ';' before '{'
pure2-assert-expected-not-null.cpp2(7): error C2143: syntax error: missing ';' before '}'
pure2-assert-expected-not-null.cpp2(9): error C2065: 'ex': undeclared identifier
pure2-assert-expected-not-null.cpp2(9): error C2672: 'cpp2::impl::assert_not_null': no matching overloaded function found
..\..\..\include\cpp2util.h(632): note: could be 'decltype(auto) cpp2::impl::assert_not_null(_T0 &&)'
..\..\..\include\cpp2util.h(638): note: could be 'decltype(auto) cpp2::impl::assert_not_null(_T0 &&)'
pure2-assert-expected-not-null.cpp2(14): error C2039: 'expected': is not a member of 'std'
predefined C++ types (compiler internal)(348): note: see declaration of 'std'
predefined C++ types (compiler internal)(347): note: see declaration of 'std'
pure2-assert-expected-not-null.cpp2(14): error C2062: type 'int' unexpected
pure2-assert-expected-not-null.cpp2(14): error C2143: syntax error: missing ';' before '{'
pure2-assert-expected-not-null.cpp2(14): error C2039: 'unexpected': is not a member of 'std'
predefined C++ types (compiler internal)(348): note: see declaration of 'std'
pure2-assert-expected-not-null.cpp2(14): error C3861: 'unexpected': identifier not found
predefined C++ types (compiler internal)(347): note: see declaration of 'std'
pure2-assert-expected-not-null.cpp2(14): error C2660: 'unexpected': function does not take 1 arguments
C:\Program Files\Microsoft Visual Studio�2nterprise\VC\Tools\MSVC .40.33807\includeh.h(33): note: see declaration of 'unexpected'
Copy link
Contributor Author

@jarzec jarzec Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Explanation] This path was always indicated as different despite the test file being updated. My conjecture is that this is due to the strange characters in the path.

pure2-assert-expected-not-null.cpp2(14): note: while trying to match the argument list '(bool)'
pure2-assert-expected-not-null.cpp2(14): error C2143: syntax error: missing ';' before '}'
pure2-assert-expected-not-null.cpp2(15): error C2065: 'ex': undeclared identifier
pure2-assert-expected-not-null.cpp2(15): error C2672: 'cpp2::impl::assert_not_null': no matching overloaded function found
..\..\..\include\cpp2util.h(632): note: could be 'decltype(auto) cpp2::impl::assert_not_null(_T0 &&)'
..\..\..\include\cpp2util.h(638): note: could be 'decltype(auto) cpp2::impl::assert_not_null(_T0 &&)'