Skip to content

Add more integration tests for the runner #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robertbrignull merged 4 commits into from
Sep 10, 2020
Merged

Add more integration tests for the runner #175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e4b0068
add more integration tests for the runner
robertbrignull Sep 8, 2020
67246cd
Update integration-testing.yml
robertbrignull Sep 9, 2020
d82e34b
Merge branch 'main' into runner-integration-tests
robertbrignull Sep 9, 2020
e608367
Merge branch 'main' into runner-integration-tests
robertbrignull Sep 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
294 changes: 294 additions & 0 deletions .github/workflows/integration-testing.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,297 @@ jobs:
- uses: ./../action/analyze
env:
TEST_MODE: true

runner-analyze-javascript-ubuntu:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Run init
run: |
# Pass --config-file here, but not for other jobs in this workflow.
# This means we're testing the config file parsing in the runner
# but not slowing down all jobs unnecessarily as it doesn't add much
# testing the parsing on different operating systems and languages.
runner/dist/codeql-runner-linux init --repository $GITHUB_REPOSITORY --languages javascript --config-file ./.github/codeql/codeql-config.yml --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Run analyze
run: |
runner/dist/codeql-runner-linux analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-javascript-windows:
runs-on: windows-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Run init
run: |
runner/dist/codeql-runner-win.exe init --repository $Env:GITHUB_REPOSITORY --languages javascript --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}
Copy link
Contributor

@sampart sampart Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why you don't pass the config file here, but you do for the Linux test?

Copy link
Contributor

@sampart sampart Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the other tests further down also don't use it. I imagine there's a reason why it's in one but not others, but I just wanted to check.

Copy link
Contributor Author

@robertbrignull robertbrignull Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's unintentional. I think it's because when I started I was going to upload the results so I wanted to config to mirror our existing codeql.yml workflow, but then I switched mid-way to not uploading.

Thinking about it, maybe the best compromise is to leave it as it is so the ubuntu javascript test has the config-file parameter but not the rest. Not sure it matters too much. It'll make that one job a bit slower as we'll be running more queries, but it means we're testing the config file parsing for the runner. Testing the parsing on multiple operating systems doesn't really add much. I'll add a comment explaining this.


- name: Run analyze
run: |
runner/dist/codeql-runner-win.exe analyze --repository $Env:GITHUB_REPOSITORY --commit $Env:GITHUB_SHA --ref $Env:GITHUB_REF --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-javascript-macos:
runs-on: macos-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Run init
run: |
runner/dist/codeql-runner-macos init --repository $GITHUB_REPOSITORY --languages javascript --config-file ./.github/codeql/codeql-config.yml --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Run analyze
run: |
runner/dist/codeql-runner-macos analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-csharp-ubuntu:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-linux init --repository $GITHUB_REPOSITORY --languages csharp --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
run: |
. ./codeql-runner/codeql-env.sh
dotnet build

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-linux analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-csharp-windows:
runs-on: windows-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-win.exe init --repository $Env:GITHUB_REPOSITORY --languages csharp --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
shell: powershell
run: |
cat ./codeql-runner/codeql-env.sh | Invoke-Expression
dotnet build

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-win.exe analyze --repository $Env:GITHUB_REPOSITORY --commit $Env:GITHUB_SHA --ref $Env:GITHUB_REF --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-csharp-macos:
runs-on: macos-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-macos init --repository $GITHUB_REPOSITORY --languages csharp --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
shell: bash
run: |
. ./codeql-runner/codeql-env.sh
dotnet build

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-macos analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true


runner-analyze-csharp-autobuild-ubuntu:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-linux init --repository $GITHUB_REPOSITORY --languages csharp --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
run: |
../action/runner/dist/codeql-runner-linux autobuild

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-linux analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-csharp-autobuild-windows:
runs-on: windows-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-win.exe init --repository $Env:GITHUB_REPOSITORY --languages csharp --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
shell: powershell
run: |
../action/runner/dist/codeql-runner-win.exe autobuild

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-win.exe analyze --repository $Env:GITHUB_REPOSITORY --commit $Env:GITHUB_SHA --ref $Env:GITHUB_REF --github-url $Env:GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-analyze-csharp-autobuild-macos:
runs-on: macos-latest

steps:
- uses: actions/checkout@v2

- name: Move codeql-action
shell: bash
run: |
mkdir ../action
mv * .github ../action/
mv ../action/tests/multi-language-repo/{*,.github} .

- name: Build runner
run: |
cd ../action/runner
npm install
npm run build-runner

- name: Run init
run: |
../action/runner/dist/codeql-runner-macos init --repository $GITHUB_REPOSITORY --languages csharp --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}

- name: Build code
shell: bash
run: |
../action/runner/dist/codeql-runner-macos autobuild

- name: Run analyze
run: |
../action/runner/dist/codeql-runner-macos analyze --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
env:
TEST_MODE: true

runner-upload-sarif:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2

- name: Build runner
run: |
cd runner
npm install
npm run build-runner

- name: Upload with runner
run: |
# Deliberately don't use TEST_MODE here. This is specifically testing
# the compatibility with the API.
runner/dist/codeql-runner-linux upload --sarif-file src/testdata/empty-sarif.sarif --repository $GITHUB_REPOSITORY --commit $GITHUB_SHA --ref $GITHUB_REF --github-url $GITHUB_SERVER_URL --github-auth ${{ github.token }}
26 changes: 0 additions & 26 deletions .github/workflows/runner.yml
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions tests/multi-language-repo/csharp.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<RootNamespace>multi_language_test</RootNamespace>
<DefaultItemExcludes>$(DefaultItemExcludes);codeql-runner/**</DefaultItemExcludes>
</PropertyGroup>

</Project>