Skip to content

Optimizie CI Runtime by Parallelizing Slow Tests #2087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ilyesbdlala merged 5 commits into from
Nov 7, 2023
Merged

Optimizie CI Runtime by Parallelizing Slow Tests #2087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
df6dd98
Allow concurrent running of zap integration tests
Ilyesbdlala Nov 7, 2023
2f8e6c2
Allow concurrent running of trivy integration tests
Ilyesbdlala Nov 7, 2023
8c7347c
Include zap-advanced in the normal CI instead of nightly
Ilyesbdlala Nov 7, 2023
eeee689
Allow concurrent running of zap-advanced integration tests
Ilyesbdlala Nov 7, 2023
c037b6e
Fixed the expected number of results for zap test.
Ilyesbdlala Nov 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
187 changes: 1 addition & 186 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@ on:
- main
- v[0-9]+.x
pull_request:
schedule:
- cron: "15 2 * * *" # Nightly-Build at 2:15 AM UTC

# The CI runs on ubuntu-22.04; More info about the installed software is found here:
# https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md
Expand Down Expand Up @@ -443,6 +441,7 @@ jobs:
- whatweb
- wpscan
- zap
- zap-advanced
steps:
- name: Checkout
uses: actions/checkout@v3
Expand Down Expand Up @@ -592,190 +591,6 @@ jobs:
echo "Operator Startup Logs"
kubectl -n securecodebox-system logs deployment/securecodebox-controller-manager

# ---- Build | Scanners | Custom Scanner | Nightly Build ----

# This Section contains Scanners that are developed by the secureCodeBox project or require a nightly build
# The tag for these images will be the Semver of the release

zap-advanced:
name: "Zap-advanced integration-test"
if: ${{ github.event_name == 'schedule' }}
needs:
- sdk
- operator
- nvm-setup
- k8s-setup
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Download nvm and Node.js
uses: actions/download-artifact@v3
with:
name: nvm-node
path: ./nvm-node

- name: Extract nvm and Node.js
run: |
tar xzf nvm-node/nvm-node.tar.gz -C $HOME
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm use

- name: Download Kind
uses: actions/download-artifact@v3
with:
name: kind
path: ./kind

- name: Download Kubectl
uses: actions/download-artifact@v3
with:
name: kubectl
path: ./kubectl

- name: Download Helm
uses: actions/download-artifact@v3
with:
name: helm
path: ./helm

- name: Make binaries globally available
run: |
chmod +x ./kind/kind && sudo mv ./kind/kind /usr/local/bin/kind
chmod +x ./kubectl/kubectl && sudo mv ./kubectl/kubectl /usr/local/bin/kubectl
chmod +x ./helm/helm && sudo mv ./helm/helm /usr/local/bin/helm

- name: Verify tools
run: |
kind version
kubectl version || true
helm version

- name: Install Dependencies
working-directory: ./scanners/zap-advanced/
run: make install-deps

- name: Unit Tests
working-directory: ./scanners/zap-advanced/
run: make unit-tests

- name: Download Parser SDK Image
uses: actions/download-artifact@v3
with:
name: parser-sdk-image
path: /tmp

- name: Load Parser SDK Image
run: |
docker load --input /tmp/parser-sdk.tar
docker images | grep sdk

- name: Build Images
working-directory: ./scanners/zap-advanced/
run: make docker-build

- name: Export Docker Images
working-directory: ./scanners/zap-advanced/
run: make docker-export

- name: "Start kind cluster"
run: |
kind version
kind create cluster --wait 3m --image "$KIND_NODE_IMAGE"

- name: "Inspect kind cluster"
run: |
kubectl config current-context
kubectl get node

- name: Download Operator Image
uses: actions/download-artifact@v3
with:
name: operator-image
path: ./operator

- name: Download Lurker Image
uses: actions/download-artifact@v3
with:
name: lurker-image
path: ./operator

- name: Import Operator & Lurker Image
working-directory: ./operator
run: make kind-import

- name: Kind Import Images
working-directory: ./scanners/zap-advanced/
run: make kind-import

- name: Install Operator
working-directory: ./operator
run: |
make helm-deploy

- name: Deploy zap-advanced
working-directory: ./scanners/zap-advanced/
run: make deploy

- name: Deploy Test Dependencies
working-directory: ./scanners/zap-advanced/
run: make deploy-test-deps

- name: Start Integration Tests
working-directory: ./scanners/zap-advanced/
run: make integration-tests

# ---- Debuging Cluster on Failure ----

- name: Inspect Post Failure
if: failure()
run: |
echo "List all 'HelmCharts' in all namespaces"
helm list --all-namespaces
echo "List all 'Scans' in all namespaces"
kubectl get scans -o wide --all-namespaces
echo "List all 'Jobs' in all namespaces"
kubectl get jobs -o wide --all-namespaces
echo "List all 'Pods' in all namespaces"
kubectl get pods -o wide --all-namespaces
echo "List all 'Services' in all namespaces"
kubectl get services -o wide --all-namespaces
echo "Describe Pods in 'integration-tests' namespace"
kubectl describe pod -n integration-tests

- name: "Inspect Operator"
if: failure()
run: |
echo "Deployment in namespace 'securecodebox-system'"
kubectl -n securecodebox-system get deployments
echo "Deployment in namespace 'securecodebox-system'"
kubectl -n securecodebox-system get pods
echo "Operator Startup Logs"
kubectl -n securecodebox-system logs deployment/securecodebox-controller-manager

- name: Notify Slack Dev Channel
if: failure()
uses: slackapi/slack-github-action@v1.18.0
with:
payload: |
{
"text": "GitHub Action build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "GitHub Action build result: ${{ job.status }}\n${{ github.event.pull_request.html_url || github.event.head_commit.url }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_DEVELOPMENT_CHANNEL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

# ---- Test | Hooks ----

test-hooks:
Expand Down
10 changes: 5 additions & 5 deletions scanners/trivy/integration-tests/trivy.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ const {scan} = require("../../helpers");

jest.retryTimes(3);

test(
test.concurrent(
"trivy image scan for a vulnerable juiceshop demo target",
async () => {
const { categories, severities, count } = await scan(
Expand All @@ -26,7 +26,7 @@ test(
3 * 60 * 1000
);

test(
test.concurrent(
"trivy filesystem scan with exiting files should not fail",
async () => {
const { categories, severities, count } = await scan(
Expand Down Expand Up @@ -67,7 +67,7 @@ test(
3 * 60 * 1000
);

test(
test.concurrent(
"trivy repo scan with exiting repo should not fail",
async () => {
const { categories, severities, count } = await scan(
Expand All @@ -84,7 +84,7 @@ test(
3 * 60 * 1000
);

test(
test.concurrent(
"Invalid argument should be marked as errored",
async () => {
await expect(
Expand All @@ -99,7 +99,7 @@ test(
3 * 60 * 1000
);

test(
test.concurrent(
"trivy k8s scan should not fail",
async () => {
const { categories, severities, count } = await scan(
Expand Down
6 changes: 3 additions & 3 deletions scanners/zap-advanced/integration-tests/zap-advanced.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ const {scan} = require("../../helpers");

jest.retryTimes(3);

test(
test.concurrent(
"ZAP-advanced scan without config YAML against 'bodgeit' container should only find couple findings",
async () => {
const { count } = await scan(
Expand All @@ -22,7 +22,7 @@ test(
60 * 31 * 1000
);

test(
test.concurrent(
"ZAP-advanced scan without config YAML against 'juiceshop' should only find couple findings",
async () => {
const { count } = await scan(
Expand All @@ -38,7 +38,7 @@ test(
60 * 31 * 1000
);

test(
test.concurrent(
"ZAP-advanced scan without config YAML against 'swagger-petstore' should only find couple findings",
async () => {
const { count } = await scan(
Expand Down
18 changes: 9 additions & 9 deletions scanners/zap/integration-tests/zap.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

const {scan} = require("../../helpers");

test(
test.concurrent(
"zap automation scan without config against 'bodgeit' should only find couple findings",
async () => {
const { count } = await scan(
Expand All @@ -27,10 +27,10 @@ test(
// There must be at least one finding
expect(count).toBeGreaterThanOrEqual(1);
},
60 * 5 * 1000
60 * 8 * 1000
);

test(
test.concurrent(
"zap automation scan without config against 'swagger-petstore' should only find couple findings",
async () => {
const { count } = await scan(
Expand All @@ -53,10 +53,10 @@ test(
// There must be at least one finding
expect(count).toBeGreaterThanOrEqual(1);
},
60 * 5 * 1000
60 * 8 * 1000
);

test(
test.concurrent(
"zap automation scan against a plain nginx container should only find a couple of findings",
async () => {
const { count } = await scan(
Expand All @@ -79,10 +79,10 @@ test(

expect(count).toBeGreaterThanOrEqual(4);
},
60 * 5 * 1000
60 * 8 * 1000
);

test(
test.concurrent(
"authenticated zap automation scan with little spider time against a juice shop container should find some findings",
async () => {
const { count } = await scan(
Expand All @@ -103,9 +103,9 @@ test(
}],
);

expect(count).toBeGreaterThanOrEqual(5);
expect(count).toBeGreaterThanOrEqual(4);
},
60 * 5 * 1000
60 * 8 * 1000
);

/*test(
Expand Down