You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Makefile-based Scanner integration test flow is very cumbersome to run locally #756
There is no longer a general integration-test folder /secureCodeBox/tests/integration/-> all hook related integrationtests are moved
Check AMASS
Check angularjs-csti-scanner
Check cmseek
Check git-repo-scanner
Check gitleaks
Check kube-hunter
...
Long read
At the moment, running integration tests locally is fairly painful. The goal is that a single make integration-tests in a scanner / hook directory should Just Work, but it very much does not at the moment.
In the worst case (modifications to the operator, no previous compiled state), in my experience the following is the minimum set of commands to run integration tests for a scanner (it may be that there is a simpler way, but I have not found it):
# If you also need to reset your cluster:# kind delete cluster# kind create cluster# Prepare operatorcd operator
make docker-build
make docker-export
make kind-import
rm operator.tar lurker.tar # clean up
make helm-deploy
# Prepare parser and hook SDK as dependency for the parser/hook# In case of a scannercd ../parser-sdk/nodejs/
make docker-build-sdk
# In case of a hookcd ../../hook-sdk/nodejs/
make docker-build-sdk
# Run the actual integration tests, with the example of semgrepcd ../../scanners/semgrep/
make docker-build
make docker-export
make kind-import
rm parser-semgrep.tar scanner-semgrep.tar # clean up
make deploy
make deploy-test-deps
# Ideally, the following is the only line I would need for all of this, maybe excluding the operator
make integration-tests
Some of this can probably be solved by settings dependencies inside the Makefile, but this may then require changes to the CI pipeline as well to avoid duplicate work being done there.
In an ideal world, make integration-tests in a scanner / hook directory would:
Check if the cluster is present, and give a warning if it is not
Validate the state of the cluster (is the operator running?) and get it into a good state if it isn't already
Compile and load all dependencies for the docker build (i.e., hook / parser sdk)
Compile and load the current scanner into the cluster
Start the integration tests
I am marking this as both a bug and an enhancement because I was told that theoretically, this should already be the case with the current makefiles and it may be that the functionality was lost over time.
It however also includes unit-tests and does not build/deploy operator and sdks. SDK built is made neccesary through #712 in common.mk (by changing latest to commit sha). Not sure if this was intentional (since SDK doesn't change often, if ever).
changed the title [-]Makefile-based integration test flow is very cumbersome to run locally[/-][+]Makefile-based Scanner integration test flow is very cumbersome to run locally[/+]on Mar 9, 2022
Activity
EndPositive commentedon Oct 22, 2021
Some more context:
A larger target already exists in
common.mk:test: | clean-integration-tests unit-tests docker-build docker-export kind-import deploy deploy-test-deps integration-testsIt however also includes unit-tests and does not build/deploy operator and sdks. SDK built is made neccesary through #712 in
common.mk(by changing latest to commit sha). Not sure if this was intentional (since SDK doesn't change often, if ever).[-]Makefile-based integration test flow is very cumbersome to run locally[/-][+]Makefile-based Scanner integration test flow is very cumbersome to run locally[/+]