Skip to content

Makefile-based Scanner integration test flow is very cumbersome to run locally #756

Closed

Description

@malexmave

Acceptance Criteria:

  • In each Scanner directory (with existing integrationtests) implement an valid integration-test target: make integration-test

  • Review & Update existing documentation regarding "integration-tests": https://docs.securecodebox.io/docs/contributing/integrating-a-scanner/makefile

  • There is no longer a general integration-test folder /secureCodeBox/tests/integration/-> all hook related integrationtests are moved

  • Check AMASS

  • Check angularjs-csti-scanner

  • Check cmseek

  • Check git-repo-scanner

  • Check gitleaks

  • Check kube-hunter

  • ...

Long read

At the moment, running integration tests locally is fairly painful. The goal is that a single make integration-tests in a scanner / hook directory should Just Work, but it very much does not at the moment.

In the worst case (modifications to the operator, no previous compiled state), in my experience the following is the minimum set of commands to run integration tests for a scanner (it may be that there is a simpler way, but I have not found it):

# If you also need to reset your cluster:
# kind delete cluster
# kind create cluster

# Prepare operator
cd operator
make docker-build
make docker-export
make kind-import
rm operator.tar lurker.tar  # clean up
make helm-deploy

# Prepare parser and hook SDK as dependency for the parser/hook
# In case of a scanner
cd ../parser-sdk/nodejs/
make docker-build-sdk
# In case of a hook
cd ../../hook-sdk/nodejs/
make docker-build-sdk

# Run the actual integration tests, with the example of semgrep
cd ../../scanners/semgrep/
make docker-build
make docker-export
make kind-import
rm parser-semgrep.tar scanner-semgrep.tar  # clean up
make deploy
make deploy-test-deps
# Ideally, the following is the only line I would need for all of this, maybe excluding the operator
make integration-tests

Some of this can probably be solved by settings dependencies inside the Makefile, but this may then require changes to the CI pipeline as well to avoid duplicate work being done there.

In an ideal world, make integration-tests in a scanner / hook directory would:

  • Check if the cluster is present, and give a warning if it is not
  • Validate the state of the cluster (is the operator running?) and get it into a good state if it isn't already
  • Compile and load all dependencies for the docker build (i.e., hook / parser sdk)
  • Compile and load the current scanner into the cluster
  • Start the integration tests

I am marking this as both a bug and an enhancement because I was told that theoretically, this should already be the case with the current makefiles and it may be that the functionality was lost over time.

Activity

  1. EndPositive commented on Oct 22, 2021

    @EndPositive
    Contributor

    Some more context:

    A larger target already exists in common.mk:

    test: | clean-integration-tests unit-tests docker-build docker-export kind-import deploy deploy-test-deps integration-tests

    It however also includes unit-tests and does not build/deploy operator and sdks. SDK built is made neccesary through #712 in common.mk (by changing latest to commit sha). Not sure if this was intentional (since SDK doesn't change often, if ever).

  2. added this to the v3.9.0 milestone on Mar 9, 2022
  3. changed the title [-]Makefile-based integration test flow is very cumbersome to run locally[/-] [+]Makefile-based Scanner integration test flow is very cumbersome to run locally[/+] on Mar 9, 2022
  4. modified the milestones: v3.9.0, v3.10.0 on Mar 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBugsenhancementNew feature or requestplannedIssues we will do in the next sprint.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions