Skip to content

Update CI Workflow With New Build-Push-Action And more parallelization - #283

Merged
J12934 merged 64 commits into
secureCodeBox:mainfrom
nigthknight:update/ci
Feb 8, 2021
Merged

J12934 merged 64 commits into
secureCodeBox:mainfrom
nigthknight:update/ci

Conversation

@nigthknight

@nigthknight nigthknight commented Jan 27, 2021 •

Copy link
Copy Markdown
Member

This PR will update the CI with a newer version of the docker-build-push-action.
To make this possible I had to introduce docker-meta-action to ensure all docker images are build with the tags needed.
Additional I changed the workflow to parallelize jobs where possible
I also extracted some variables:

  • DOCKER_NAMESPACE (Namespace for the Docker images)
  • DOCKER_USERNAME (User used to push the images)
  • DOCKER_TOKEN (Access Token for the user)
  • CC_TEST_REPORTER_ID (Codeclimate Test Reporter ID)

I pinned the Scanner Versions using Dockerfile ARG
I read the Scanner Version from the corresponding Chart.yaml
I updated the Contributing Page for the new GH secrets

Before Merging:

  • Regenerate the codeclimate ID
  • Add CC_TEST_REPORTER_ID
  • Add DOCKER_NAMESPACE to the secrets
  • Add DOCKER_USERNAME to the secrets
  • Add DOCKER_TOKEN to the secrets

closes #269

I deleted everything in the CI to test a possible setup with the newer
version of the build-push-action for docker images

This Setup will build docker images with tags for:
- current branch
- current commit hash
- custom tag `ci-local`
- `lastest` tag (only if git tag is created)
- semver tag: MAJOR.MINOR.PATCH

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>

Fix Docker Repo

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
To make working on forks more easy I introduced the github secret
`DOCKER_NAMESPACE`.
This secret contains the namespace in wich the docker images will be
created.
For example for the "golden" repository the namespace is currently
`securecodebox`

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
I introduced comments dividing the CI into multiple parts

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Notice that the lines actually executing the tests are commented out.
This has to be done because my Fork does not have an id for code
coverage

The Job remains the same except for the name

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
I added a local Registry to Cache the Docker Image Build from the nikto
Repository
To use the Image from the local registry I changed the Baseimage in the
Dockerfile to localhost:5000/sullo/nikto

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
To add integration tests to the CI I had to specify the repository used
for scanners as well as the parsers. This was necessary to ensure that
the tests run on forks as well.

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
I updated the contributing page because the CI should be fork friendly.
I also added a Link to the Contributing Page in the README

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
@nigthknight
nigthknight requested review from J12934 and rfelber January 27, 2021 11:41
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Comment thread .github/workflows/ci.yaml Resolved
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
@nigthknight
nigthknight marked this pull request as draft January 28, 2021 09:10
@nigthknight
nigthknight force-pushed the update/ci branch 3 times, most recently from 18f3ab3 to 6533f38 Compare January 28, 2021 09:46
The previous solution was prone to quotationmarks wrapping the version
number. Using `yq` will resolve this

Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
Signed-off-by: Yannik Fuhrmeister <yannik.fuhrmeister@iteratec.com>
@nigthknight
nigthknight marked this pull request as ready for review January 28, 2021 12:06
@nigthknight
nigthknight requested a review from J12934 January 28, 2021 12:07
@rfelber rfelber added the ci Changes to the continuous integration setup label Feb 6, 2021
@J12934
J12934 merged commit c8e4b3e into secureCodeBox:main Feb 8, 2021
@J12934

J12934 commented Feb 9, 2021 •

Copy link
Copy Markdown
Member

After merge I've noticed two problems:

  1. The ssh-scan parser image is now build as parser-ssh_scan instead of parser-ssh-scan
  2. The teams hook image is build as teams-webhook instead of hook-teams-notification

Both problems seem to come from the problem that their images and folder names are different.
Best thing for ssh_scan is probably to give in and rename the folder to ssh-scan even thought the tools spells itself with a underscore.

With the teams webhook I'm not so sure, should we rename the image to teams-webhook or change the folder name of it to match the image?

What do you think @fuhrmeistery @rseedorff ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Changes to the continuous integration setup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate to docker/build-push-action@v2

3 participants