Skip to content

Allows the Dashboard importer to be built through the makefiles - #1143

Merged
J12934 merged 16 commits into
mainfrom
bugfix/fix-dashboard-pull-dockerhub
May 11, 2022
Merged

J12934 merged 16 commits into
mainfrom
bugfix/fix-dashboard-pull-dockerhub

Conversation

@Ilyesbdlala

Copy link
Copy Markdown
Member

Description

closes #1110

This is done to avoid uppercase letters in names.
for example: docker images not allowed to have uppercase letters

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
…mage

These are run as requirements for the corresponding main makefile targets
i.e `make docker-build` also runs  `dashboard-importer-docker-build`

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
@Ilyesbdlala Ilyesbdlala added bug Bugs planned Issues we will do in the next sprint. labels Apr 28, 2022
@Ilyesbdlala Ilyesbdlala self-assigned this Apr 28, 2022
Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Weltraumschaf
Weltraumschaf previously approved these changes Apr 28, 2022
Locally, it takes the pods 18 minutes to intialize on a fresh kind cluster.
This would increase the CI time required, and should probably go into the nightly build

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Weltraumschaf
Weltraumschaf previously approved these changes Apr 28, 2022
Comment thread hooks/persistence-elastic/Makefile Outdated
deploy: ## 💾 Deploy this module via HelmChart into namespace "integration-tests"
@echo ".: 💾 Deploying '$(name)' $(hook-prefix) HelmChart with the docker tag '$(IMG_TAG)' into kind namespace 'integration-tests'."
helm -n integration-tests upgrade --install $(name) . --timeout 10m --wait \
helm -n integration-tests upgrade --install $(name) . --timeout 20m --wait \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😂

@Ilyesbdlala Ilyesbdlala Apr 28, 2022 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem right now is that the deployment of persistence-elastic locally takes at least 18min. I set the timeout 20min to see if that's enough for the CI to get it done. But it doesn't seem like it. (so expect another similar commit 😂) This probably has to be moved to a nightly build.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, 20 mins is even for an integration test very long. We should consider moving such a long running test suite into something like a nightly build.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ilyesbdlala do you know why this is taking so long?

helm upgrade --install persistence-elastic secureCodeBox/persistence-elastic --wait

Takes 3 min on my local machine, and 2 min on the second run (once the images were cached)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@J12934 That was the case for me until I deleted my kind cluster and created a new one to have the same initial state as the CI. The first run takes about 18 min, the next one is less than 1 minute ( after a helm delete persistence-elastic)
The problem is reproducible through the following:

kind delete cluster
kind create cluster

# install SCB
helm repo add secureCodeBox https://charts.securecodebox.io
kubectl create namespace securecodebox-system
helm --namespace securecodebox-system upgrade --install securecodebox-operator secureCodeBox/operator

# install persistence-elastic
cd hooks/persistence-elastic
make docker-build
make docker-export
make kind-import
make deploy # this is where the issue is.

@SebieF SebieF May 1, 2022 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is reproducible through the following:

kind delete cluster
kind create cluster

# install SCB
helm repo add secureCodeBox https://charts.securecodebox.io
kubectl create namespace securecodebox-system
helm --namespace securecodebox-system upgrade --install securecodebox-operator secureCodeBox/operator

# install persistence-elastic
cd hooks/persistence-elastic
make docker-build
make docker-export
make kind-import
make deploy # this is where the issue is.

For me, it took only about 3-5 min for make deploy to run locally (both on the main branch and the PR branch)..
I had to execute kubectl create namespace integration-tests and cd hook-sdk/nodjs make docker-build in addition to the steps above. I checked by executing kubectl -n integration-test get pods (persistence-elastic and elasticsearch-master are in state Running). OS: Ubuntu 20.04

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is reproducible through the following:

kind delete cluster
kind create cluster

# install SCB
helm repo add secureCodeBox https://charts.securecodebox.io
kubectl create namespace securecodebox-system
helm --namespace securecodebox-system upgrade --install securecodebox-operator secureCodeBox/operator

# install persistence-elastic
cd hooks/persistence-elastic
make docker-build
make docker-export
make kind-import
make deploy # this is where the issue is.

For me, it took only about 3-5 min for make deploy to run locally (both on the main branch and the PR branch).. I had to execute kubectl create namespace integration-tests and cd hook-sdk/nodjs make docker-build in addition to the steps above. I checked by executing kubectl -n integration-test get pods (persistence-elastic and elasticsearch-master are in state Running). OS: Ubuntu 20.04

Yeah, it seems my theory, that the error is due to the amount of time required for persistence elastic to deploy, is false.
The error log from the CI is not helpful. I'm trying to figure out if there's a flag I can set to allow for more debug logs.

@J12934 J12934 May 4, 2022 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ilyesbdlala The pods have been pending over the entire time, meaning that something is missing for kubernetes to start the pods (see the Inspect Post Failure step: https://github.com/secureCodeBox/secureCodeBox/runs/6286731595?check_suite_focus=true#step:27:31)

I've added a kubectl describe pods over the integration-tests namespace. This prints out the detailed output for the pods, which should tell us the reason why kubernetes can't start the pods.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hosted github action runners are apperently too small to run elasticsearch...
https://github.com/secureCodeBox/secureCodeBox/runs/6294995302?check_suite_focus=true#step:27:228

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
J12934 added 3 commits May 4, 2022 18:15
…ther with --all-namespaces...

Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Elasticsearch pods have been pending in the namespace the events should tell us more why that is

Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
@J12934
J12934 force-pushed the bugfix/fix-dashboard-pull-dockerhub branch from 60b260d to cbced53 Compare May 4, 2022 16:17
J12934 and others added 3 commits May 4, 2022 20:09
Signed-off-by: Jannik Hollenbach <jannik.hollenbach@iteratec.com>
Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
The fix was not effective

This reverts commit 7ffb244.

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Reason mentioned in issue #1165

Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
Signed-off-by: Ilyes Ben Dlala <ilyes.bendlala@iteratec.com>
@Ilyesbdlala

Copy link
Copy Markdown
Member Author

This PR closes the issue #1110. A new issue #1165 was discovered.

@J12934
J12934 merged commit 93874ab into main May 11, 2022
@J12934
J12934 deleted the bugfix/fix-dashboard-pull-dockerhub branch May 11, 2022 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bugs planned Issues we will do in the next sprint.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard importer is pulled from Dockerhub instead of being built

4 participants