0

After submitting the "run" using Jupyter notebook, when I go to the Kubeflow pipeline dashboard, I can see my "run" submitted & running but it doesn't end even after few hours & doesn't show any logs either.

I have tried running the "DEMO XG-BOOST" but the same issue persists & the "run" doesn't end.

Can someone please help me understand if there is any issue with the account settings or any other issue which I'm missing because I have tried to check documentation & other websites but couldn't understand why this occurs.

Please note that this "run" was working fine (creating the pipeline flow) on coursera using quicklabs (Covertype classifier)

error snapshot

Using below versions: kfp 1.8.12 ; kfp-pipeline-spec 0.1.14; kfp-server-api 1.8.1; Python 3.7.12

If there is any other information which I have missed to put above, please let me know so I could share that information that could help us solve this issue.

Improve this question

1 Answer 1

Reset to default
0

The issue mentioned above has been resolved. The part where "DEMO XG-BOOST" run doesn't end has been resolved by selecting "Use emissary executor" option while creating the pipeline.

see the snapshot

When we launch the pipeline with this setting, it solved the issue & we can run complete pipelines now.

More details: We took support from GCP & they have mentioned that the issue might have been caused by a recent upgrade of the GKE cluster which removes the docker runtime (https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#docker-executor). Namely, the Docker executor is the default workflow executor and depends on docker container runtime, which is deprecated on Kubernetes 1.20+. We were using a GKE cluster whose version was 1.21.6. Hence the issue. So, we used the documentation (https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/#migrate-to-emissary-executor) & migrated to the Emissary executor (instead of Docker) which has solved our issue.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.