422 questions
1
vote
0
answers
26
views
Is this possible: Use output of upstream component within a dsl.If; otherwise, use None
I want to have a downstream component that has an optional dsl.Dataset input. That optional input is the output of an upstream task within a dsl.If. All of these are within a dsl.ParallelFor, so that ...
- 101
0
votes
1
answer
72
views
ansible regex to search and append ip as a registry host for container images
We are trying to update the image: field in Kubernetes YAML manifests by dynamically appending an IP address (10.20.30.40/) to the image URL, except in certain cases. Specifically, we have three ...
- 533
0
votes
0
answers
119
views
Kustomize Replacement and prefix Issue: prefix container registry host to Image with missing registry and Registry Replacements for existing images
I'm working on a Kustomize setup where I need to:
Replace the registry for images that already have a registry (e.g., docker.io → 10.20.30.40).
Prepend the registry for images without one (e.g., mysql:...
- 533
1
vote
0
answers
207
views
How to get the outputs and artifacts of a pipeline in kubeflow?
I'm using the pythagorean example in kubeflow v2:
@dsl.component
def square(x: float) -> float:
return x ** 2
@dsl.component(packages_to_install=['numpy==1.21.6', 'pyyaml==6.0'])
def add(x: ...
- 11
1
vote
0
answers
162
views
RBAC: Access Denied when deploying Kubeflow using already hosted Istio
We're deploying Kubeflow on an EKS cluster that already has both istio 1.23 and cert-manager installed on it.
We therefore skipped istio and cert-manager installation and used option 3 from oauth2 ...
- 485
1
vote
0
answers
38
views
failed: failure while getting executionCache: failed to list tasks: rpc error: code = PermissionDenied desc = RBAC: access denied
I am migrating towards kubeflow v2. This is the pipeline I am using (from the tutorial page):
from kfp import dsl
@dsl.component
def say_hello(name: str) -> str:
hello_text = f'Hello, {name}!'
...
- 11
-1
votes
1
answer
180
views
Python dependencies in kubeflow spark operator
I wanted to ask if there is a way to use python as a .wheel or .egg or just .py dependency in kubeflow spark operator.
The resulting file i have in mind would look something like this, the dependecy ...
- 33
0
votes
1
answer
200
views
What causes a TypeError under all conditions on Kubeflow KFP v2.0.0+?
kfp==2.8.0 # and on 2.7.0
kfp-pipeline-spec==0.3.0
kfp-server-api==2.0.5
Example code:
from kfp import dsl
import kfp
@dsl.component
def add(a: float, b: float) -> float:
'''Calculates sum of ...
- 31.7k
1
vote
1
answer
443
views
What is the expected way to specify an image pull secret in Kubeflow?
I have generated the secret, but cannot configure it in kubeflow's implementation.
If I look in kfp, I see that all references to imagePull are in the deprecated folder:
$ grep -rn 'imagePull' kfp/
./...
- 31.7k
0
votes
1
answer
1k
views
How do I increase the resources allocated to individual docker container instances running in Colima?
I see that I can increase the resources of the Colima VM, but the docker containers are executed with the default configuration:
$: colima start --cpu 10 --memory 20 --disk 50
$: colima list
PROFILE ...
- 31.7k
1
vote
1
answer
324
views
Kubeflow 2.2 pipeline error: No existing KFP Config file found
I am trying to use the kubeflow v2 following the document below but but I am not able to run this succesfully. Below is my code snippet, Could you please let me know if something wrong.
https://www....
- 53
0
votes
1
answer
429
views
How to create a kubeflow component from already running python scripts
I am very very new to both AI and MLOP's, please forgive me if my question is dumb. I am trying to learn about kubeflow but there is too much information on kubeflow documentation, then there are ...
- 53
0
votes
1
answer
342
views
Unable to run notebooks on GPU (on-premises), running kubeflow on ranchers k3s cluster
I have installed kubeflow on k3s cluster.
I have configured the cluster to be able to access GPUs and it's working fine for test pods on gpu, and I can see nvidia-smi output on pods.
But, I want ...
1
vote
1
answer
328
views
select runtimeClassName for kubeflow notebooks
I configured my kubeflow gpu server recently, but the GPU does not get detected in there - the reason is cluster using wrong runtime.
I prefer not to change the default runtime system-wide and tried ...
- 740
0
votes
1
answer
294
views
How to connect a secret containing Docker registry credentials to a Kubeflow pipeline for accessing a base image from a private docker hub repository?
I'm using a Docker image that I've pushed to a Docker hub public repository as the base_image for a Kubeflow component:
subtract_op = create_component_from_func(func=subtract, base_image="<my ...
1
vote
0
answers
39
views
Propagate exception outside of a Kubeflow component to stop a VertexAI pipeline
In short, VertexAI pipelines are not stopping when an exception is thrown inside a function that is inside an ephemeral_component
An example pipeline, with this stack trace in the logs
2024-04-09 14:...
- 1,493
1
vote
0
answers
395
views
How to create a "multi-node" (node=machine) Kubernetes/Kubeflow cluster for Machine Learning Training?
I have multiple computers/machines. Each computer/machine has multiple GPUs. I am currently training my machine learning model using tensorflow. I can train on a single machine/computer, or I can use ...
- 11
0
votes
0
answers
180
views
How are you able to access the Kubeflow dashboard?
I am new to kubeflow and I am trying to access the dashboard, but no success. I am running kubernetes locally installed via minikube. My lasta attempt was running the following command:
kubectl create ...
1
vote
0
answers
345
views
Setting parallelism of `ParallelFor` in Vertex AI dynamically, based on the pipeline parameter
I have a Vertex AI pipeline that takes in a country argument (a string such as "uk", or "de"...). The first task of the pipeline get_country_dr_keys generates all "dr keys&...
- 2,548
1
vote
0
answers
261
views
Kubeflow bug deleting pipelines
I had working pipelines in kubeflow, but recently when I go to the dashboard, I get 'No graph to show' when it should be showing the pipeline components. I have done nothing to pipeline to cause this, ...
- 23
0
votes
0
answers
243
views
Trouble Pulling Images from Self-Hosted Private GitLab Registry in Kubeflow Pipelines on k3s/WSL2
I am new on Kubernetes and Kubeflow and trying to pull images from my self hosted private gitlab registry in Kubeflow Pipelines. This is my pipeline:
@container_component
def container_component_trial(...
- 39
0
votes
0
answers
476
views
I have some troubles with authentication in istio
I do all this on my server.
I followed this guide https://www.deploykf.org/guides/local-quickstart/.
I reached the end of the guide and now I can’t reach my kubeflow.
I added what I thought were valid ...
- 11
-1
votes
1
answer
325
views
Can i set Kubeflow pipeline storage path?
I'm trying to install Kubeflow with S3.
The artifact repository has been set up.
I would like to know how to specify the s3 prefix of the storage where the pipeline is stored.
Is this possible?
how ...
- 3
1
vote
0
answers
475
views
Kubeflow: access denied to Minio artifacts from the pipeline (run) graph view
I installed Kubeflow on my Docker Desktop Kubernetes. Everything seems to work so far, apart from the following problem:
When trying to access any artifacts (input/output, logs, etc.) from the Run ...
1
vote
2
answers
595
views
I have installed kubeflow pipeline but not have jupyter notebook in UI
I have installed kubeflow pipeline in mac M1 but not able to see "notebook" option in UI.
I have used these step
# env/platform-agnostic-pns hasn't been publically released, so you will ...
4
votes
0
answers
2k
views
Custom type parameters for KFP components
I'm working with KFP components and I want to pass a custom type parameters, it threw this error TypeError: Artifacts must have both a schema_title and a schema_version, separated by @. Got: ...
- 1,035
0
votes
1
answer
468
views
Kubeflow error:Invalid input error: Job has no experiment. code:3 message
` I am following tutorial of Kubeflow Pipelines API: Kubeflow & Kubeflow Pipelines API
I've installed Kubeflow on a MicroK8s and started experimenting with the Kubeflow Pipelines API, I was ...
3
votes
2
answers
246
views
AI Platform Pipeline 2.0.0 deployment not visible on GCP Console
On GCP, I created a new Kubernetes cluster 1.26 and deployed kubeflow pipelines 2.0.0 on it. The deployment went fine but on the AI Platform Pipeline page, I don't see the new deployment. Due to this, ...
- 4,059
0
votes
1
answer
419
views
Always 404 on kfp 2.0.1
I was using kfp==2.0.0b12 and everything was working great. But as soon as i got into 2.0.1 I am always getting 404 when trying to use methods from client, like this:
kfp_server_api.exceptions....
- 1
1
vote
1
answer
480
views
Adding arbitrary metadata to kubeflow pipeline component
Kubeflow pipelines docs mentions that there is a way to add metadata annotations as a key-value pairs. From the other hand, I could not find any relevant documentation of how it can be done.
Is there ...
- 43
0
votes
0
answers
522
views
kfp.dsl. placeholders (used as component inputs) work on default machine but are not transformed in a custom machine
kfp version 1.8.11
I have a pipeline and I need to use some pipeline/task parameters to keep track of my experiments and do the pathing for GCS.
I provide this as inputs of the components:
kfp.dsl....
- 401
0
votes
0
answers
607
views
Building a Kubeflow Pipeline using Docker images from a private GitLab Registry
I'm relatively new to Kubeflow and I'm trying to create a pipeline that uses Docker images stored in my private GitLab registry. I've looked through the Kubeflow documentation, but I couldn't find a ...
- 39
0
votes
0
answers
153
views
GCP AI Platform doesn't list Kubeflow pipeline application on Pipelines page
I am doing a lab titled as "TFX on Cloud AI Platform Pipelines" from Qwiklabs Machine Learning path. The instructions asks me to run a shell file by which I grant access to corresponding ...
0
votes
0
answers
397
views
my istio-ingressgateway pod does come up when instaling Kubeflow from the kubeflow/manifest (istio 1.17)
I will be glad if someoone kindly help me with this please. I am a newbie in DEVOPS.
I have installed a standalone cluster of kubernetes using kubectl 1.27.4 and minikube version: v1.31.1 on ubuntu 22....
1
vote
0
answers
82
views
how to get model from KF Containerized Python Component into Vertex AI model registry properly
if custom model training happens in Containerized Python Component, producing model file and metrics, what is the proper way of uploading the model and its metrics into Vertex AI so that they are ...
- 4,085
2
votes
0
answers
199
views
what replaces `aiplatform.CustomContainerTrainingJobRunOp` in `google-cloud-pipeline-components` V2?
Apparently this class I've been using heavily is removed in V2 and it is not exactly clear from the docs what it should be replaced with.
"Remove aiplatform.CustomContainerTrainingJobRunOp and ...
- 4,085
0
votes
2
answers
841
views
Change uri name of a KFP artifact to .csv instead of "dataset"
When I output a KubeFlow Output[Dataset] that I know will be in a CSV format output URI is ending with the text dataset.
Is it possible to change the name of the URI to dataset.csv or to training_data....
- 11
0
votes
1
answer
585
views
Kubeflow pipeline crashes because the "argoexec wait" container gets OOMKilled
I have a Kubeflow pipeline whose first step does nothing but downloads one file. This step always crashes in an out-of-memory error:
This step is in Error state with this message: OOMKilled (exit ...
- 3,717
2
votes
0
answers
2k
views
kubeflow jupyter notebook error-Could not find CSRF cookie XSRF-TOKEN in the request
I installed kubeflow v1.4.0 on ubuntu 20.04 with the following specifications:
k v1.21.7
kustomize v3.1.0
however when i try to create a new notebook, i encounter the following error:
Could not find ...
- 21
0
votes
1
answer
173
views
Kubeflow Notebook: connect to SQL Server
I'm using kuberflow notebook (python 3) and trying to connect to my sql server. So far here's my code:
import jaydebeapi
driver_name = "net.sourceforge.jtds.jdbc.Driver"
connection_url = f&...
- 3
0
votes
1
answer
372
views
Missing buttons for logs, pod etc on components of pipeline runs in KFP UI
We have set up an on-prem Kubeflow 1.7 installation and are trying out Kubeflow Pipelines v2.x.
When executing a run on the provided pipeline "[Tutorial] Data passing in python components", ...
- 1,101
1
vote
0
answers
188
views
kubeflow deployment failed with knative domainmapping-webhook crashloopbackoff
Just install Charmed Kubeflow. Everything is ok but missing knative.
Keep crashing with log below
{"severity":"ERROR","timestamp":"2023-07-16T18:37:03.684598799Z&...
0
votes
1
answer
197
views
How to install Kubeflow cli
I am playing with Kubeflow's containerized python components. As per the below doc, we can push component to container registry by using "target_image" flag in the definition and running the ...
- 21
0
votes
1
answer
3k
views
Got error "Waited for xxx due to client-side throttling, not priority and fairness" when Kubeflow Training Operator
I have a local Kubernetes created by Rancher Desktop. I am trying to deploy based on the installation guide.
However, after deploying by
kubectl apply -k "github.com/kubeflow/training-operator/...
- 50.7k
0
votes
1
answer
222
views
Different library version during the run of vertex pipeline vs in the docker container
I am using a Component in a vertex ai pipeline with a custom component.
@component(
base_image=f"gcr.io..."
)
def temp_step():
It's supposed to have the same library versions during the ...
- 43
1
vote
1
answer
994
views
Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow
I am trying to follow the Kubeflow v2 quickstart.
First, I deployed Kubeflow to a local Kubernetes cluster by
export PIPELINE_VERSION="2.0.0-alpha.4"
kubectl apply -k "github.com/...
- 50.7k
0
votes
1
answer
280
views
I am trying to create a simple vertex ai pipeline to read a gcs folder containing .parquet files using kubeflow
Error importing aiplatform
Tried following the vertex ai documentation and while running:
from google_cloud_pipeline_components import aiplatform as gcc_aip
I get an error:
Import error: Cannot import ...
- 11
2
votes
1
answer
495
views
How to trigger a component in a Kubeflow pipeline regardless of whether a previous condition is skiped or completed
My graph starts with a common component - after which the process is split in two leaves, one for variables and another for events. Both leaves use a conditional to execute some components if there is ...
0
votes
2
answers
757
views
Same component works when defined through @component but it fails when created with create_component_from_func
I have a Docker container in gcloud with all the code I need (multiple files, etc.). I'm able to run it when I defined the component using the @component decorator. I define the function and I set up ...
- 401
3
votes
1
answer
5k
views
HorizontalPodAutoscaler not found on minikube when installing kubeflow
I installed the latest minikube version according to its website (https://minikube.sigs.k8s.io/docs/start/):
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo ...
- 1,029