6,992 questions
0
votes
0
answers
53
views
Google tag gateway in GKE Gateway API
I'm implementing server side tagging for Google Tag Manager through my GKE Gateway API.
I was successfully able to configure server side tag manager in same domain and GET https://www.example.com/...
3
votes
3
answers
236
views
Node.js google-auth-library Transient Failure During Scheduled IAM Token Refresh on GKE
I'm running a Node.js application on GKE with Workload Identity, and I'm using ioredis with IAM authentication. I'm hitting a really frustrating, persistent issue during my scheduled token refresh ...
-3
votes
0
answers
65
views
OpenTelemetry + ReactiveFeign + Spring WebFlux not propagating traceparent between microservices (header injected but missing in downstream service)
I’m running two Java microservices (A → B) in a Kubernetes cluster (GKE), using:
Spring Boot 3.5 (WebFlux)
ReactiveFeign (Playtika Feign Reactor 4.2.1)
OpenTelemetry Java Agent (2.20.1)
OTLP ...
0
votes
0
answers
72
views
Google Cloud error 429: The request was aborted because there was no available instance
I have a function in Cloud Run that takes files from a bucket, loads them to BigQuery and then deletes them.
It is activated using the Cloud Scheduler and an HTTP call.
To avoid processing the same ...
0
votes
0
answers
40
views
GKE Autopilot: Preventing Scale-Down Interruptions for Long-Running Quartz Jobs
I’ve been tasked with migrating an old monolith to GKE Autopilot (my first time with GKE Autopilot).
The monolith can scale horizontally, but it runs jobs using Quartz; these jobs can last for hours. ...
0
votes
1
answer
160
views
POST requests from Cloud Run arriving as GET requests in my Django REST API
I have a serverless pipeline on Google Cloud. It consists of three total steps:
A video uploader that sends videos to Google Cloud Storage. Working fine.
An eventarc + pub/sub trigger that fires ...
-1
votes
1
answer
105
views
What JSON keys do GKE recognize in structured logs? [closed]
From "About GKE logs > What logs are collected" I gather that Application Logs can be JSON (I guess it's the preferred way).
The "About GKE logs > Best practices" mention in ...
-1
votes
1
answer
131
views
Trying to sync GCP secret with GKE [closed]
I am trying to sync GCP Secret to my GCP Gke secret.
I created SecretProviderClass.
% kubectl -n dev get secretproviderclass backend-secret -o yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: ...
0
votes
0
answers
101
views
Google Cloud Trace Explorer not showing OpenTelemetry service name (Node.js on GKE)
I am using OpenTelemetry with Node.js in my application running on Google Kubernetes Engine (GKE).
The traces are successfully sent to Google Cloud Trace, but in Google Cloud Trace Explorer the ...
1
vote
1
answer
125
views
Is GKE's 15s pod termination grace period always guaranteed on preemptible node shutdown? [closed]
I'm running Pods on GKE preemptible nodes, and I've observed some inconsistencies with graceful shutdown behavior.
According to GKE documentation:
When Compute Engine needs to reclaim the resources ...
1
vote
1
answer
261
views
K8 gateway issue and the gateway api routes does not work [closed]
GKE k8 gateway api route not working saying "no healthy upstream" even when the gateway and routes are created healthy .Below are the configs
apiVersion: networking.gke.io/v1
kind: ...
0
votes
1
answer
97
views
GCP GKE how to not create pod with selector managed-by cloud-console
I have been manually deploying new pods with GKE for the past three years. Once done, I then configure Cloud Build for my CI/CD to deploy to GKE once the image is created, etc.
For the CD from ...
0
votes
1
answer
60
views
GKE reports StatefulSet specified does not have a readiness probe but spec does contain it
GKE is reporting the alert "PDB_STATEFULSET_WITHOUT_PROBES" (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-disruption-readiness) but the statefulsets it references do have ...
0
votes
1
answer
146
views
Pod to service traffic on GKE Dataplane v2 vs Cilium
I am looking into different behaviour between Cilium CNI and GKE Dataplane V2.
Mainly, for Pod to ClusterIp traffic, with kube-proxy replacement enabled on:
Cilium - traffic coming from LXC interface ...
1
vote
1
answer
105
views
Intermittent Failures with ESPv2 Sidecar on GKE Autopilot (NAP Nodes)
We’re experiencing intermittent failures with the endpoints-runtime (ESPv2) sidecar container on GKE Autopilot. Our setup uses ESPv2 as a sidecar alongside a Spring Boot backend container to expose an ...
0
votes
0
answers
46
views
Not able to mount themes stored in GCS bucket to gke keyclaok
i'm trying to add my custom theme from google cloud storage bucket to my bitnami keycloak using gcs-fuse-csi driver but the theme is not showing in the keycloak. I hosted my bitnami keycloak in gke ...
0
votes
1
answer
70
views
error fetching latest gke min version using data.google_container_engine_versions
I'm using
data "google_container_engine_versions" "latest_version" {
provider = google-beta
location = var.region
version_prefix = "1.31."
}
...
0
votes
0
answers
23
views
Unable to setup the Multicluster Gateway
I was trying to setup Multicluster gateway using cross region internal load balancer following this guide - https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-multi-cluster-gateways#...
0
votes
0
answers
62
views
Hybrid cloud model in GCP
I recently came across this article on RudderStack’s hybrid cloud model: https://www.rudderstack.com/blog/reinventing-the-on-prem-deployment-model/
The core idea is to split the architecture into two ...
0
votes
1
answer
115
views
Issue while deploying container in GKE
I am getting this error while deploying in GKE :
Error from server: Get "https://10.x.x.x:10200/containerLogs/server-center/server-center-dev-86f67jkilo-rwrnm/server-center-dev": No agent ...
0
votes
1
answer
146
views
Websockets on GKE with Nginx Ingress
I am trying to get websockets to work on GKE. Seems very trivial, but I am failing to get this to work, I just continuously keep getting 400 at Nginx Ingress.
The manifest is like this:
apiVersion: v1
...
0
votes
0
answers
59
views
How to enable anthos service mesh on two GCP host projects (each project has it's own shared VPC)?
I have 2 different google cloud projects, they are the host projects for shared VPCs and have service projects attached to them. Let's call the host projects P1 and P2.
Now a service runs inside P1 ...
0
votes
2
answers
148
views
Scaling down unexpectedly after 15 minutes with api-metrics trigger of KEDA on GKE
I'm running KEDA v2.17 on Google Kubernetes Engine (GKE) Autopilot, and I've observed that my deployment scales down to 1 replica after about 15 minutes, even though my API consistently returns a ...
1
vote
1
answer
200
views
Error when trying to migrate a GKE Autopilot cluster to cgroupv2
We're trying to migrate our GKE Autopilot cluster from cgroupv1 to cgroupv2 following the official documentation (https://cloud.google.com/kubernetes-engine/docs/how-to/migrate-cgroupv2#autopilot_1), ...
0
votes
1
answer
49
views
How to avoid GKE evicts job pod and results in Prefect flow run failure?
We use Prefect (v3.0) in our data platform to orchestrate pipelines. We use GKE (Kubernetes) for Prefect workers and flow run deployment. Our GEK cluster is in auto scale (horizontal) mode.
In the ...
0
votes
0
answers
56
views
Segmentation issue, running PyTorch on GPU supported GKE node pool
I have a node pool of n1-highmem-4 machines with 1 NVIDIA Tesla T4 attached with a COS_CONTAINERD image. I am running a transformer model in python on a pod to execute the model on GPU. I get an ...
0
votes
2
answers
171
views
Correct way to use GKE Connect Gateway client?
I'm trying to use the Connect Gateway client in python to connect to GKE but I keep getting 400, even if everything seemingly is correct.
Example
from google.cloud.gkeconnect import gateway_v1
client ...
0
votes
1
answer
87
views
Airbyte GCP/GKE logging
I was looking at how stdout/stderr kubernetes pod logs are stored within gke.
My problem is storing Airbyte connectors log to GCP (Airbyte is part of a GKE cluster).
Unfortunately pod stdout/stderr is ...
0
votes
0
answers
193
views
apicurio schema registry v2.6 - Java Kafka Producer not working
I've Apicurio Schema Registry Operator installed, also Schema Registry v2.6 installed. This is the default version of Schema Registry installed when using Apicurio Schema Registry Operator. Keycloak(...
0
votes
1
answer
166
views
GKE Multi-Cluster Ingress (MCI) ignores BackendConfig Health Check, uses default HTTP / check
Environment:
GKE Standard Clusters (Version: ~1.31.6-gke.1064001)
2 Clusters (sglang-us in us-central1, sglang-asia in asia-northeast1)
Clusters registered to a Fleet, MCI enabled, sglang-us is the ...
1
vote
2
answers
189
views
Kubernetes startupProbe fails even though app becomes healthy within allowed threshold
I'm running into an issue with my (GKE) Kubernetes deployment's startupProbe. My container exposes a /v1/health endpoint that returns JSON with a "status" field. The probe is configured as ...
0
votes
0
answers
150
views
403 Access denied. Missing IAM permission: bigtable.tables.readRows after 30 minutes of working fine
I'm trying to connect to bigtable instance and read data using following code to create the client:
import logging
from os import path
from google.cloud.bigtable.data import BigtableDataClient
from ...
-3
votes
1
answer
148
views
Kubernetes engine error with apiVersion and kind
I'm getting an error
This apiVersion and/or kind does not reference a schema known by Cloud Code. Please ensure you are using a valid apiVersion and kind.
on both manifest.yaml and nordered-service....
0
votes
0
answers
70
views
Why Prometheus stack and adapter not fetching the custom metrics in GKE
As same as below document, we have implemented the custom metrics pod autoscaling in our project.
https://medium.com/@MetricFire/prometheus-metrics-based-autoscaling-in-kubernetes-129deb8e7fa6
We have ...
0
votes
0
answers
46
views
Spring Boot Swagger UI not working after deployment to GKE
After deploying my spring boot 3.1.4 Kotlin Gradle app to GKE, and trying to access the swagger ui from the /swagger-ui.html or /swagger-ui/index.html or even /v3/api-docs I receive the whitelabel ...
0
votes
1
answer
78
views
How to start a container in a kubernetes pod only after its proxy is running?
I have a kubernetes cluster and a PostgreSQL Database running on Google Cloud.
The pod that has the problem is a cronjob with the following configuration:
apiVersion: batch/v1
kind: CronJob
metadata:
...
0
votes
0
answers
30
views
how to connect to open telemetry operator in kubernetes
I have this yaml file defining a open-telemetry-collector in a kubernetes cluster. I have already installed open-telemetry opertor. This is the file:
apiVersion: opentelemetry.io/v1beta1
kind: ...
0
votes
1
answer
289
views
GKE pod with filestore RWX volume takes 30 minutes to start Error syncing pod, skipping" err="unmounted volumes=[filestore-rwx-volume],
I have a GKE pod mounted with RWX volume with Filestore. Below are my storage class,PV,PVC configs.
GKE Version - 1.30.9-gke.1127000
For all the pods which uses this multishare volume takes about 30 ...
0
votes
0
answers
41
views
GKE ManagedCertificate "FAILED_NOT_VISIBLE" & Missing Ingress External IP
I'm using GKE with an Ingress and a ManagedCertificate for my domain (details redacted). My YAML manifest is as follows:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-name-...
-1
votes
2
answers
100
views
How to provision a GKE cluster - Enterprise Tier with Terraform
As the terraform docs for google_container_cluster resource (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/container_cluster), to provision a GKE cluster we need to ...
0
votes
1
answer
60
views
JWT Validation Failure in Spring Boot API due to PKIX Path Issue with Keycloak on GKE (Gateway API, TLS Termination)
I have a GKE architecture with the following components:
External Traffic Flow:
Public F5 load balancer (terminates TLS for *.example.com) routes to:
app.example.com, api.example.com, idp.example....
1
vote
1
answer
534
views
Autopilot GKE Cluster: GCE quota exceeded and insufficient CPU error
I have been experiencing a problem running an Autopilot GKE cluster. This problem actually inhabilitates pods to run so i'ts a little bit frustrating.
Actually mi configuration is only of two ...
0
votes
1
answer
76
views
Django Cloud Storage NoSuchKey
Update
I find this,
https://django-storages.readthedocs.io/en/latest/backends/gcloud.html
Since my Django is 5.1, I modify the settings.py like this.
STORAGES = {
"default": {
&...
0
votes
1
answer
343
views
GKE Ingress defaultBackend not working despite healthy pods and correct NEG configuration
I have a GKE Ingress configured with a defaultBackend to handle requests for undefined hosts. The pods are healthy, but the default backend does not respond. Here’s my setup:
Configuration Details
...
1
vote
1
answer
205
views
How to config a gke-l7-rilb Gateway for HTTPs?
Google documentation doesn't clarify how to do it, but my goal is to expose an App deployed in K8s into a VPC only, but could be seen by an user that uses a VPN in another project, using HTTPs ...
0
votes
0
answers
160
views
GCP - GKE IAP give access to external user
We have services on GKE protected by IAP. So far, whenever we need to give access to a new internal user, we provide them with the role roles/IAP.httpsResourceAccessor.
The organization policy ...
1
vote
0
answers
172
views
Kubernetes external-secret.io operator can not unmarshal GCP secret value stored as plain text
Created GCP secret and stored plain text value e.g. userpassword
Created and applied external-secrets.io yaml manifest as shown below
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
...
0
votes
1
answer
101
views
Not getting the all the namespaces in big query billing export table
I have enabled the GCP cloud billing data to export into bigquery. For GKE I'm not getting all the workloads namespaces into gcp_billing_export table.
I have enabled the GKE Metering, It's created the ...
1
vote
1
answer
230
views
Are both RBAC and IAM permissions needed for GKE authorization?
I am trying to authorize to a GKE cluster using a service account with the following permissions
cluster.deployments.get
container.clusters.get
I create a kubeconfig by making API calls to get ...
0
votes
1
answer
192
views
Unable to setup GKE workload identity invalid argument
Following google docs on how to setup workload identity:
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity. I have enabled the GKE metadata server on all nodes as well as ...