731 questions
0
votes
0
answers
122
views
What is the right gcp cloud monitoring promql query to capture "unspecified" (or all resource types) metrics?
The following promql query returns the logging.googleapis.com/log_entry_count metric in the global namespace:
increase({"__name__"="logging.googleapis.com/log_entry_count","...
- 32.5k
0
votes
1
answer
609
views
How do I convert this alert policy MQL to PromQL?
I'm trying to convert an MQL query I'm using in a GCP alert policy to PromQL while also aggregating by user labels.
This is the MQL:
fetch gce_instance
| metric 'agent.googleapis.com/cpu/load_5m'
| ...
- 32.5k
0
votes
1
answer
166
views
How do I fix a custom metric that can't be aggregated by two projects. It says chart definition is invalid
I'm experimenting with a custom metric. The same metric and labels were being written in different projects at the same time. They should have been identical, but when I view the metric I see this:
...
- 32.5k
0
votes
1
answer
96
views
Using catalina_globalrequestprocessor_requestcount over time as a metric for custom metric HPA
Is there a way to use catalina_globalrequestprocessor_requestcount metric (or another MBean metric) to track requests as a rate (e.g., requests per second) for use with HPA?
Background
We're using an ...
- 27.5k
0
votes
1
answer
254
views
Why do I set the resource type when writing the metric but not when creating a descriptor?
This is very confusing because the docs make it sound like I choose a resource type when I'm manually creating a descriptor: https://cloud.google.com/monitoring/custom-metrics/creating-metrics#create-...
- 32.5k
0
votes
2
answers
849
views
Is there a good explanation of resource labels versus metric labels?
When writing custom metrics you have resource labels and metric labels.
Resource labels are labels that are required by the resource. So if I choose the generic_task resource type and exclude the ...
- 32.5k
1
vote
1
answer
119
views
Ruby exceptions not appearing in Google Cloud Error Reporting
I have a Rails app running in GKE and I'm trying to set up Google Cloud Error Reporting. I have installed the stackdriver gem and required it on application startup. I can manually report errors from ...
- 2,541
1
vote
1
answer
238
views
HPA unable to find "untyped" custom metric from Prometheus / Stackdriver adapter
We are using HPA with custom metrics from a Java application in GKE. (More on that in this previous question.) We would like to use a untyped metric, though.
What we have done
So far, we did this:
...
- 27.5k
-1
votes
1
answer
191
views
Are GCP monitoring uptime check metrics suppose to be aggregatable by user labels?
I'm trying to understand the utility of "user labels" for my uptime checks: https://cloud.google.com/monitoring/uptime-checks#create
I was expecting it to work like other resources. I want ...
- 32.5k
0
votes
1
answer
623
views
Horizontal Pod Autoscaler with Stackdriver Custom Metric in GKE fails with "Invalid metric name" error
We're trying to use a custom JVM metric (jvm_memory_bytes_used{area="heap"}) to scale a deployment in our GKE cluster using a Horizontal Pod Autoscaler (HPA).
Setup:
Enabled Stackdriver ...
- 27.5k
0
votes
1
answer
397
views
How do I increase the retention period of non-custom GCP metrics?
I'm having a hard time finding out if this is even supported because the docs don't outright say I can increase this retention at all or not:
https://cloud.google.com/monitoring/quotas#...
- 32.5k
0
votes
0
answers
271
views
Post-Migration Logging Issue in Python 3 GAE Application
We have recently migrated our application from Python 2 to Python 3 on Google App Engine and are encountering a recurring issue related to logging. Despite the application's functionality appearing ...
1
vote
1
answer
146
views
LoggingAppender getting `projectId` from service account key json instead of env variable `GOOGLE_CLOUD_PROJECT`
I have these two variables set in intellij GOOGLE_CLOUD_PROJECT and SPRING_CLOUD_GCP_LOGGING_PROJECT_ID but still LoggingAppender class is getting projectId null and ultimately reading from service ...
- 390
1
vote
1
answer
345
views
How to disable stackdriver logging in a spring application for a GCP project without creating a new docker image
I have a spring application packaged inside a docker image and running in GKE pods. I want to disable stackdriver logging but only for this application in gke. Is there any env variable/property I can ...
- 390
0
votes
1
answer
478
views
How to reduce the Cloud SQL Logging cost in GCP?
I am currently checking for an option to reduce Cloud SQL logging cost (Cloud Logging cost in GCP). Currently the Cloud SQL instance is writing millions of log entries in stackdriver per day. Out of ...
- 119
0
votes
0
answers
156
views
Correlating logs for App Engine Standard in Python3 using Flask
There has been many posts about this topic, but I was not able to find a definite answer on establishing correlated logs under the HTTP Request logs like it used to be in App Engine using Python 2.7
I ...
- 927
0
votes
1
answer
2k
views
How to ingest GCP monitoring metrics into Prometheus
I have a GCP project with various metrics that are gathered automatically by GCP monitoring (ex. Stackdriver). I also have a GKE cluster running a prometheus server. I am trying to ingest all metrics ...
- 190
1
vote
1
answer
106
views
Stack driver GCP
Where I can view the stack driver logs in Log Explorer GCP console?
I didn't get any information regarding this. Previously there used to be a console saying Stackdriver Logging.
Is stackdriver logs ...
- 33
1
vote
1
answer
289
views
Mysterious Stackdriver error seems to suggest a bug with Stackdriver itself
For around three weeks now, I see these errors in Google Cloud console, for a Golang Appengine app:
{"errorReference":"ixWACIGAwIxxxxxxxxMCJu5TG-ykYAjIKGICAwIm7lMb7KWiBgMCJu5TG-ykD&...
- 43
1
vote
0
answers
990
views
Migrating from fluentd to fluent-bit for stackdriver logs on Container-Optimized OS
I'm attempting to upgrade from the legacy fluentd logging agent to fluent-bit on Container-Optimized OS, as recommended here. These are Java logs in JSON format, produced by logback, and configured ...
- 3,300
1
vote
0
answers
220
views
Issue with Airflow remote logs on Google Stackdriver
I have Airflow 2.5.1 deployed on a Kubernetes Cluster with the official helm chart. The executor is KubernetesExecutor.
I want to write and read the airflow logs remotely on Stackdriver.
In my code I ...
- 151
2
votes
2
answers
904
views
How do I build a query using MQL to fetch data from Log based metric?
I have a log based metric where the resource type is not defined. How do I build a MQL query for a log based metric for the following:
logging.googleapis.com/user/MY_METRIC
If I use the "...
- 155
1
vote
0
answers
275
views
How to initialize GCP MetricServiceClient with an API_KEY in C#
I want to initialize Google Cloud Metric Service Client (AKA Stackdriver)(Google.Cloud.Monitoring.V3.MetricServiceClient) for a local C# application.
gcloud auth application-default login with ...
0
votes
1
answer
127
views
Getting an error during initialization of the Google Stack Driver appender
Have a google stack driver appender in my log4net configuration but during initialization I get an error about not being able to cast as an IAppender. Here's the reference for the appender.
<...
- 916
1
vote
1
answer
1k
views
Stackdriver in GKE throws rpc error "Failed to export to Stackdriver" "context deadline exceeded"
I see the following two lines in my GKE Cluster logs from my golang app containers. The error is intermittent (few errors every couple of days). I have been unable to find much information on it.
...
- 655
0
votes
1
answer
197
views
How to monitor client version distribution parsed from user-agent headers?
I have a native app that is configured to include a version number in the user-agent request header in a regex-parseable format. I'd like to include a metric in a Cloud Monitoring dashboard that shows ...
- 267
0
votes
2
answers
1k
views
GCP Error when performing Stackdriver AsyncReporter health check
My application runs in the following error when trying to perform Stackdriver AsyncReporter health check. I can't seem to find which authentication scopes are required for this from the documentation.
...
- 63
0
votes
1
answer
580
views
What is the unit of the Threshold value in Cloud Monitoring?
I am trying to wrap my head around Threshold value in the context of creating an alerting policy for an external HTTP load balancer:
Resource type: https_lb_rule
Metric: https/request_count
The docs ...
- 499
0
votes
0
answers
222
views
API for incident management
We use alert policies in order to monitor issues with some of our services. For a small subset of our alert policies, we'd like to react in an automated way to incidents raised by one of our alert ...
- 560
0
votes
0
answers
238
views
Group of GCP open incidents trigger 1 email notification
We have around 100 alerting policies and each incident triggers an email notification to relevant notification channels. I am looking for a way to send out 1 email notification based on the open ...
- 179
1
vote
1
answer
356
views
Specify a single gke cluster for gcp error reporting service
I've multiple gke clusters (stg, dev, prod), And I'm using the error reporting service to get alerts about errors, so is there anyway to filter the errors based on the gke cluster name, e.g (getting ...
0
votes
1
answer
903
views
How to relate a log-based-metric to a monitoring policy?
Initially I was trying to accomplish what I describe in my previous question:
How to create log-based-metric alert (email notification) with Google Cloud SDK?
Basically, only using the Google Cloud ...
- 103
0
votes
1
answer
1k
views
Fluentbit not parsing jsonPayload properly in Anthos cluster
Default fluentbit config for Anthos clusters on AWS
[FILTER]
Name kubernetes
Match k8s_container.*
Kube_URL https://kubernetes....
- 5,561
1
vote
1
answer
657
views
Logs from Airflow DAGs not written to GCP Cloud Logging
In our Airflow DAG code, we are trying to write the airflow logs directly into stack-driver using the sample code present in:
https://cloud.google.com/logging/docs/reference/libraries#log-entries
...
- 313
3
votes
1
answer
2k
views
GCP Cloud monitoring for Container Optimized OS
I'm using GCP compute Engine with Container Optimized OS. I enabled Logging and Monitoring on that instance.
Logging and Monitoring has write permissions
But it always show Not applicable for ...
- 1,533
1
vote
1
answer
2k
views
Missing logs from GKE on Cloud logging
I'm having an issue with some missing logs from GKE container in Cluod Logging.
I have an Spring boot application deployed on GKE with Log4j2. All the logs generated by the application are always ...
- 25
1
vote
1
answer
4k
views
How to view container logs in GKE's Logs Explorer
In my GKE cluster (version v1.21.10-gke.2000) I would like to see output produced by pods/containers as logs in Logs Explorer.
But clicking on Container Logs link in Workloads' Overview tab makes Logs ...
- 43
5
votes
1
answer
3k
views
NestJS logs not matching Google Cloud Stackdriver verbosity levels
I have a NestJS application that I am running on cloud run in Google Cloud (GCP). When viewing the logs in GCP from "Logs Explorer," I can see all log output from my NestJS app only when the ...
- 898
1
vote
2
answers
1k
views
GKE cluster creator in GCP
How can we get the cluster owner details in GKE. Logging part only contains the entry with service account operations and there is no entry with principal email of userId anywhere.
It seems very ...
- 425
2
votes
2
answers
2k
views
Have Log-Based Metric Automatically Determine Labels from JSON Payload of Filtered Logs
Background (skip to "The Problem" for end goal)
Take an example endpoint /process-batch that processes a batch of items given a single request. Upon receiving that request, it logs the ...
- 6,157
3
votes
0
answers
741
views
GCE Disk Utilization Metric Inactive even if Stackdriver is enabled
I have a Compute Engine Instance using a Container-optimized OS. I had setup stackdriver in order to track disk utilization of the VM instance.
I enabled the stackdriver monitoring agent so using the ...
- 31
0
votes
1
answer
901
views
Customize Error Reporting using Structured Logs and Monitoring/Alerting on GCP
I've added structured error reporting to my Google Cloud Function like mentioned here: https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs. I'm wondering if I can ...
- 8,076
0
votes
0
answers
106
views
Stackdriver API to update a policy
I would like to update a stackdriver alert policy using the monitor api with gcloud. I'm getting an error for the condition. I'm basically trying to add a filter to get rid of /dev/loop.* devices. ...
1
vote
0
answers
296
views
Google Cloud(Stackdriver): MQL Date Time conversion from UTC to EST
Was hoping if anyone of anyone could assist - basically I am experiencing an issue where DateTime format in my Payload is UTC(Non human-readable format) "2022-03-02T20:48:29.847676541Z".
Now ...
- 29
1
vote
1
answer
2k
views
How do I write a stack driver log query that samples 5% of all logs returned?
I keep reading that I can write a log query to sample a percentage of logs but I have found zero examples.
https://cloud.google.com/blog/products/gcp/preventing-log-waste-with-stackdriver-logging\
You ...
- 32.5k
1
vote
0
answers
139
views
Retrieve the Cloud dataflow custom counters from Grafana using Google Cloud Monitoring datasource
Am trying to read the custom counters from our Cloud Dataflow jobs into Grafana using Google Cloud Monitoring datasource but unable to do that (am able to get these from Stackdriver monitoring). Am ...
- 11
1
vote
2
answers
528
views
Top hitting IPs on my Google Compute engine instances
Any Stackdriver monitoring metric available to identify the top hitting IP Addresses on my Google Compute engine instances or Google Load Balancer Service .
Any other native GCP tool available which ...
- 1,554
0
votes
0
answers
461
views
Writing to Stackdriver can be slow
Lately, our team is occasionally experiencing really slow response when writing log messages to Stackdriver. We're using the .NET SDK Google.Cloud.Logging.V2 version 2.3.0 library. Measurement is done ...
- 160
1
vote
2
answers
5k
views
Error while installing ops agent on Google VM
I'm trying to install a Ops agent on a Google VM and some index files failed to download (error 404)
Command i used following documentation :
curl -sSO https://dl.google.com/cloudagents/add-google-...
- 21
1
vote
2
answers
1k
views
notification on different notification channel on different alert condition of same GCP metric
I have to create an alert on CPU utilization metric of a VM in GCP and it must follow the below condition:
first notification will send to email when threshold >50%
second notification will send ...
- 369