Repository navigation
Reduce the set of metrics exposed by the kubelet #68522
Description
Background
In 1.12, the kubelet exposes a number of sources for metrics directly from cAdvisor. This includes:
- cAdvisor prometheus metrics at
/metrics/cadvisor - cAdvisor v1 Json API at
/stats/,/stats/container,/stats/{podName}/{containerName}, and/stats/{namespace}/{podName}/{uid}/{containerName} - cAdvisor machine info at
/spec
The kubelet also exposes the summary API, which is not exposed directly by cAdvisor, but queries cAdvisor as one of its sources for metrics.
The Monitoring Architecture documentation describes the path for "core" metrics, and for "monitoring" metrics. The Core Metrics proposal describes the set of metrics that we consider core, and their uses. The motivation for the split architecture is:
- To minimize the performance impact of stats collection for core metrics, allowing these to be collected more frequently
- To make the monitoring pipeline replaceable, and extensible.
Current kubelet metrics that are not included in core metrics
- Pod and Node-level Network Metrics
- Persistent Volume Metrics
- Container-level (Nvidia) GPU Metrics
- Node-Level RLimit Metrics
- Misc Memory Metrics (e.g. PageFaults)
- Container, Pod, and Node-level Inode metrics (for ephemeral storage)
- Container, Pod, and Node-level DiskIO metrics (from cAdvisor)
Deprecating and removing the Summary API will require out-of-tree sources for each of these metrics. "Direct" cAdvisor endpoints are not often used, and have even been broken for multiple releases (#62544) without anyone raising an issue.
Working Items
- [1.13] Introduce Kubelet
pod-resourcesgrpc endpoint; KEP: KEP: Support Device Monitoring community#2454 - [1.14] Introduce Kubelet Resource Metrics API
- [1.15] Deprecate the "direct" cAdvisor API endpoints by adding and deprecating a
--enable-cadvisor-json-endpointsflag - [1.18] Default the
--enable-cadvisor-json-endpointsflag to disabled - [1.21] Remove the
--enable-cadvisor-json-endpointsflag - [1.21] Transition Monitoring Server to Kubelet Resource Metrics API (requires 3 versions skew)
- [TBD] Propose out-of-tree replacements for kubelet monitoring endpoints
- [TBD] Deprecate the Summary API and cAdvisor prometheus endoints by adding and deprecating a
--enable-container-monitoring-endpointsflag - [TBD+2] Remove "direct" cAdvisor API endpoints
- [TBD+2] Default the
--enable-container-monitoring-endpointsflag to disabled - [TBD+4] Remove the Summary API, cAdvisor prometheus metrics and remove the
--enable-container-monitoring-endpointsflag.
Open Questions
- Should the kubelet be a source for any monitoring metrics?
- For example, metrics about the kubelet itself, or DiskIO metrics for empty-dir volumes (which are "owned" by the kubelet).
- What will provide the metrics listed above, now that the kubelet no longer does?
- cAdvisor can provide Network, RLimit, Misc Memory metrics, Inode metrics, and DiskIO metrics.
- cAdvisor only works for some runtimes, but is a drop-in replacement for "direct" cAdvisor API endpoints
- Container Runtimes can be a source for container-level Memory, Inode, Network and DiskIO metrics.
- NVidia GPU metrics provided by a daemonset published by NVidia
- No source for Persistent Volume metrics?
- cAdvisor can provide Network, RLimit, Misc Memory metrics, Inode metrics, and DiskIO metrics.
/sig node
/sig instrumentation
/kind feature
/priority important-longterm
cc @kubernetes/sig-node-proposals @kubernetes/sig-instrumentation-misc
Activity
jellonek commentedon Sep 12, 2018
Shouldn't be container/pod metrics gathered from CRI instead of from cAdvisor? Same with info about inodes/bytes usage of image store and for container root filesystems.
That's strongly runtime dependent and we should not expect that cAdvisor will support all types of runtimes supported by CRI (e.g. VM based runtimes like https://github.com/Mirantis/virtlet/ ).
DirectXMan12 commentedon Sep 12, 2018
cc @brancz
Is there a proposal out for this, besides the design/scratch doc I put out a while ago on using Prometheus metrics for this purpose?
brancz commentedon Sep 12, 2018
I don’t think there is anything else.
dashpole commentedon Sep 12, 2018
@jellonek you are correct. That is why I list cAdvisor as "one of its sources for metrics". The other notable source is metrics from the runtime through CRI.
I added the container runtime as a source of metrics that the kubelet won't provide in the future.
DirectXMan12 commentedon Sep 12, 2018
I don't think we should introduce the core metrics API as described in the proposal you linked above. It's basically just summary 2.0 in terms of format, which makes it highly kube-specific. I'd much rather just have a stable Prometheus endpoint, as discussed previously. That way, tools that collect metrics are much less likely to have to learn a new format.EDIT: too much speed reading
dashpole commentedon Sep 12, 2018
The format of the core metrics API is one of the non-goals listed in the proposal linked above. It just lists the contents of the API. We can debate format during the proposal process.
I intend for this to be an umbrella issue. I am trying to document current state, rough timeline, and gaps I see in terms of which metrics don't have a transition path to the new architecture.
DirectXMan12 commentedon Sep 12, 2018
Ah, apologies -- too much speed reading plus trying to recall from the last time I looked at things :-)
tianshapjq commentedon Sep 21, 2018
hi~I would like to know how to
Deprecate the "direct" cAdvisor API endpoints?Is that deprecating the port for direct cAdvisor in kubelet first, and then removing the cAdvisor vendor codes from kubelet after all?
DirectXMan12 commentedon Sep 24, 2018
removing the vendored cadvisor would require making sure we have out-of-tree sources for all the metrics. Deprecating "direct" cAdvisor API endpoints is mostly just about deprecating that piece of API, I believe. Removing the rest will come later.
91 remaining items
Add support for enable-cadvisor-json-endpoints with Kubelet
MadhavJivrajani commentedon Jun 29, 2021
/remove-kind design
/kind feature
kind/designwill soon be removed fromk/kin favor ofkind/feature. Relevant discussion can be found here: kubernetes/community#5641ganga1980 commentedon Jan 20, 2022
/track
yosizelensky commentedon Jun 15, 2023
/track
haq204 commentedon Mar 7, 2024
/track
dashpole commentedon Oct 3, 2025
Kubernetes users have come to rely on these metrics, and the project's state is more mature than when the issue was opened. At this point, it doesn't make sense to remove metrics endpoints that users have come to rely on.
Migrate oom watcher not relying on cAdviosr any more, it is part of plan
Default the --enable-cadvisor-endpoints flag to disabled