[performance] tail plugin exhibits high CPU usage with inotify_watcher enabled #11096
Description
Bug Report
I am conducting a performance test of various log collectors and noticed that Fluent Bit's CPU usage is significantly higher than that of Grafana Alloy. Even after following the official performance tips, there was no significant improvement.
Describe the bug
All collectors are configured to gather logs from the exact same source (Kubernetes) and are running on the same node for an accurate comparison.
- Initial Comparison (with default configuration, implicitly
inotify_watcher: true)
Fluent Bit's CPU usage is approximately 2x that of Alloy's in this comparison.
$ kubectl top pod
NAME CPU(cores) MEMORY(bytes)
alloy-xl92g 57m 115Mi
fluent-bit-sphqq 117m 16Mi
promtail-d7nvd 68m 67Mi
- Comparison after setting
inotify_watcher: false
After setting inotify_watcher: false in the [INPUT:tail] configuration, I observed a dramatic improvement in performance.
$ kubectl top pod
NAME CPU(cores) MEMORY(bytes)
alloy-xl92g 60m 121Mi
fluent-bit-w26tz 13m 29Mi
promtail-d7nvd 71m 66Mi
nb@nb:~/data/github/internal-lab/charts$
The CPU usage of Fluent Bit dropped from 117m to 13m, indicating that the inotify_watcher mechanism in the tail plugin is causing unexpectedly high CPU consumption under my testing conditions.
To Reproduce
install alloy with config
discovery.kubernetes "pods" {
role = "pod"
}
loki.source.kubernetes "pods" {
targets = discovery.kubernetes.pods.targets
forward_to = [loki.write.local.receiver]
}
loki.write "local" {
endpoint {
url = "http://loki-write.monitor.svc:3100/loki/api/v1/push"
tenant_id = "alloy"
}
}
install Fluent Bit with the configuration below
Your Environment
- Version used: 4.1.1
- Configuration:
[SERVICE]
Daemon Off
Flush 1
Log_Level error
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
json.escape_unicode Off
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Mem_Buf_Limit 128MB
buffer_chunk_size 5MB
buffer_max_size 25MB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name loki
Match kube.*
Host loki-write.monitor.svc
Port 3100
Tenant_ID default
TLS Off
- Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.33.5
- Server type and version: arm64
- Operating System and version: ubuntu 24.04