Repository navigation
Description
What is the problem this feature will solve?
Hey!
This feature will help with more precise detection of long-running tasks in Node.js event loop.
When calculating p99, it will help make long event-loop stalls more visible
What is the feature you are proposing to solve the problem?
My proposal is to add additional metrics like:
- exceededIntervalCounter / okIntervalCounter
- to calculate ratio between both of them
- exceededIntervalDelay
Thanks to them, it would be easier to detect event-loop stalls.
Cause ok samples and exceeded samples are treated separately.
Before going further let me first explain current metrics, to be sure, that my understanding is correct.
Cause maybe there is something, I still don't understand well and this issue doesn't make sense.
Note: During issue I'm writing typically about task duration. But I'm aware that more precise interpretation is event loop probe delay.
Event Loop Utilisation
My understanding of Event Loop Utilization (ELU) is:
- ELU isn't aware about task duration
- ELU is similar to CPU utilization, but it's not CPU utilization
- ELU like 85% may mean:
- there is a problem with event loop
- there is no problem with event loop, all tasks are short and event loop is healthy
- ELU is great to detect skew in node.js cluster mode
- worker1: ELU 20%
- worker2: ELU 15%
- worker3: ELU 85% - probably something wrong with this worker
Event Loop Delay
My understanding of Event Loop Delay (ELD) is:
- when calculating
p99- it can be low, because a lot of short tasks can hide long tasks
maxis always helpful- it's rather like: catastrophe detector
Code looks like this:
function monitorSample(diffNanos) {
delay.observe(diffNanos / 1e9);
// delay.observe((diffNanos - intervalNanos ) / 1e9); i'm not sure
}
About those metrics:
- exceededIntervalCounter
- exceededIntervalDelay
My understanding of those Event Loop those metrics is:
- exceededIntervalCounter ( together with okIntervalCounter )
- how often event loop was doing tasks longer than interval?
- to be more precise: how often probe wasn't able to execute?
okIntervalCounteris needed to calculate ratioexceededRatio = exceededIntervalCounter / (okIntervalCounter + exceededIntervalCounter)
- how often event loop was doing tasks longer than interval?
- exceededIntervalDelay
- if probe wasn't able to execute, how long it was delayed?
Let's assume interval 10ms to describe event loop status:
- event loop idle or tasks below 10ms
- always healthy sample
- task between 10ms and 20ms - healthy or unhealthy sample
- cause we don't know when task started
- task higher than 20ms
- always unhealthy sample
Code could look like this:
function monitorSample(diffNanos) {
const lagNanos = Math.max(0, diffNanos - intervalNanos - intervalNanos);
if (lagNanos > 0) {
const missedSamples = Math.floor(diffNanos / intervalNanos);
exceededCounter.inc(missedSamples);
exceededHistogram.observe(lagNanos / 1e9);
} else {
okCounter.inc();
}
}
The best part in all of that
- all of those metrics still do not directly measure end-user latency
- splitting one big task into smaller tasks may cause tasks overlap
- from logical point of view, there is a lot of concurrent cpu tasks
- so higher response times during day, faster during night, may suggest, that tasks are overlapping
- we every healthy event loop metric won't tell anything about that
What alternatives have you considered?
Just use current Event Loop Utilization and Event Loop Delay metrics.
I'm open to your feedback!
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsAwaiting Triage
Activity