Skip to content

Event Loop Additional Metrics to make event-loop stalls more visible #67649

Open

Description

@mateusz-nalepa

What is the problem this feature will solve?

Hey!
This feature will help with more precise detection of long-running tasks in Node.js event loop.
When calculating p99, it will help make long event-loop stalls more visible

What is the feature you are proposing to solve the problem?

My proposal is to add additional metrics like:

  • exceededIntervalCounter / okIntervalCounter
    • to calculate ratio between both of them
  • exceededIntervalDelay

Thanks to them, it would be easier to detect event-loop stalls.
Cause ok samples and exceeded samples are treated separately.

Before going further let me first explain current metrics, to be sure, that my understanding is correct.
Cause maybe there is something, I still don't understand well and this issue doesn't make sense.

Note: During issue I'm writing typically about task duration. But I'm aware that more precise interpretation is event loop probe delay.

Event Loop Utilisation

My understanding of Event Loop Utilization (ELU) is:

  • ELU isn't aware about task duration
  • ELU is similar to CPU utilization, but it's not CPU utilization
  • ELU like 85% may mean:
    • there is a problem with event loop
    • there is no problem with event loop, all tasks are short and event loop is healthy
  • ELU is great to detect skew in node.js cluster mode
    • worker1: ELU 20%
    • worker2: ELU 15%
    • worker3: ELU 85% - probably something wrong with this worker
utilization

Event Loop Delay

My understanding of Event Loop Delay (ELD) is:

  • when calculating p99
    • it can be low, because a lot of short tasks can hide long tasks
  • max is always helpful
  • it's rather like: catastrophe detector

Code looks like this:

function monitorSample(diffNanos) {
  delay.observe(diffNanos / 1e9);
  // delay.observe((diffNanos - intervalNanos ) / 1e9); i'm not sure
}
event_loop_delay

About those metrics:

  • exceededIntervalCounter
  • exceededIntervalDelay

My understanding of those Event Loop those metrics is:

  • exceededIntervalCounter ( together with okIntervalCounter )
    • how often event loop was doing tasks longer than interval?
      • to be more precise: how often probe wasn't able to execute?
      • okIntervalCounter is needed to calculate ratio
      • exceededRatio = exceededIntervalCounter / (okIntervalCounter + exceededIntervalCounter)
  • exceededIntervalDelay
    • if probe wasn't able to execute, how long it was delayed?

Let's assume interval 10ms to describe event loop status:

  • event loop idle or tasks below 10ms
    • always healthy sample
  • task between 10ms and 20ms - healthy or unhealthy sample
    • cause we don't know when task started
  • task higher than 20ms
    • always unhealthy sample

Code could look like this:

function monitorSample(diffNanos) {
  const lagNanos = Math.max(0, diffNanos - intervalNanos - intervalNanos);

  if (lagNanos > 0) {
    const missedSamples = Math.floor(diffNanos / intervalNanos);
    exceededCounter.inc(missedSamples);
    exceededHistogram.observe(lagNanos / 1e9);
  } else {
    okCounter.inc();
  }
}
exceeded

The best part in all of that

  • all of those metrics still do not directly measure end-user latency
    • splitting one big task into smaller tasks may cause tasks overlap
    • from logical point of view, there is a lot of concurrent cpu tasks
  • so higher response times during day, faster during night, may suggest, that tasks are overlapping
    • we every healthy event loop metric won't tell anything about that
yield_and_semaphore

What alternatives have you considered?

Just use current Event Loop Utilization and Event Loop Delay metrics.

I'm open to your feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestIssues requesting new Node.js features.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions