A Kafka poison-pill head-of-line block problem - #790

Summary

This PR adds kafka_poison_pill_hol_block, which is a Kafka poison-pill head-of-line block problem

A common production incident for Kafka-based microservice platforms is poison pill. It is a single malformed record that a consumer cannot deserialize. Because order is preserved within a partition, that one record blocks every valid message behind it. The offset is never committed. Meanwhile the lag grows, and yet every pod still reports healthy. Real outages have been attributed to this, including a malformed record that blocked a payment pipeline for around 45 minutes. So far, I could not find fault of this kind in SREGym. A poison pill lives in the Kafka log, outside the control plane, so a restarted consumer replays the bad record. This PR adds that scenario.

Relevant References:
Conduktor - Dead Letter Topics: Handling Poison Pills
Redpanda - Reliable message processing with a dead letter queue
Lydtech Consulting - Kafka Poison Pill

Code changes

3 files are added and one is edited:

• sregym/generators/fault/inject_kafka.py - fault injector. A self-contained producer and consumer are deployed into the astronomy-shop namespace, and one poison record is seeded into a dedicated topic at offset 20.
• sregym/conductor/oracles/data_plane_progress.py - DataPlaneProgressOracle, the mitigation oracle. Data-plane progress is graded from the consumer's logs.
• sregym/conductor/problems/kafka_poison_pill_hol_block.py - the problem definition.
• sregym/conductor/problems/registry.py - one import and one registry entry (see registry.patch.md).

After the problem starts and the diagnosis stage opens, an interval of about 1 to 2 minutes should be allowed before investigation begins, so the stalled consumer and the growing lag are clearly visible.

Validation

uv run python tests/integration/validate_problem.py --problem kafka_poison_pill_hol_block

Expected behaviour

While the fault is active, the consumer logs repeat unprocessable record at offset=20, the committed offset stays at 20, and lag grows, while every pod remains Running and Ready. Diagnosis is scored by the existing LLM judge. The mitigation oracle prints its checks and ends with either a pass or a clear failure reason (offset not advanced past poison record, data loss: valid records skipped, or not restart-resistant). Results are written to the cli.py results panel, or to the run CSV when main.py is used. Inspection commands used here are:

kubectl logs deployment/orders-validator -n astronomy-shop --tail=20  # ... unprocessable record at offset=20 ...
kubectl get pods -n astronomy-shop -l app=orders-validator # 1 pod running
kubectl logs deployment/order-stream  -n astronomy-shop --tail=5   # producer still streaming

Mitigation

The expected fix is to move the consumer group past the poison record without dropping the valid messages queued behind it. Since an offset reset requires the group to have no active members, the consumer is stopped first, the group offset is reset to 21 (one past the poison), and the consumer is started again:

kubectl scale deployment/orders-validator -n astronomy-shop --replicas=0
# now need to wait ~60s so the broker evicts the stopped consumer and the group goes inactive
kubectl run kafka-admin --rm --restart=Never -n astronomy-shop \
  --image=apache/kafka:3.9.0 -- \
  /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --group orders-validator --topic orders-fulfillment \
  --reset-offsets --to-offset 21 --execute

kubectl scale deployment/orders-validator -n astronomy-shop --replicas=1

The consumer resumes at offset 21, drains the backlog, and keeps pace with the producer. A plain restart is rejected, since the same offset is re-read and the stall returns. A reset to latest is rejected, since the valid backlog is skipped and counted as data loss.