Skip to content

Commit bf3f220

Browse files
muhammetsafakmuhammetsafak
committed
feat: Redis delayed publish and zset poison DLQ, SQS delay, Kafka retry tiers and re-inject, conformance v1.6.0
Implements R1 Redis for the Python runtime: Laravel-layout delayed publish with a pcall-guarded reserve script, unconditional dead-lettering of poison bodies (binding 1.1), ADR-0034 retry backoff for SQS, Kafka retry tiers with a re-inject consumer, and the vendored conformance suite bumped to v1.6.0 with no known gaps.
1 parent 4aee52f commit bf3f220

24 files changed

Lines changed: 6223 additions & 1250 deletions

‎.github/workflows/ci.yml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,8 @@ jobs:
104104
105105
- name: Run full suite with coverage gate (>=90%)
106106
env:
107+
# A real redis:7 is mandatory here: no silent skip or fakeredis fallback (shared runner contract).
108+
BQ_REQUIRE_REDIS: "1"
107109
BABELQUEUE_TEST_REDIS: redis://localhost:6379/0
108110
BABELQUEUE_TEST_AMQP: amqp://guest:guest@localhost:5672/
109111
SQS_ENDPOINT: http://localhost:9324
@@ -162,9 +164,19 @@ jobs:
162164
conformance:
163165
name: Conformance suite in sync + results.json
164166
runs-on: ubuntu-latest
167+
services:
168+
redis:
169+
image: redis:7
170+
ports:
171+
- 6379:6379
172+
options: >-
173+
--health-cmd "redis-cli ping"
174+
--health-interval 5s
175+
--health-timeout 3s
176+
--health-retries 10
165177
env:
166178
# The pinned conformance release (ADR-0032). Bump together with tests/conformance/.
167-
BQ_CONFORMANCE_VERSION: 1.5.0
179+
BQ_CONFORMANCE_VERSION: 1.6.0
168180
# sha256 of that release's fetch.sh: the script runs on the runner, so it is pinned rather
169181
# than trusted from a mutable tag. Bump together with BQ_CONFORMANCE_VERSION.
170182
BQ_FETCH_SH_SHA256: 8bb6ba567b288fb0ac71a70fd17041b2503df119e01998d76fdad372570bf849
@@ -190,11 +202,13 @@ jobs:
190202
- name: Install
191203
run: |
192204
python -m pip install --upgrade pip
193-
pip install -e ".[dev,celery,django,otel]" opentelemetry-sdk jsonschema
205+
pip install -e ".[dev,celery,django,otel,redis]" opentelemetry-sdk jsonschema
194206
195207
- name: Run the conformance suite and write results.json
196208
env:
197209
BQ_CONFORMANCE_RESULTS: ${{ runner.temp }}/results.json
210+
BQ_REQUIRE_REDIS: "1"
211+
BABELQUEUE_TEST_REDIS: redis://localhost:6379/0
198212
run: pytest -q tests/test_conformance.py
199213

200214
- name: Validate results.json (schema, bindings, sections)

‎CHANGELOG.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,71 @@ The envelope wire format is versioned separately by `meta.schema_version`
99

1010
## [Unreleased]
1111

12+
Implements ADR-0033 (Redis) and ADR-0034 (Kafka tiers, back-off, poison). The envelope stays at
13+
`schema_version: 1`; the vendored conformance suite moves to v1.6.0 and
14+
`conformance-known-gaps.json` is now empty.
15+
16+
### Fixed
17+
- **Redis, Laravel-compatible layout: a poison reservation is dead-lettered in every
18+
configuration.** The DLQ write no longer depends on `dead_letter_enabled` (default `False`) or
19+
`on_unknown_urn`: an undecodable body or one without a numeric `attempts` is written raw to
20+
`<queue>.dlq` and acked instead of being logged and dropped (broker-bindings section 1.1,
21+
ADR-0033; same as the Go reference). Other version-gated routes keep their previous behaviour.
22+
- **Redis, Laravel-compatible layout: a job with a non-numeric `attempts` or an undecodable body no
23+
longer aborts the reserve script.** It is reserved raw (`pcall` guard), dead-lettered byte for
24+
byte and acked, leaving nothing in `:reserved` (ADR-0033 (c)).
25+
- **Redis, legacy layout: a `kill -9` no longer strands a message in `<queue>:processing`.** Every
26+
reservation writes a `<token>:<raw>` member into `<queue>:processing:deadlines`; a sweep before
27+
each reserve moves lapsed reservations back to the end of `<queue>`. Ack, release and dead-letter
28+
are gated on their own reservation, so a late ack never touches a newer one. The legacy mode
29+
remains the default and the `<queue>:processing` key name is unchanged.
30+
- **Kafka: a record without a valid `job` / `urn` is poison** (ADR-0034 section 10): dead-lettered
31+
wrapped with `dead_letter.reason: "poison"` and never retried, instead of being routed as an
32+
unknown URN.
33+
34+
- **Redis, Laravel-compatible layout: the reserve script no longer aborts on a body that decodes
35+
but cannot be re-encoded** (`cjson.encode` of `inf`/`NaN`, e.g. `1e400`, raised after the `LPOP`
36+
and lost the job). The encode is now inside `pcall`; the job is reserved raw and takes the poison
37+
path. Note: fakeredis' `cjson` serialises `inf` as `Infinity` and masks this; it is proven by a
38+
real-Redis test (CI `redis:7`) and by a Lua test against a failing `cjson` stub.
39+
- **`on_unknown_urn="delete"` no longer discards a Redis poison reservation.** It is dead-lettered
40+
byte for byte like the Kafka poison record already was; a genuine unknown URN is still deleted.
41+
- **Redis, Laravel-compatible layout: `dead_letter` is atomic and gated.** One script `ZREM`s the
42+
reservation and writes the dead-letter record and its notify entry; an expired reservation writes
43+
nothing (and logs a warning) instead of publishing a duplicate.
44+
- **A message dropped because it is undeliverable and no dead-letter queue is configured is
45+
logged** (`babelqueue.app`, WARNING) instead of disappearing silently.
46+
- **Kafka `RetryReinjector` no longer loses records:** it kept one held record per partition, so a
47+
second record on the same partition overwrote the first. It now keeps all of them in order.
48+
- **A worker on a tier-less Kafka transport with `max_attempts` > 1 warns at start-up** that the
49+
first handler failure stops the worker and leaves the record uncommitted.
50+
- Internal: the legacy `:processing` settle script takes `[token, raw]` and builds the deadline
51+
member itself.
52+
53+
### Added
54+
- `RedisTransport(retry_after=90, migration_batch_size=-1)` (also `?retry_after=` in the URL;
55+
`None` / `null` disables recovery). The Laravel-compatible `retry_after` default is now 90 s
56+
(was 60 s).
57+
- SQS: `SqsTransport.publish(..., delay_seconds=)` (`DelaySeconds` 0..900; out of range or FIFO
58+
raises `BabelQueueError`).
59+
- Kafka: `KafkaTransport(retry_tiers_ms=...)` with `DEFAULT_RETRY_TIERS_MS`, `retry_topic()` and
60+
the `babelqueue.kafka_retry.RetryReinjector` consumer that moves parked records from
61+
`<topic>.retry.<n>` back to the work topic (broker-bindings section 6.4). Opt-in: without tiers a
62+
retry / `publish_delayed` raises `BabelQueueError`.
63+
- Worker-side retry back-off: `BabelQueue(retry_backoff=, retry_backoff_multiplier=,
64+
retry_backoff_max=, retry_jitter=)` (ADR-0034 section 9); the defaults keep an immediate release.
65+
- Conformance runner covers the v1.6.0 sections (`redis_layout`, `redis_delay`, `retry_backoff`,
66+
`kafka_poison`, ...). CI runs the Redis cases against a real `redis:7` with `BQ_REQUIRE_REDIS=1`.
67+
68+
### Changed
69+
- **Kafka retries go to the retry tiers.** With `retry_tiers_ms` set a failed message is parked on
70+
`<queue>.retry.<n>` and needs a running `RetryReinjector`; the retry topics must exist. Without
71+
tiers a retry now raises instead of republishing onto the work topic.
72+
73+
### Documentation
74+
- README: Redis layouts, the A3 limit (no out-of-band headers in the Laravel-compatible layout),
75+
Kafka retry tiers, `LogAppendTime` on the tier topics, retry back-off, SQS `delay_seconds`.
76+
1277
## [1.21.1] - 2026-10-20
1378

1479
### Fixed

‎README.md‎

Lines changed: 79 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ app.run() # consume forever (Ctrl-C to stop)
141141
messages on `<queue>.dlq`. `on_unknown_urn` = `fail` | `delete` | `release` | `dead_letter`.
142142
- **Transports:** `redis://` (reliable-queue pattern; add `?laravel=1` to share a
143143
Redis queue with a Laravel BabelQueue worker using its reserved-set semantics —
144-
see below), `amqp://` (RabbitMQ via `pika`, with the contract AMQP properties)
144+
see below), `kafka://` (opt-in retry tiers, see below), `amqp://` (RabbitMQ via `pika`, with the contract AMQP properties)
145145
and `memory://` (in-process, great for tests/local). Bring your own by passing
146146
`transport=...`.
147147

@@ -194,7 +194,10 @@ transport **always** releases with `ChangeMessageVisibility` (broker-bindings
194194
attempt counter, so a shutdown release also consumes an attempt. The default
195195
delay is **0 s** (`retry_backoff` / `unknown_urn_release_delay`), i.e. the
196196
message is visible again immediately; delays outside 0..43200 s are clamped with
197-
a warning.
197+
a warning. A **per-message delay** at publish time is
198+
`SqsTransport.publish(queue, body, delay_seconds=n)` (`DelaySeconds`, 0..900 on a
199+
standard queue; a larger value, or any per-message delay on a FIFO queue, raises
200+
`BabelQueueError` instead of being truncated or dropped).
198201

199202
**Poison-loop risk:** a message that always fails is redelivered at once until
200203
`max_attempts` is reached. If the receive count is unavailable the SDK cannot
@@ -203,11 +206,68 @@ advance attempts at all. **Configure a native `RedrivePolicy`**
203206
queue as the broker-side backstop, and set a non-zero `retry_backoff` if the
204207
handler's dependencies need time to recover.
205208

209+
### Retry back-off
210+
211+
`BabelQueue(..., retry_backoff=, retry_backoff_multiplier=, retry_backoff_max=, retry_jitter=)`
212+
set the worker-side retry delay (ADR-0034 section 9): after the n-th failure the message is
213+
released with `d(n) = min(retry_backoff_max, retry_backoff * retry_backoff_multiplier ** (n - 1))`
214+
seconds (`retry_backoff_max=0` means no cap) and `retry_jitter="full"` draws `uniform(0, d(n))`.
215+
The defaults (`retry_backoff=0`, multiplier 1, jitter `"none"`) keep the previous behaviour: an
216+
immediate release. How the delay is honoured depends on the transport: SQS uses it as the
217+
visibility timeout, the Laravel-compatible Redis layout as the `:delayed` score and Kafka picks a
218+
retry tier (see below).
219+
220+
### Kafka retry tiers and the re-inject consumer
221+
222+
Kafka has no per-message delay, so delayed delivery and retry back-off are SDK-owned **retry
223+
topics** (broker-bindings section 6.4, ADR-0034). They are opt-in: pass `retry_tiers_ms` to the
224+
transport (the recommended set is `DEFAULT_RETRY_TIERS_MS` = 5 s, 30 s, 5 min, 30 min):
225+
226+
```python
227+
from babelqueue import BabelQueue
228+
from babelqueue.kafka_transport import KafkaTransport, DEFAULT_RETRY_TIERS_MS
229+
from babelqueue.kafka_retry import RetryReinjector
230+
231+
transport = KafkaTransport("kafka://localhost:9092", retry_tiers_ms=DEFAULT_RETRY_TIERS_MS)
232+
app = BabelQueue(transport=transport, queue="orders", max_attempts=5, dead_letter=True)
233+
234+
# Run one next to the workers, otherwise a parked message is never delivered again:
235+
RetryReinjector("kafka://localhost:9092", work_topics=["orders"],
236+
retry_tiers_ms=DEFAULT_RETRY_TIERS_MS).run()
237+
```
238+
239+
A failed message is republished (attempts + 1) to `<queue>.retry.<n>`, where
240+
`n = min(attempts, N - 1) + 1`; `publish_delayed` picks the smallest tier that covers the requested
241+
delay. Create the `<queue>.retry.<n>` topics up front. The `RetryReinjector` copies each record's
242+
value and key byte for byte back to its work topic once the tier delay has elapsed (headers
243+
`bq-delay` / `bq-original-topic` are dropped) and commits only after the broker confirmed the
244+
write, so delivery stays at-least-once. It holds every record it has accepted per partition (oldest
245+
first, a record never overtakes the one before it) and a failed write rewinds the partition to the
246+
failed record. Without `retry_tiers_ms`, `publish_delayed` and a retry
247+
raise `BabelQueueError` and nothing is produced. A worker started without tiers while
248+
`max_attempts` is above 1 logs a warning at start-up: the first handler failure stops that worker
249+
and leaves the record uncommitted (it is not lost, but it is not retried either) until tiers are
250+
configured. The re-inject delay is measured from the record
251+
timestamp: set `message.timestamp.type=LogAppendTime` on the `.retry.<n>` topics so that
252+
timestamp is the broker's, not a producer clock's.
253+
254+
**Records without an identity are poison.** A Kafka record whose JSON object has neither a valid
255+
string `job` nor a valid string `urn` (ADR-0034 section 10) is dead-lettered as poison (wrapped with
256+
`dead_letter.reason: "poison"`, key preserved) and never retried; it is not routed by
257+
`on_unknown_urn`.
258+
206259
### Sharing a Redis queue with Laravel
207260

208261
By default the Redis transport owns its queue end-to-end (`RPUSH` to produce;
209-
`BLMOVE` into a `<queue>:processing` list to reserve; `LREM` on ack). To consume a
210-
**shared** Laravel BabelQueue Redis queue instead, enable Laravel-compatible mode:
262+
`BLMOVE` into a `<queue>:processing` list to reserve; `LREM` on ack). Each reservation also
263+
writes a `<token>:<raw>` member (32 lowercase hex characters of token) into the
264+
`<queue>:processing:deadlines` sorted set, and before every reserve a sweep moves the lapsed
265+
reservations back to the end of the list, so a `kill -9` no longer strands a message in
266+
`:processing`; a late ack of a swept reservation touches nothing of the newer one. `retry_after`
267+
(default **90** seconds; `None` / `?retry_after=null` disables recovery) sets that deadline, and
268+
`migration_batch_size` (`-1` = all, otherwise `>= 1`) bounds how many due members one sweep or
269+
migration moves. To consume a **shared** Laravel BabelQueue Redis queue instead, enable
270+
Laravel-compatible mode:
211271

212272
```python
213273
app = BabelQueue("redis://localhost:6379/0?laravel=1", queue="orders")
@@ -224,8 +284,21 @@ member a Python worker writes is identical to a Laravel worker's — either side
224284
ack (`ZREM`) the other's reservation, so a Python worker and a Laravel worker share
225285
one queue without losing or double-processing messages. Before each pop, expired
226286
reserved/delayed jobs migrate back to the ready list, so a crashed worker's
227-
in-flight job is re-reserved. Options: `?prefix=queues:` and `?retry_after=60`
228-
(or the `key_prefix` / `retry_after` constructor args).
287+
in-flight job is re-reserved. Options: `?prefix=queues:` and `?retry_after=90`
288+
(or the `key_prefix` / `retry_after` / `migration_batch_size` constructor args). A job whose body
289+
does not decode, or whose `attempts` is not a number, is reserved raw, dead-lettered byte for byte
290+
and acked, so it never lingers in `:reserved`. That holds when the body cannot be re-encoded
291+
either (for example a number such as `1e400`, which real Redis' `cjson` refuses to serialise), and
292+
it holds under `on_unknown_urn="delete"` too: such a message is dead-lettered, never deleted. Without
293+
a dead-letter queue it is dropped, and the drop is logged as a warning. Dead-lettering a
294+
reserved message is one atomic script (`ZREM` the reservation, then `RPUSH` the dead-letter
295+
record and its notify entry); if the reservation expired first, nothing is written and a warning
296+
says so, because the job was already migrated back and will be delivered again.
297+
298+
**Limit (A3):** the Laravel-compatible layout stores the bare envelope (Laravel's reserve script
299+
has to decode it), so out-of-band transport headers such as the W3C `traceparent` cannot be
300+
carried there. They are dropped without an error; the `trace_id` inside the envelope is still
301+
preserved. The default (legacy) layout carries them.
229302

230303
## Framework adapters — Celery & Django
231304

‎conformance-known-gaps.json‎

Lines changed: 1 addition & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,84 +1 @@
1-
{
2-
"gaps": [
3-
{
4-
"section": "delay",
5-
"case": "standard-1s",
6-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
7-
},
8-
{
9-
"section": "delay",
10-
"case": "standard-60s",
11-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
12-
},
13-
{
14-
"section": "delay",
15-
"case": "standard-900s-is-the-cap",
16-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
17-
},
18-
{
19-
"section": "delay",
20-
"case": "standard-901s-rejected",
21-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
22-
},
23-
{
24-
"section": "delay",
25-
"case": "standard-3600s-rejected",
26-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
27-
},
28-
{
29-
"section": "delay",
30-
"case": "fifo-per-message-delay-rejected",
31-
"reason": "Python SqsTransport has no per-message DelaySeconds publish path (broker-bindings section 3.4, ADR-0012); not implemented in this core, tracked in the r1 follow-up plan"
32-
},
33-
{
34-
"section": "backoff",
35-
"case": "below-first-tier",
36-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
37-
},
38-
{
39-
"section": "backoff",
40-
"case": "exactly-first-tier",
41-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
42-
},
43-
{
44-
"section": "backoff",
45-
"case": "just-above-first-tier",
46-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
47-
},
48-
{
49-
"section": "backoff",
50-
"case": "exactly-third-tier",
51-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
52-
},
53-
{
54-
"section": "backoff",
55-
"case": "exactly-largest-tier",
56-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
57-
},
58-
{
59-
"section": "backoff",
60-
"case": "above-largest-tier-rejected",
61-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
62-
},
63-
{
64-
"section": "backoff",
65-
"case": "no-tiers-configured-rejected",
66-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
67-
},
68-
{
69-
"section": "backoff",
70-
"case": "retry_republish/first-retry",
71-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
72-
},
73-
{
74-
"section": "backoff",
75-
"case": "retry_republish/third-retry-picks-larger-tier",
76-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
77-
},
78-
{
79-
"section": "backoff",
80-
"case": "retry_republish/no-tiers-configured-rejected",
81-
"reason": "Python KafkaTransport has no retry-topic tier routing (broker-bindings sections 6.4/6.5, ADR-0015); not implemented in this core, tracked in the r1 follow-up plan"
82-
}
83-
]
84-
}
1+
{"gaps": []}

‎src/babelqueue/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
from .routing import UnknownUrnStrategy
3232
from .transport import (
3333
BatchPopper,
34+
DeadLetterWriter,
3435
HeaderPublisher,
3536
InMemoryTransport,
3637
ReceivedMessage,
@@ -55,6 +56,7 @@
5556
"InMemoryTransport",
5657
"ReceivedMessage",
5758
"BatchPopper",
59+
"DeadLetterWriter",
5860
"HeaderPublisher",
5961
"Redeliverer",
6062
"Rewinder",

0 commit comments

Comments
 (0)