Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@ The envelope wire format is versioned separately by `meta.schema_version`

## [Unreleased]

## [1.4.0] - 2026-11-03

### Added

- **Dead-letter observer (ADR-0039 section 5.1).** `PulsarConsumer.Builder#deadLetterObserver(DeadLetterObserver)` takes the core's optional `com.babelqueue.DeadLetterObserver` (needs a `babelqueue-core` that ships it). It is called once per record written to the DLQ by the dead-letter producer, right after the write succeeded: an exhausted retry (`failed`), an unmapped URN (`unknown_urn`), a poison body (`poison`) and a raw version-gated body (empty `reason`). `DeadLetterEvent.queue` is `meta.queue` for an exhausted envelope, else the topic name the message came from; `deadLetterQueue` is the dead-letter producer's topic. It is not called when the write fails, when no dead-letter producer is set (the drop), or while the retry cap is not reached; an observer failure is logged and never changes the settle outcome. Without an observer nothing changes.
- **W3C `traceparent` out-of-band propagation (ADR-0028).** `PulsarPublisher`: `publishWithHeaders(Envelope, Map<String,String>)` (the produce-side seam for the core's `com.babelqueue.otel.HeaderSender`) and a `publish(..., Map<String,String> headers)` overload carry out-of-band headers (e.g. `traceparent`, `tracestate`) as Pulsar message properties beside the contract `bq-` entries. A contract entry always wins a collision; blank keys/values are skipped; with no/empty headers the message is identical to a plain `publish`. The envelope body is untouched (GR-1, `schema_version` stays 1).
- `BabelHandler.headersOf(Message<byte[]>)` surfaces the delivered Pulsar message properties as a `Map<String,String>` — the `Supplier` source for `Tracing.wrapHandler(tracer, handler, headers)`, so the consumer span becomes a true child of the producer span. No OpenTelemetry dependency is added (the core's OTel API stays optional).

## [1.3.1] - 2026-10-26

### Changed
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ the .NET, Python, Go and Node SDKs. Implements
<dependency>
<groupId>com.babelqueue</groupId>
<artifactId>babelqueue-pulsar</artifactId>
<version>1.3.1</version>
<version>1.4.0</version>
</dependency>
```

Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

<groupId>com.babelqueue</groupId>
<artifactId>babelqueue-pulsar</artifactId>
<version>1.3.1</version>
<version>1.4.0</version>
<packaging>jar</packaging>

<name>BabelQueue Apache Pulsar</name>
Expand Down Expand Up @@ -49,7 +49,7 @@
<maven.compiler.release>17</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<babelqueue-core.version>1.14.0</babelqueue-core.version>
<babelqueue-core.version>1.15.0</babelqueue-core.version>
<pulsar.version>3.3.5</pulsar.version>
<junit.version>5.10.3</junit.version>
<mockito.version>5.12.0</mockito.version>
Expand Down
27 changes: 26 additions & 1 deletion src/main/java/com/babelqueue/pulsar/BabelHandler.java
Original file line number Diff line number Diff line change
@@ -1,9 +1,24 @@
package com.babelqueue.pulsar;

import com.babelqueue.Envelope;
import java.util.Map;
import org.apache.pulsar.client.api.Message;

/** Processes one decoded, validated envelope and the raw Pulsar message it arrived on. */
/**
* Processes one decoded, validated envelope and the raw Pulsar message it arrived on.
*
* <p>The raw message carries the out-of-band transport headers (ADR-0028) as properties.
* {@link #headersOf} surfaces them as a plain map, ready for the optional OpenTelemetry wrapper so the
* consumer span becomes a true child of the producer span:
*
* <pre>{@code
* PulsarConsumer.builder(consumer)
* .handler(urn, (env, message) -> Tracing
* .wrapHandler(tracer, h, () -> BabelHandler.headersOf(message))
* .handle(env))
* .build();
* }</pre>
*/
@FunctionalInterface
public interface BabelHandler {

Expand All @@ -16,4 +31,14 @@ public interface BabelHandler {
* acknowledged and dropped otherwise (ADR-0036).
*/
void handle(Envelope envelope, Message<byte[]> message) throws Exception;

/**
* The message's properties as a {@code Map<String, String>} (blank values dropped) — the
* delivered-headers {@code Supplier} source for {@code com.babelqueue.otel.Tracing#wrapHandler}.
* Contains the contract {@code bq-} properties and any out-of-band header (e.g. {@code traceparent},
* {@code tracestate}). Empty when {@code message} is {@code null}. Needs no OpenTelemetry dependency.
*/
static Map<String, String> headersOf(Message<byte[]> message) {
return PulsarProperties.toMap(message);
}
}
58 changes: 52 additions & 6 deletions src/main/java/com/babelqueue/pulsar/PulsarConsumer.java
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package com.babelqueue.pulsar;

import com.babelqueue.BabelQueueException;
import com.babelqueue.DeadLetterEvent;
import com.babelqueue.DeadLetterObserver;
import com.babelqueue.DeadLetters;
import com.babelqueue.Envelope;
import com.babelqueue.EnvelopeCodec;
Expand Down Expand Up @@ -77,6 +79,9 @@
* with a valid {@code job}/{@code urn} identity — is never handed to a handler. With a dead-letter producer it is
* written to the DLQ at once, without retry, in the ADR-0035 shape ({@code reason: "poison"}, the original body
* preserved), then acknowledged; without one it takes the bounded retry above and is finally dropped.
*
* <p>An optional {@link Builder#deadLetterObserver DeadLetterObserver} (ADR-0039 section 5.1) is called once per DLQ
* write the dead-letter producer confirmed; see its javadoc for what is and is not counted.
*/
public final class PulsarConsumer {

Expand Down Expand Up @@ -121,6 +126,7 @@ public interface RoutingMismatchHandler {
private final RoutingMismatchHandler onRoutingMismatch;
private final int maxAttempts;
private final Producer<byte[]> deadLetterProducer;
private final DeadLetterObserver deadLetterObserver;
private final Producer<byte[]> retryProducer;
private final PulsarClient client;
/** Retry producers this consumer created from {@code client}, by topic; closed by {@link #close()}. */
Expand All @@ -135,6 +141,7 @@ private PulsarConsumer(Builder builder) {
this.onRoutingMismatch = builder.onRoutingMismatch;
this.maxAttempts = builder.maxAttempts;
this.deadLetterProducer = builder.deadLetterProducer;
this.deadLetterObserver = builder.deadLetterObserver;
this.retryProducer = builder.retryProducer;
this.client = builder.client;
}
Expand Down Expand Up @@ -262,7 +269,7 @@ record = EnvelopeCodec.encode(annotated).getBytes(StandardCharsets.UTF_8);
leaveUnsettled(message, envelope, error);
return;
}
writeDeadLetterAndAcknowledge(message, envelope, record, properties);
writeDeadLetterAndAcknowledge(message, envelope, record, properties, () -> eventQueue(message, envelope), reason);
}

/**
Expand All @@ -288,7 +295,8 @@ private void settleVersionGated(Message<byte[]> message, String declaredVersion)
}
byte[] body = message.getValue() == null ? new byte[0] : message.getValue();
Map<String, String> original = message.getProperties();
writeDeadLetterAndAcknowledge(message, null, body, original == null ? Map.of() : original);
writeDeadLetterAndAcknowledge(message, null, body, original == null ? Map.of() : original,
() -> queueName(message), "");
}

/**
Expand Down Expand Up @@ -318,22 +326,38 @@ record = PoisonRecord.build(body, queueName(message), reconciled, cause.getMessa
return;
}
Map<String, String> original = message.getProperties();
writeDeadLetterAndAcknowledge(message, partial, record, original == null ? Map.of() : original);
writeDeadLetterAndAcknowledge(message, partial, record, original == null ? Map.of() : original,
() -> queueName(message), "poison");
}

/** Sends {@code record} to the DLQ; the original is acknowledged only after the write succeeded. */
/**
* Sends {@code record} to the DLQ; the original is acknowledged only after the write succeeded. The
* {@link DeadLetterObserver} (ADR-0039 section 5.1) is told once, right after the confirmed write, with the source
* {@code queue} and the {@code reason} the record carries (empty for a raw version-gated body); a failed write
* never reaches it.
*/
private void writeDeadLetterAndAcknowledge(
Message<byte[]> message, Envelope envelope, byte[] record, Map<String, String> properties)
throws PulsarClientException {
Message<byte[]> message, Envelope envelope, byte[] record, Map<String, String> properties,
java.util.function.Supplier<String> queue, String reason) throws PulsarClientException {
try {
send(deadLetterProducer, record, properties);
} catch (PulsarClientException | RuntimeException error) {
leaveUnsettled(message, envelope, error);
return;
}
notifyDeadLetter(queue, reason);
consumer.acknowledge(message);
}

/** Reports a DLQ write the producer just confirmed (ADR-0039 section 5.1); an observer failure never escapes. */
private void notifyDeadLetter(java.util.function.Supplier<String> queue, String reason) {
if (deadLetterObserver == null) {
return;
}
DeadLetterObserver.notifySafely(deadLetterObserver,
new DeadLetterEvent(queue.get(), deadLetterProducer.getTopic(), reason));
}

/**
* Retries a delivery. With a {@link Builder#retryProducer retry producer} the message is re-published to the
* same topic with {@code bq-attempts = counted} (and the body's {@code attempts} in sync for an envelope; a poison
Expand Down Expand Up @@ -439,6 +463,16 @@ private String originalQueue(Message<byte[]> message, Envelope envelope) {
return queue != null && !queue.isBlank() ? queue : queueName(message);
}

/**
* The source queue of a dead-letter <em>event</em> (ADR-0039 section 5.1): {@code meta.queue} unless it is empty or
* ASCII-blank (ADR-0031), else the consumed queue. Deliberately separate from {@link #originalQueue}: the DLQ
* record's {@code original_queue} is wire data and keeps its own rule.
*/
private String eventQueue(Message<byte[]> message, Envelope envelope) {
String queue = envelope.meta() == null ? null : envelope.meta().queue();
return !EnvelopeCodec.isBlankAscii(queue) ? queue : queueName(message);
}

/** The short queue name of the consumed topic: tenant/namespace and a {@code -partition-N} suffix removed. */
private String queueName(Message<byte[]> message) {
String topic = message.getTopicName();
Expand Down Expand Up @@ -557,6 +591,7 @@ public static final class Builder {
private RoutingMismatchHandler onRoutingMismatch;
private int maxAttempts = DEFAULT_MAX_ATTEMPTS;
private Producer<byte[]> deadLetterProducer;
private DeadLetterObserver deadLetterObserver;
private Producer<byte[]> retryProducer;
private PulsarClient client;
private SubscriptionType subscriptionType;
Expand Down Expand Up @@ -659,6 +694,17 @@ public Builder subscriptionType(SubscriptionType subscriptionType) {
return this;
}

/**
* Optional observer called once per record written to the DLQ by the {@link #deadLetterProducer dead-letter
* producer}, after the write is confirmed (ADR-0039 section 5.1): an exhausted retry, an unknown URN, a poison
* body, a raw version-gated body. Not called when the write fails or when no dead-letter producer is set
* (the drop). A failure of the observer is logged and never changes the settle outcome.
*/
public Builder deadLetterObserver(DeadLetterObserver deadLetterObserver) {
this.deadLetterObserver = deadLetterObserver;
return this;
}

/**
* @throws IllegalStateException when neither a {@link #retryProducer} nor a {@link #client} is given and the
* declared {@link #subscriptionType} is not {@code Shared} or {@code Key_Shared} (including when it
Expand Down
43 changes: 43 additions & 0 deletions src/main/java/com/babelqueue/pulsar/PulsarProperties.java
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
package com.babelqueue.pulsar;

import com.babelqueue.Envelope;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.apache.pulsar.client.api.Message;

/**
* Projects the envelope's contract fields onto native Pulsar message properties
Expand All @@ -28,6 +31,46 @@ static Map<String, String> of(Envelope envelope) {
return props;
}

/**
* Adds the out-of-band {@code extra} headers (e.g. {@code traceparent}, ADR-0028) to {@code props}
* beside the contract properties. Blank keys/values are skipped, a key already present always wins
* (the contract is never clobbered), and keys are added in sorted order so the result is deterministic.
* A {@code null}/empty map leaves {@code props} untouched.
*/
static void addOutOfBand(Map<String, String> props, Map<String, String> extra) {
if (extra == null || extra.isEmpty()) {
return;
}
List<String> keys = new ArrayList<>(extra.keySet());
keys.removeIf(k -> k == null || k.isEmpty());
keys.sort(String::compareTo);
for (String key : keys) {
String value = extra.get(key);
if (value != null && !value.isEmpty() && !props.containsKey(key)) {
props.put(key, value);
}
}
}

/**
* A delivered message's properties as a flat {@code Map<String, String>} (blank values dropped) — the
* consume-side counterpart of {@link #addOutOfBand}. Both the contract {@code bq-} properties and
* out-of-band ones (e.g. {@code traceparent}) surface. Empty when the message is {@code null} or has none.
*/
static Map<String, String> toMap(Message<byte[]> message) {
Map<String, String> out = new LinkedHashMap<>();
Map<String, String> properties = message == null ? null : message.getProperties();
if (properties == null) {
return out;
}
for (Map.Entry<String, String> e : properties.entrySet()) {
if (e.getKey() != null && e.getValue() != null && !e.getValue().isEmpty()) {
out.put(e.getKey(), e.getValue());
}
}
return out;
}

private static void put(Map<String, String> props, String key, String value) {
if (value != null && !value.isEmpty()) {
props.put(key, value);
Expand Down
40 changes: 37 additions & 3 deletions src/main/java/com/babelqueue/pulsar/PulsarPublisher.java
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@
* String id = PulsarPublisher.create(producer)
* .publish("urn:babel:orders:created", Map.of("order_id", 1042));
* }</pre>
*
* <p>Out-of-band transport headers (e.g. the W3C {@code traceparent}, ADR-0028) ride as extra message
* properties <b>beside</b> the contract {@code bq-} properties — never inside the envelope (GR-1); see
* {@link #publishWithHeaders(Envelope, Map)}, the seam for {@code com.babelqueue.otel.HeaderSender}.
*/
public final class PulsarPublisher {

Expand Down Expand Up @@ -53,15 +57,45 @@ public String publish(String urn, Map<String, Object> data, String traceId) thro
*/
public String publish(String urn, Map<String, Object> data, String traceId, Duration delay)
throws PulsarClientException {
return publish(urn, data, traceId, delay, null);
}

/**
* Publish as {@link #publish(String, Map, String, Duration)}, additionally carrying the
* out-of-band {@code headers} (e.g. {@code traceparent}) as message properties beside the contract
* {@code bq-} properties. A contract property already present always wins a collision; blank keys or
* values are skipped; {@code null}/empty {@code headers} is byte-identical to the plain overload.
*/
public String publish(String urn, Map<String, Object> data, String traceId, Duration delay,
Map<String, String> headers) throws PulsarClientException {
Envelope envelope = EnvelopeCodec.make(urn, data, topicName(), traceId);
return send(envelope, delay, headers);
}

/**
* Publish an already-built {@code envelope} together with out-of-band transport {@code headers}
* (e.g. a W3C {@code traceparent}, ADR-0028). This is the produce-side seam the optional
* {@code com.babelqueue.otel.HeaderSender} wires to. The message value stays the byte-identical
* envelope JSON. Returns {@code meta.id}.
*/
public String publishWithHeaders(Envelope envelope, Map<String, String> headers) throws PulsarClientException {
return send(envelope, null, headers);
}

private String send(Envelope envelope, Duration delay, Map<String, String> headers)
throws PulsarClientException {
TypedMessageBuilder<byte[]> message = producer.newMessage()
.value(EnvelopeCodec.encode(envelope).getBytes(StandardCharsets.UTF_8));
for (Map.Entry<String, String> property : PulsarProperties.of(envelope).entrySet()) {
Map<String, String> properties = PulsarProperties.of(envelope);
boolean delayed = delay != null && !delay.isZero() && !delay.isNegative();
if (delayed) {
properties.put("bq-delay", Long.toString(delay.toMillis()));
}
PulsarProperties.addOutOfBand(properties, headers);
for (Map.Entry<String, String> property : properties.entrySet()) {
message.property(property.getKey(), property.getValue());
}
if (delay != null && !delay.isZero() && !delay.isNegative()) {
message.property("bq-delay", Long.toString(delay.toMillis()));
if (delayed) {
message.deliverAfter(delay.toMillis(), TimeUnit.MILLISECONDS);
}

Expand Down
Loading
Loading