SubmitQueue controllers are built around eventual consistency. Controllers advance a workflow through durable state checkpoints, and every component must tolerate retries before the next checkpoint is recorded.
The core model is:
Load durable state, reconcile it toward this controller's checkpoint, then replay the checkpoint's fanout until it is accepted.
Optimistic locking protects checkpoints from concurrent writers. Failures and races are expected to be uncommon, so the system may leave harmless partial or orphaned data from attempts that never reached a checkpoint. That data can be cleaned up separately if it becomes a problem.
Each controller owns a small set of state transitions. It must classify the latest state before writing:
Process(message):
entity = load latest durable state
if state is before my checkpoint:
perform retry-safe preparation
record checkpoint with optimistic locking
if the version changed:
return ErrVersionMismatch
if state is at my checkpoint:
replay complete fanout using stable message identities
return success
if state is beyond or supersedes my checkpoint:
return success
return invalid-state error
The important states are:
| State relative to this controller | Behavior |
|---|---|
| Before checkpoint | Perform retry-safe work and record the checkpoint. |
| At checkpoint | Skip the state transition and replay the complete fanout. |
| Beyond checkpoint | A downstream controller already consumed the handoff. Acknowledge without regressing state. |
| Superseded | Cancellation, failure, or another outcome made this work unnecessary. |
| Invalid | Return an error rather than inventing a transition. |
Prefer one reconciliation pass per delivery. The consumer framework is the retry loop:
controller returns error
-> error processor classifies it
-> consumer nacks retryable errors
-> redelivery re-enters Process and reloads durable state
Controllers should not classify ordinary backend failures merely because replay would be convenient. Return the raw wrapped error and let the configured classifiers decide whether it is transient. A permanent publish or storage failure must eventually reach the DLQ rather than retry forever.
For a state transition followed by queue fanout:
persist checkpoint
publish complete fanout
ack delivery
The checkpoint proves that the state transition happened. It does not prove that every output was published.
If a process fails after recording the checkpoint, redelivery observes the checkpoint, skips the transition, and republishes the complete fanout. Every replayed output must use the same topic, partition key, logical message ID, and payload.
Optimistic locking answers whether an entity changed since it was read. It does not decide whether a lifecycle transition is valid.
A controller must write only from states it owns. For example, speculate may transition Created to Speculating; it must not load Merging and write it back to Speculating.
Version arithmetic follows the storage optimistic-locking contract: compute the new version in the controller and update the in-memory entity only after the write succeeds.
| Batch state | Behavior |
|---|---|
Created |
Start speculation: publish to build, then record Speculating. |
Speculating |
Once dependencies resolve, publish to merge and record Merging. |
Merging |
Acknowledge without regressing the batch; the merge controller owns recovery. |
Cancelling, terminal |
The transition was superseded or another controller owns recovery. |
Each row writes only from a state speculate owns; states owned by other controllers (such as Merging) are acknowledged, never rewritten. If a publish fails after a state transition is recorded, redelivery reloads the batch, skips the completed transition, and republishes the fanout with stable message identities.
An external effect whose outcome was not recorded cannot be made safe by queue deduplication alone:
provider accepts operation
controller fails before recording the result
Such effects require a provider-supported idempotency key, a stable operation identity that can be queried, or an explicit acceptance that duplicate or orphaned work is harmless.
For each controller, make these answers clear:
- What durable checkpoint does it own?
- Is all work before that checkpoint safe to retry?
- How does each possible durable state classify relative to the checkpoint?
- Can the complete fanout be reconstructed and replayed with stable message identities?
- Which controller or DLQ path owns superseded and terminal recovery?
See the consumer error contract, the orchestrator workflow, and the SQL queue RFC.