Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 33 additions & 6 deletions specs/api/agent-inbox-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ inbox's "Mine" folder works without inventing users. Multi-Agent identity is a l
increment, added when there is a second Agent.

This document covers the whole v1 surface. **HT-17 implemented §3's read paths and the
conventions below; HT-18 implemented §4a–4b; HT-16 amended §4a with send idempotency.**
conventions below; HT-18 implemented §4a–4b; HT-16 amended §4a with send idempotency; HT-49
amended §4a's `References` derivation to append the reply's own minted id (a provider —
Gmail, confirmed live — can rewrite `Message-ID` on send; threading.md §2a).**
The v1.1 additions land per-ticket: HT-26 (status model), HT-27 (`preview` + `number`),
HT-28 (notes), HT-29 (tags), HT-30 (delete), HT-31 (assignee), HT-32 (open tracking).

Expand Down Expand Up @@ -198,11 +200,19 @@ from the conversation, so the client never sets recipients or threading headers:
- **`subject`** = the conversation's `subject`, prefixed with `Re:` plus a space if it
isn't already (case-insensitive check — never double-prefix to `Re: Re:`).
- **`In-Reply-To`** = the `messageId` of the conversation's most-recent INBOUND thread (the
customer message being answered), if it has one; **`References`** = the `messageId`s of
all prior threads in chronological order that have one. These are for the customer's mail
client to thread the reply in THEIR inbox — Helpthread's own threading never depends on
them (it is outbound-token-anchored; threading.md §2). Omitted when no prior message-id
exists (e.g. an inbound message that arrived without a `Message-ID`).
customer message being answered), if it has one; omitted when no prior message-id exists
(e.g. an inbound message that arrived without a `Message-ID`).
- **`References`** = the `messageId`s of all prior threads in chronological order that have
one, followed by this reply's OWN freshly-minted `messageId` as the FINAL entry — appended
by `sendReply` itself (`src/mail/send.ts`), unconditionally, even when no prior thread has a
`messageId` at all (a first reply then gets a one-element `References: [messageId]`, never
omitted the way `In-Reply-To` can be). These are for the customer's mail client to thread
the reply in THEIR inbox — Helpthread's own threading never depends on them (it is
outbound-token-anchored; threading.md §2) — but the reply's own minted id riding in
`References` is now load-bearing in one specific way (HT-49, threading.md §2a): some
providers (Gmail, confirmed live) rewrite the wire `Message-ID` to their own generated id,
so `References` — which such providers do NOT rewrite — is the channel that actually gets
the signed token back into the customer's reply when that happens.

The handler then calls `sendReply` (`src/mail/send.ts`), passing the `Idempotency-Key` value
through. `sendReply` mints the reply token into the outbound `Message-ID` (on a genuinely
Expand Down Expand Up @@ -387,6 +397,23 @@ above.

## 7. Changelog

- **v1.1 (2026-07-17, HT-49 review fix).** `InboxApiDeps.selfEchoGuard` (optional, absent
by default): when present — and when the sender reports a provider message id for a
resolvable outbound mailbox — the send path best-effort pre-seeds a successful reply's
own sent-message echo as suppressed in the inbound delivery ledger, so a transport that
reflects sent mail back into its own mailbox (Gmail, confirmed live) normally does not
re-ingest it as a phantom inbound message — a consequence of the `References` change
below now carrying a verifiable token into that self-echo too. Best-effort, not a
guarantee: reconcile can win the documented pre-seeding race and ingest that one echo
first (`inbound-ingestion.md` §5's HT-49 amendment, "Known residual"). See
`src/mail/send.ts`'s "The reply token's own self-echo" section for the full mechanism.
No other §4a behavior changed; a deployment that leaves this absent behaves exactly as
before.
- **v1.1 (2026-07-17, HT-49).** §4a's `References` derivation now appends the reply's own
freshly-minted `messageId` as the final entry, after the derived ancestor chain — fixing
live-observed thread splits where a provider (Gmail, confirmed) rewrites the outbound
wire `Message-ID`, discarding the token from its one prior channel. See
threading.md §2a and sending.md §4 for the full mechanism; no other §4a behavior changed.
- **v1.1 (2026-07-16, HT-46).** `ThreadView.attachments`: inbound attachment metadata +
a signed `BlobStore` URL, `[]` by default and config-gated (absent `attachments` deps
at the composition root, §4's `InboxApiDeps`, same posture as open tracking) — a
Expand Down
44 changes: 44 additions & 0 deletions specs/mail/inbound-ingestion.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,50 @@ loop yet; the suppression that matters now is the verifiable own-message loop ru
A suppressed message is recorded in the ledger (`suppressed`, with the reason) — visible,
auditable, never a silent drop.

**HT-49 amendment: the `Message-ID` loop-suppression correlation is defeated for a
provider that rewrites it (Gmail, confirmed live) — a second, ledger-level guard closes
the resulting gap.** The rule above ("our exact outbound `Message-ID` ... appearing as
this message's `Message-ID`") assumes the provider transmits `Message-ID` unaltered end
to end. `specs/mail/sending.md`'s HT-49 amendment records live evidence that Gmail's
`users.messages.send` does not: it accepts the engine's verbatim `Message-ID` but
substitutes its own generated id on the wire. Concretely, this means: (1) every outbound
reply now also carries the reply token as the FINAL entry of its own `References` chain
(threading.md §2a) — a second, provider-durable channel for the SAME token — and (2) when
that reply's own sent copy is reflected back into the mailbox it was sent from (Gmail
delivers a sent message into the same mailbox; the reconcile pipeline, `src/mail/
gmail-reconcile.ts`, ingests it like any other inbound message), `isOwnMessageReflection`
above never fires for it — the message's OWN `Message-ID` is Gmail's substitute, not our
token, so the one correlation this rule implements finds nothing. Without a further guard,
`decideThreading` would then find the token in `References` and `append` — the agent's own
sent reply stored a second time as a phantom `direction: 'inbound'` message in the very
conversation it belongs to.

The closing guard is NOT an extension of the `Message-ID`/`References`/`In-Reply-To`
correlation above (deliberately — see this section's own warning against trusting
sender-controlled headers, and the `isOwnMessageReflection` doc comment's note that a
customer's own out-of-office reply legitimately carries our token in exactly the same
`References` position and must still be ingested). Instead, `src/mail/send.ts` — right
after a successful send whose sender reports a `providerMessageId`
(`EmailSendResult.providerMessageId`, the SAME id the transport later reports for that
exact message during reconcile) — pre-seeds `(mailboxId, providerMessageId)` as an
ALREADY-`suppressed` row in the delivery ledger itself
(`InboundDeliveryStore.preSuppressOwnSend`, §4's idempotency/claim machinery, unchanged).
When reconcile later lists that same provider id, `claim()`'s ordinary "terminal row, do
not double-process" branch (§4) absorbs it — no new suppression code path, no change to
`decideThreading`, no heuristic on message content. This is a ledger-level, `providerMessageId`-keyed
correlation — a DIFFERENT (and more precise) mechanism than the `Message-ID` correlation
this section otherwise describes, chosen precisely because it does not touch the
customer-autoresponder case above at all.

**Known residual: a race, conceded rather than corrected.** The pre-seed happens
AFTER the send resolves; if reconcile's own `claim()` for the same provider id wins that
race first (an unusually fast push-triggered reconcile), the message ingests normally
before the pre-seed ever runs — `preSuppressOwnSend` then finds the key already claimed
and is a no-op (it never overwrites an existing row, `src/store/inbound-deliveries.ts`'s
doc comment). This reproduces the PRE-HT-49-fix failure mode for that one send, not a new
one, and is not silently hidden: the phantom message is still recorded and visible in the
conversation, exactly as it would have been before this guard existed.

## 6. Observability and the forged-token signal

Each ingest emits a structured record: `mailboxId`, `providerMessageId`, the transport
Expand Down
58 changes: 53 additions & 5 deletions specs/mail/sending.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Outbound sending & the reply-token lifecycle

Status: accepted (HT-15, HT-16). Companion to [threading.md](./threading.md) — that
Status: accepted (HT-15, HT-16, HT-49). Companion to [threading.md](./threading.md) — that
spec decides which conversation an *inbound* message joins; this one covers how an
*outbound* reply is minted, persisted, and sent, and is where the threading
model's authority actually originates.
Expand Down Expand Up @@ -174,6 +174,50 @@ accept raw MIME; reject any that will not carry `Message-ID` unaltered. The
in-repo fake used by the engine tests proves only that `sendReply` *passes* the
value to the seam — not that any given adapter preserves it on the wire.

**A compliant adapter is not sufficient — the provider's OWN infrastructure can
still rewrite `Message-ID` after transmission (HT-49, live production evidence,
2026-07-17).** Gmail's `users.messages.send` accepted the Gmail adapter's
verbatim `Message-ID` on the request and substituted its own generated id on
the wire — a rewrite downstream of transmission, outside the adapter's control,
and not a violation of the contract above (the Gmail adapter's own wire-level
contract test still passes: it proves what it sends, not what Gmail's server
does with it afterward). See threading.md §2a for the full story and the fix:
`sendReply` (`src/mail/send.ts`) now ALSO places its own minted `messageId` as
the final entry of that same reply's `References` header — a channel Gmail
does not rewrite — so the token survives even when `Message-ID` itself does
not. The adapter contract for `References` is: every atom transmitted is
transmitted verbatim and in its given order — never rewritten, reordered, or
substituted. Unlike `Message-ID`, though, the ancestor portion of `References`
carries attacker-influenced inbound msg-ids, so an adapter MAY sanitize by
DROPPING an individual unsafe atom (header-injection / oversize defense — the
Gmail adapter's `isSafeMsgId` filter, `src/providers/adapters/gmail/mime.ts`,
does exactly this rather than letting one crafted stored ancestor id block
every future reply to its conversation). The engine-minted final entry passes
any such filter by construction (`reply-token.ts`'s bounded `[A-Za-z0-9_-]` /
`.` / `@` charset contains no control characters and stays far under the
octet bound) and MUST reach the wire intact — an adapter that drops or alters
IT is as unusable as one that rewrites `Message-ID`. The HT-49 fix is in what
the engine puts into `References` before handing it to the adapter, not a
change to this adapter contract.

**Review-fix amendment: a self-reflecting transport requires ALSO suppressing the sent
message's own echo (`src/mail/send.ts`'s "The reply token's own self-echo" section).**
Placing the reply token in `References` unconditionally has a consequence the paragraph
above does not by itself address: Gmail (confirmed live) delivers the SENT message back
into the mailbox it was sent from, and that self-echo now carries a verifiable token —
one `inbound-ingestion.md` §5's `Message-ID`-only loop guard cannot recognize, for the
exact same reason as above (Gmail rewrites the echo's `Message-ID` too). Left alone, the
echo would `append` into its own conversation as a phantom inbound message. `sendReply`
closes this immediately after a successful send: if `EmailSender.send()` returned an
`EmailSendResult.providerMessageId` (`src/providers/email-sender.ts`) — the SAME id the
transport later reports for that message during reconcile — it resolves `SendReplyInput.
from` to its `MailboxRecord` and pre-seeds `(mailboxId, providerMessageId)` as an
already-`suppressed` row in the inbound delivery ledger (`InboundDeliveryStore.
preSuppressOwnSend`; `inbound-ingestion.md` §5's HT-49 amendment has the full mechanism
and its one known residual race). This is OPTIONAL (`SendReplyDeps.selfEchoGuard`) and a
no-op wherever absent or wherever the sender reports no `providerMessageId` — a deployment
with no self-reflecting transport configured behaves exactly as before this guard existed.

**Recommended: a provider SHOULD de-duplicate on `Message-ID` (HT-16).** This
is not a precondition the engine requires — at-least-once delivery (§3a) holds
with or without it — but it is not an aside either: it is the one thing
Expand Down Expand Up @@ -213,10 +257,14 @@ Deliberately narrow; each deferral below has a named later home:
directly (e.g. from a test or a manual trigger), never on a timer.
- **Reply to an existing conversation only.** Agent-*initiated* brand-new
conversations are a separate later flow.
- **`In-Reply-To`/`References` are caller-supplied** (from the inbound message
being answered). Deriving the full `References` chain from stored threads is a
later refinement. Once persisted into `send_envelope` (§3a) that snapshot is
authoritative for every retry regardless of how it was originally derived.
- **`In-Reply-To`/`References` are caller-supplied ANCESTOR ids** (from the
inbound message being answered; `agent-inbox-v1.md` §4a's `deriveReplyHeaders`
derives them from stored threads today). `sendReply` then APPENDS its own
freshly-minted `messageId` as the final `References` entry unconditionally
(HT-49; threading.md §2a) — the caller-supplied field is never itself the
reply's own id. Once persisted into `send_envelope` (§3a) that full chain,
own id included, is authoritative for every retry regardless of how the
ancestor portion was originally derived.
- **A missing or deleted conversation is refused** — the token is minted first
(before `appendThread` resolves) and then discarded on refusal; only
persistence and sending are skipped, and the sender is never called (mirrors
Expand Down
12 changes: 12 additions & 0 deletions specs/mail/threading.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,18 @@ The properties that ARE the spec, independent of encoding:

**Contrast with the observed reference format.** The fixtures show a reference helpdesk emitting Message-IDs shaped like `<FS_reply-{threadId}-{token}@{domain}>` — e.g. `<FS_reply-36-{token}@helpdesk.example.test>` (reply-with-reference.json, `agentReplyEmail.messageId`; the token value in the committed fixtures is a redacted placeholder — the real capability token is never published). Notably `{threadId}` there is a *thread* id (36), not the conversation id (15) — conversation is resolved via the thread's parent, not encoded directly. This is cited only as evidence the "signed token in the outbound Message-ID" pattern works in production (charter §2); Helpthread's `sig` derivation, secret, and truncation are unrelated to whatever that system does internally, which was never observed.

## 2a. The token also rides in `References` (HT-49 — a provider can rewrite `Message-ID`)

**RESOLVED, live production evidence, 2026-07-17 (first HT-44 run against real Gmail).** §2 describes the token embedded in the outbound `Message-ID`; that remains true and unchanged. But `Message-ID` is not guaranteed to survive transmission unaltered: Gmail's `users.messages.send` accepted the engine's verbatim-set `Message-ID` on the request and REPLACED it on the wire with a Gmail-generated id (`<CAKWkAL3...@mail.gmail.com>` — confirmed from the raw copy Gmail itself returned on reconcile of the sent message's self-echo). This is not a violation of the `EmailSender` contract (specs/mail/sending.md §4, `src/providers/email-sender.ts`) — the adapter transmitted `Message-ID` verbatim as required; the rewrite happens server-side, downstream of transmission, outside any adapter's control. Its effect on threading is the same either way: the customer's reply carried `In-Reply-To`/its trailing `References` entry pointing at Gmail's substituted id, with our token nowhere on the wire — `decideThreading` correctly found no verified token and (§3 rule 4, invariant #5) started a NEW conversation instead of appending. Tonight's failure is preserved as a fixture reproducing it exactly: `src/mail/ingest.test.ts`'s "the exact live-production failure" test.

**The fix: the outbound reply's own minted `messageId` ALSO rides as the FINAL entry of that reply's own `References` header** (`src/mail/send.ts`, `sendReply`), appended after any ancestor ids — unconditionally, even on a first reply with no ancestors (a one-element `References: [messageId]`). `References`, unlike `Message-ID`, is not rewritten by Gmail. An RFC-5322-compliant reply's own `References` is built as `{original References} + {original Message-ID}` (§3.6.4) — so when the customer replies, their client's `References` becomes `[...ourOutboundReferences, gmailRewrittenId]`, i.e. `[...ancestors, ourMintedToken, gmailRewrittenId]`. The token lands ONE POSITION BEFORE the trailing foreign id — never last, never in `In-Reply-To` (which still correctly names the specific ancestor message being answered, not this reply's own id — left unchanged by this fix).

**Zero threading-decision code changed.** §3's algorithm already scans `References` newest-first (`src/mail/thread.ts`'s `buildCandidates`, reversing wire order before scanning) — exactly what is needed to skip the foreign trailing id and find our token immediately behind it. This section moves *where the token rides on the outbound side*; it does not touch how an inbound decision is made, does not add a heuristic, and does not weaken "no verified token ⇒ new conversation." Verified, not assumed: `src/mail/thread.ts` is unmodified by HT-49, and the fixture above passes through the existing scan unchanged.

**Every future outbound reply therefore carries the token TWICE** — once in `Message-ID` (§2, the primary channel, un-rewritten by providers that respect the `EmailSender` contract) and once as the final `References` entry (§2a, the provider-durable backup channel). Either surviving to the customer's reply is sufficient for `decideThreading` to append correctly; both surviving is redundant, not conflicting (the newest-first scan tries `In-Reply-To` first, so an un-rewritten `Message-ID`/`In-Reply-To` pair is still found first when it survives).

**Review-fix amendment: this section's own reply also reflects back into its own mailbox — a SEPARATE guard, not this section's scan, closes it.** Putting the token in `References` unconditionally means the SENT message's own self-echo (Gmail delivers a sent message into the mailbox it was sent from; `src/mail/gmail-reconcile.ts` ingests it like any other inbound message) now ALSO carries a verifiable token in `References` — and `inbound-ingestion.md` §5's `Message-ID`-based loop guard (`isOwnMessageReflection`) cannot catch it, for the exact reason this section exists: Gmail rewrites the self-echo's `Message-ID` too. Left unguarded, that self-echo would `append` into its own conversation as a phantom inbound message. This is NOT fixed by touching `decideThreading` or this section's scan (doing so would also misfire on a customer's legitimate autoresponder reply, which carries our token in `References` the exact same way) — it is fixed one layer earlier, in the delivery ledger itself, before `decideThreading` ever runs: see `inbound-ingestion.md` §5's "HT-49 amendment" for the full mechanism (`src/mail/send.ts`'s `preSuppressOwnSend`) and its one known residual race.

## 3. Inbound threading decision — the algorithm

Ordered, testable procedure applied to every inbound message:
Expand Down
Loading
Loading