## Problem

Two needs motivated this work; running against the bench surfaced two
more engine gaps that are shipped fixed in the same PR.

1. **Memento has no published end-to-end benchmark.** The MCP-registry
launch built credibility on architectural commitments and a clean API;
the next step toward trust is "here are the numbers, here is how to
reproduce them." We need a reusable harness against the de-facto
industry datasets (LoCoMo, LongMemEval) so a sceptical engineer can
re-run a baseline on their laptop and verify.
2. **Memento's `extract_memory` contract had several first-try-wrong
footguns for AI assistants doing distillation.** While building the
benchmark provider, every distillation attempt failed silently because
the candidate shape, the topic-line requirement, and the async-mode
receipt semantics weren't surfaced in the MCP tool description (some
were in the skill but invisible to a tool consumer reading only
`tools/list`). The tool's own example was itself non-validating.
Tag-regex failures returned a bare `(root): Invalid`. These are the kind
of friction that turns "give Memento a try" into "give Memento up."
3. **FTS5 recall was stem-blind for prose queries.** The bench's first
low-score question (LongMemEval `d24813b1`, "tips on what to bake for
colleagues") missed the gold-truth memory because the memory said
"colleague's going-away party" and the query said "colleagues" — the
default `unicode61` FTS5 tokenizer treats them as different tokens. The
`retrieval.fts.tokenizer` config key advertised `porter` as a tunable
alternative, but no migration or runtime code ever read it: dead-code
tunability. Vector embedding rescued some morphological misses, but not
enough, and the failure mode is exactly the one a durable memory layer
needs to handle well (the speaker's wording and the future question's
wording rarely match in surface form).
4. **`embedder-local.embedBatch` was sequential under the hood.** The
implementation looped `extractor(text, ...)` per row with a comment
pointing at the transformers.js v2 limitation. transformers.js v3
(already pinned via `^3.0.0`) accepts an array input and runs one
forward pass for the whole batch — verified row-by-row numerically
identical to the single-call form.

## Change

Four coordinated workstreams ship together. The bench surfaced (3) and
(4); the fixes that close them improve Memento for every assistant doing
memory work, not just for the bench's score.

- **Bench driver — `scripts/bench.mjs` + `docs/guides/benchmark.md`.** A
vanilla-Node ESM driver that builds Memento, stages a memorybench fork
at a pinned ref (or a local checkout via `--memorybench-dir`), spawns
one `bun run src/index.ts run -p memento -b <bench>` per requested
benchmark, and writes a single summary markdown to `bench/<ts>.md` (the
`bench/` directory is git-ignored). Defaults to LoCoMo + LongMemEval
with `sonnet-4.6` pinned for judge + answering + distillation — the
model class that actually shows up on the conversation side in real
Memento usage (Claude Code, Cursor, and Claude Desktop are the
MCP-using-client majority, and `extract_memory` distillation happens in
*that* same assistant). Sonnet 4.6 supports `temperature=0`
(deterministic at the model layer) and the alias is registered in the
fork's `MODEL_CONFIGS`. Top-K=30, 180s indexing deadline. Per-phase
concurrency flags pass through to memorybench so a slow embedder or a
throttled Anthropic endpoint can be tamed (`--concurrency-ingest=1` is
the safe knob for `sonnet-4.6` under bursty rate-limit pressure, and it
also lets the provider's per-session distillation cache hit when
questions share sessions). The driver spawns the locally built CLI via
`process.execPath` and asserts `better-sqlite3` loads under that exact
Node before doing any expensive work, so a `nvm + homebrew` PATH
cocktail can't crash the run with a confusing "MCP error -32000:
Connection closed". A `--resume=<runId>` flag picks up at the failed
phase of the failed question for a crashed run (memorybench's
orchestrator checkpoints after every phase boundary); the runId is
logged on a dedicated line in the bench log and reprinted as a
copy-pasteable command on any non-zero exit. `--out` anchors to the
Memento repo root regardless of `cwd` so running from inside the fork
checkout doesn't leak the output directory into the fork worktree. The
provider implementation itself lives in a fork of
`supermemoryai/memorybench`
([`veerps57/memorybench@add-memento-provider`](https://github.com/veerps57/memorybench/tree/add-memento-provider)).

- **`extract_memory` tool surface + distillation craft.** The MCP tool
description on `extract_memory` states the candidate-shape difference
from `write_memory` (flat `kind` enum, top-level
`rationale`/`language`), the `topic: value\n\nprose` requirement for
`preference`/`decision` kinds, the `storedConfidence: 0.8`
async-default, and the receipt-not-failure semantics of `mode: "async"`.
The inline example exercises four kinds with the correct field placement
— including a `preference` candidate that opens with the required
topic-line and a `decision` candidate with top-level `rationale`.
`TagSchema` carries a custom error message listing the allowed charset
so `April 15, 2026` produces an actionable diagnostic instead of a bare
"Invalid". The skill (`skills/memento/SKILL.md`), persona-snippet guide
(`docs/guides/teach-your-assistant.md`), and the landing-page
persona-snippet mirror (`packages/landing/src/App.tsx`) carry a
"Distillation craft" section that frames the task as **retrieval
indexing for unknown future queries** (not summarisation for a reader)
and codifies six rules in priority order: (1) preserve specific terms —
proper nouns, identity qualifiers, named entities, places, and the
specific object of every action; (2) capture facts about every named
participant, not only the user — a friend the user mentions ("my friend
Alex is moving to Berlin for a SAP job") or a co-speaker in a
multi-party transcript both deserve candidates attributed to the right
named person, not collapsed onto the user; (3) emit a candidate for
every dated event with the date resolved against the session anchor,
never collapsing it to an untimed habit; (4) capture precursor actions
alongside outcomes — "researched X then chose Y" emits two candidates,
since future questions can target either step; (5) don't squash
enumerations into category labels; (6) bias toward inclusion — the
server dedups via embedding similarity, so over-including is cheap and
under-including is permanent. A pre-emit self-check ("did every date,
named entity, and verb-with-specific-object map to a candidate?") sits
alongside the rules in each surface.

- **Porter stemming for FTS5 — migration 0008 + honoured config +
default flip.** `memories_fts` is now built with `tokenize='porter
unicode61'` instead of the default `unicode61`. The chain runs
right-to-left: unicode61 splits + diacritic-folds first (so non-ASCII
content still tokenises correctly — German umlauts, French diacritics,
Japanese katakana all survive intact, covered by tests), then porter
stems the resulting ASCII tokens. "colleague", "colleagues", and
"colleague's" share a stem and match each other; "bake" matches "baking"
/ "baked" / "bakes"; "research" matches "researched" / "researches";
"agency" matches "agencies". The `retrieval.fts.tokenizer` config key
now defaults to `porter` and is documented as honoured by the FTS index
(it was previously declared but ignored by the migration — dead-code
tunability that this change makes real). Migration 0008 drops and
rebuilds `memories_fts` with the new tokenizer, preserving stable rowids
via the `memories_fts_map` table; the runner applies it on first server
start after upgrade, so no operator action is required. Six new unit
tests cover stem-variant matching (plural/singular, verb-form pairs),
pre-migration re-indexing, the insert/update/delete triggers carrying
the new tokenizer through write-path operations, and non-ASCII
preservation. The trade-off accepted is porter's known over-stems
(organize/organic, universe/university); for Memento's dominant query
distribution — assistants asking about durable user state in natural
language — recall on stem variants is worth more than precision on these
edge cases. Operators who need the older behaviour can author a
follow-up migration; the config key documents the option.

- **Embedder perf — real batched feature-extraction in
`@psraghuveer/memento-embedder-local`.** `embedBatch` now uses
transformers.js v3's array-input pipeline, which runs one forward pass
for the whole batch instead of looping per text. Numerically identical
to the single-call form (verified row-by-row against the same input).
Measured ~1.8× speedup on a 3-input batch with `bge-base-en-v1.5` on
CPU; the speedup grows with batch size because tokenisation and pipeline
setup amortise across the batch. The loader contract now returns `{
embed, embedBatch? }` instead of a bare `embed` function; loaders that
omit `embedBatch` fall back to the previous sequential behaviour, so
test fixtures and bespoke implementations keep working unchanged. Seven
new unit tests cover the fast path, the sequential fallback, empty-input
short-circuit, runtime-row-count mismatch, per-row dimension validation,
batched `maxInputBytes` truncation, and whole-batch timeout. The
`EmbeddingProvider.embedBatch` surface in `@psraghuveer/memento-core` is
unchanged and remains optional; existing call sites that go through
`embedBatchFallback` (`pack.install`, `import`, `embedding.rebuild`, the
synchronous extract slow-path) automatically pick up the fast path.

## Justification against the four principles

- **First principles.** The benchmark driver introduces no new
behavioural constants in Memento itself — every knob is a CLI flag or
env var declared in `DEFAULTS` at the top of `bench.mjs`. The
tool-description changes surface constraints that already existed in
code (schema validation, the conflict-detector's topic-line parsing)
where an assistant reading `tools/list` will see them. The FTS-tokenizer
change does flip a default (`retrieval.fts.tokenizer: unicode61 →
porter`), but the migration that effects it is the canonical Memento way
of evolving stored state — and the config key that controls it has
shipped since the registry release and is now actually honoured. The
embedder change is purely a perf path; semantics are byte-identical to
the previous behaviour.
- **Modular.** `scripts/bench.mjs` is a thin driver — the provider lives
in a fork of memorybench, the harness is memorybench's, and the
judge/answering models are memorybench's. The Memento side is one script
+ one guide + a pinned fork ref. The distillation-clarity changes touch
documentation and a single Zod error message; no behavioural code paths
are added. The FTS change is a single migration file + one config-key
default flip. The embedder change extends the existing `EmbedRuntime`
shape with an optional `embedBatch` and adds a single fast-path branch
in the wrapper; the rest is sequential-fallback compatibility.
- **Extensible.** Adding a third benchmark (ConvoMem) is a one-line
change to `DEFAULTS.benchmarks`. Adding a different judge family means
pointing `--judge` at a different model alias; the script's API-key
check fans out by family. The skill's distillation-craft section is
positioned so a future contributor can extend the rules without
restructuring. The FTS migration's pattern (drop → rebuild → repopulate
→ retrigger) is the same as 0005's — future tokenizer changes follow the
same template. The loader contract's optional `embedBatch` lets bespoke
embedders opt into batching when their runtime supports it, without
forcing a contract upgrade on the others.
- **Config-driven.** Every benchmark default (model, ref, limit,
concurrency, search-K, indexing deadline) is overridable from the
command line or env. The FTS-tokenizer choice is
`retrieval.fts.tokenizer` — operators can stay on `unicode61` by setting
it before first server start and recreating the FTS table via a
follow-up migration. The embedder change adds no new config key (the
runtime contract change is internal); operators with custom loaders are
unaffected by default.

## Alternatives considered

- **Vendor memorybench inside the Memento repo.** Rejected: keeps the
harness external (so we don't own its release cadence) and lets the
provider land as a normal contribution upstream. The driver pulls a
pinned fork ref, so reproduction is exact.
- **Add LLM-driven distillation inside `extract_memory` itself.**
Rejected: Memento's architectural commitment is local-first and
LLM-agnostic. Baking in an LLM would either pull in a cloud provider
(breaking local-first) or ship a bundled local model (breaking
LLM-agnostic and adding ops complexity). Distillation belongs to the
calling AI assistant, where the conversation context lives. The bench
provider does its own distill step to mirror that flow.
- **Re-design the candidate shape so `write_memory` and `extract_memory`
accept the same payload.** Considered, rejected: the discriminated-union
shape on `write_memory` is the right design for a single-row call where
kind-specific metadata is the point; the flat shape on `extract_memory`
is the right design for a batch where the per-item type is data, not a
routing tag. Documenting the difference is correct; collapsing them
would weaken both APIs.
- **Keep `unicode61` as the FTS default and ship porter as an opt-in
only.** Rejected: the `retrieval.fts.tokenizer` config key was already
documented as the operator-tunable knob, and validation of the porter
path on a real bench question showed unicode61 missing the gold-truth
memory at the FTS layer entirely. The migration is the right place to
flip the default because anyone who actively wants unicode61 can author
a follow-up migration; the silent majority who never touched the key get
a measurable recall improvement.
- **Heavier embedder optimisations — quantisation (`dtype: 'q8'`),
worker thread, WebGPU.** Deferred: quantisation is a recall trade-off
that needs its own evaluation pass; worker threads improve event-loop
responsiveness without raising throughput on a single CPU; WebGPU only
helps browser hosts (Memento runs on Node). Real batched
feature-extraction is the largest no-trade-off win available today, so
it's the one shipped here.

## Tests

- [x] Unit — full unit suite passes on this branch, plus 13 new tests
(six for migration 0008 covering stem variants, pre-migration
re-indexing, triggers, and non-ASCII preservation; seven for the
embedder fast-path and sequential-fallback paths).
- [ ] Integration — N/A; no new integration paths added beyond the
existing extract path which is already integration-tested.
- [x] Migration — `0008_fts_porter_tokenizer` is forward-only,
idempotent on a fresh DB, and verified end-to-end against a pre-0008
install via `MIGRATIONS.slice(0, 7)` in the test suite.
- [x] End-to-end — the existing `serve` e2e passes. The bench itself is
the new end-to-end exercise but is not part of `pnpm verify` for the
reasons documented in `docs/guides/benchmark.md` (it needs network,
judge API keys, and hours of wall-clock — CI must pass offline). A
focused 1Q LongMemEval validation against the baking question confirmed
the porter fix lifts that question from 0 → 1 correct, with the
lemon-poppyseed memory ranking #4 in retrieval where previously it
didn't reach top-30.
- [ ] N/A — see above.

## Local verification

- [x] `pnpm verify` (<!-- verify-chain:begin -->lint → typecheck → build
→ test → test:e2e → docs:lint → docs:reflow:check → docs:links →
docs:check → format:packs:check → server-json:check<!-- verify-chain:end
-->) — all green at branch HEAD.
- [x] `pnpm docs:generate` — run;
`docs/reference/{cli,mcp-tools,config-keys}.md`, `AGENTS.md`,
`CONTRIBUTING.md`, `.github/copilot-instructions.md`, and
`.github/PULL_REQUEST_TEMPLATE.md` regenerated to pick up the new
`extract_memory` description, the `TagSchema` error message, and the
`retrieval.fts.tokenizer` default + description.

## ADR

- [ ] An ADR is required and is included in this PR.
- [ ] An ADR is required and exists already (link below).
- [x] No ADR required (explain why):

The bench driver, the tool-description changes, the embedder fast path,
and the FTS tokenizer migration are all within the ADR exemption list in
`AGENTS.md`:

- The bench driver is optional tooling — it adds a script and a guide,
doesn't change the public surface, the data model, scope semantics, or
any top-level dependency.
- The `extract_memory` tool-description and `TagSchema` error-message
changes make existing contracts more discoverable without changing them.
- The embedder fast path is a perf optimisation with byte-identical
output; no semantic change.
- The FTS tokenizer change is a forward-only migration that honours an
already-documented config key (`retrieval.fts.tokenizer`). The default
flip is operator-visible but it neither introduces a new behavioural
constant nor changes a contract — it activates a knob that already
shipped. Memento's stance on tokenizer choice was always
"operator-configurable, default may evolve as the use case sharpens"
(per the config key's description).

## AI involvement

- [ ] No AI assistance.
- [ ] AI assistance for boilerplate / drafting only.
- [x] AI authored substantial portions. I have verified every line.

The bench driver, the provider in the memorybench fork, the audit of
`extract_memory`'s distillation-friction surface, the porter migration +
tests, the embedder batching + tests, and the prose updates to the skill
/ persona guide / landing snippet were drafted with Claude. Every change
was reviewed and exercised end-to-end against LoCoMo and LongMemEval
smokes through the full pipeline (distill → write → indexing → search →
answer → judge). The Zod error-message change and the tool-description
text were verified against the actual code paths they describe. The
porter fix specifically was validated by re-running the same failed
bench question against the new code and confirming the gold-truth memory
now ranks at the top of the retrieved set with the same models, same
haystack, same scope.

## Linked issues

Corresponding memorybench PR:
supermemoryai/memorybench#43

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>