/v1/id/_:id/expanded endpoint - #286

Closes #285.

Summary

Adds GET and POST /v1/id/:_id/expanded — a generic, unauthenticated, browser-cacheable endpoint that returns a RERUM object with the assertions of every current Annotation targeting it already merged in.

/gog/id/:_id (#280) proved the approach but is GoG-shaped: it wraps every merged value in DEER's { value, source, evidence } object. Almost every application building entities in RERUM does this same gather-and-merge on the client, across many /query requests, so the generic version emits raw values instead — as close to what the Annotation body actually says as possible. Clients that need a particular shape format the response themselves.

The Annotation-gathering logic is now shared by both endpoints, so they cannot drift apart.

Changes

New endpoint

routes/id.js — /:_id/expanded registered ahead of /:_id. GET, POST (behind rest.verifyJsonContentType), and HEAD via Express's native GET handling. Everything else is a 405.

controllers/crud.js — idExpanded() plus three helpers:

• sanitizeExpansionFilters() reduces a POST body to the filter keys this endpoint will honor.
• assertionsFrom() reads only body and bodyValue off an Annotation, so no other property of an Annotation can leak onto the entity.
• applyRawExpansion() merges those assertions onto a clone of the entity and re-appends __rerum last.

db-controller.js — exports idExpanded.

Shared Annotation gathering

controllers/utils.js — new findLeafAnnotationsFor(targetId, filters, pagination), lifted out of controllers/gog.js expand() and widened. It constrains to current (leaf) versions and to Annotations, and it matches these target forms — each under both the http and https spelling:

target: 'uri'
target: {'id':'uri'}
target: {'@id':'uri'}
target: {'source':'uri', 'type':'SpecificResource'}   the W3C SpecificResource
target: {'source':{'id':'uri'}}                        SpecificResource with an embedded source
target: 'uri#xywh=0,0,100,100'                         a fragment of the resource

and these type spellings, under both type and @type:

Annotation    oa:Annotation    http://www.w3.org/ns/oa#Annotation

target.source* and the #fragment form are new — the GoG path previously matched only target, target.@id, and target.id, so it was silently missing every Annotation that targets a selected region rather than the whole resource. The fragment patterns are built through an escapeRegex() helper.

Indexes already exist for all six target keys, so this stays index-backed. explain("executionStats") on a real entity: nReturned 24, totalKeysExamined 30, totalDocsExamined 25.

GET — the convenience form

Imagined as the plain GET where nothing else is supplied. Recognizes ?generator= and ?creator= only, both matched across the http/https spellings of the URI. ?limit= (default 200) and ?skip= page the Annotation search.

POST — the filtered form

The JSON body is an object of literal MongoDB filter keys, ANDed into the Annotation search — so filtering by generating app means supplying the real property name, __rerum.generatedBy. URL parameters supply no filters here, though ?limit= and ?skip= still page. Not browser-cached.

Three constraints belong to the endpoint and cannot be overruled. Supplying them is not an error, they are dropped by exact name or dotted prefix:

Because every supplied filter is ANDed in, a filter can only narrow the result set — there is no way to widen it past the entity in the request URI.

What gets merged

• A body with exactly one property contributes that property, value passed through exactly as it appears. A body of {"text": {"value": "hello"}} puts {"value": "hello"} on the entity, not "hello".
• A bodyValue string and a TextualBody body both land under bodyValue; the TextualBody is kept whole so its format and language survive.
• Repeated assertions of the same property collect into an Array.
• Identity and system properties are never overwritten: @id, id, _id, __rerum, __deleted, @context. An assertion naming __proto__ is dropped — it is not data, and emitting it would hand a prototype pollution vector to every client that parses the response.
• Skipped for now: a body with more than one property, an Annotation with multiple bodies (Expansion drops Annotations that carry multiple bodies #288), the Choice/Composite/List constructs, and a body that is a bare URI referencing an external resource.

Response headers

Cache-Control: max-age=86400, must-revalidate on the GET, so browsers can cache for 24h. Annotations on an entity are volatile during initial creation and stable afterward, so this trades a hard reload in the rare stale case for not re-running the expand on every visit. Not set on the POST.

No Last-Modified — it derives from the root entity's __rerum and would not move when a targeting Annotation changed, which is the wrong freshness signal for a composed body. Same reasoning as #283. Express's ETag still gives conditional revalidation.

Two new headers let a client reason about coverage:

• Annotations-Gathered — the size of the page you received. When it equals your limit there may be more, and you were expanded from only part of the record's Annotations. This is the one to page off.
• Annotations-Merged — how many of those could contribute. Annotations with multiple bodies are not counted. A counted Annotation may still assert nothing mergeable, so this is not a count of properties received either.

Location is canonicalized to the _id form, so a slug request advertises the stable URL.

controllers/gog.js

Refactored onto the shared helper, which is where the bulk of the deletions come from. Three behavior changes ride along, all on /gog/id/:_id:

• hasOwnProperty → Object.hasOwn. An Annotation body asserting hasOwnProperty would otherwise shadow the method and break the merge. This one is load-bearing, not cosmetic.

• Annotations carrying multiple bodies are now skipped explicitly. Previously a one-element Array body merged onto the entity under the key "0". See Expansion drops Annotations that carry multiple bodies #288.

• /gog/id/:_id is now restricted to GoG-generated data (403 otherwise), and expands only with Annotations from the entity's own generator. This is the follow-through on the open note left in /gog/id/:_id expand endpoint for GoG Data #280 — "this still doesn't stop anyone from using /gog/id/:_id to expand their own data — restricting it to GoG data would need an additional guard (e.g. gating on __rerum.generatedBy / GENERATOR)" — and it lines up with Filtering by generator for gog data #284.

  Measured against production before merging: across 3,000 sampled leaf entities generated by a GoG agent, all 24,259 leaf Annotations targeting them are from the GoG production agent, so the generator filter drops nothing today. The prod/dev agent ids moved into named constants at the top of the file.

Documentation

public/API.html gets both forms under GET and POST, including the full merge rules and the ignored-filters table. openapi/contracts/core-provider.openapi.yaml gets /id/{id}/expanded with get, head, and post.

Testing

npm test — 188 pass, 0 fail.

Manually exercised against production data: control entity 614362c3e74876243131a4e0 (a GoG Gloss) and a purpose-built variant entity 6a760abbeaaeae59c6c5e3ba (a IIIF Manifest annotated to hit the edge cases — bodyValue, oa:TextualBody, fragment target, target.id, target.@id, Array-valued assertions, multi-key bodies, and hostile probes asserting __proto__, constructor, toJSON, hasOwnProperty, @id, _id, __rerum, @context).

Annotation detection was also replayed against MongoDB over synthetic documents, covering 19 valid W3C target/type forms — all matched, including array-valued type, array targets, and SpecificResource with an embedded source.

Notes and follow-ups

• No automated tests for this endpoint yet. The three helpers in controllers/crud.js are pure and the db mock already exists, so routes/__tests__/idExpanded.test.js is cheap to add. Worth doing before this sees real client traffic.
• Which Annotation a property came from is not recorded. /v1/id/:_id/expanded Endpoint #285 raised this as an open question for DEER's citationSource. The raw form deliberately does not carry it — a client that needs provenance still has to query. Worth revisiting if it turns out to be needed.
• A merged property is a scalar when one Annotation asserts it and an Array when two do, so clients should treat every merged property as possibly-an-Array.
• The default limit of 200 is comfortable today — the most-annotated single target in production carries 26 leaf Annotations.

Split out of review so they do not block this PR: #287 (Allow header on 405s), #288 (Annotations with multiple bodies), #289 (hardening filter endpoints against MongoDB operator keys).