Respect q-weights and align HEAD/GET content-type for conneg (#325) - #326
Merged
Conversation
Two related conneg bugs filed in #325: 1. Q-weights ignored in the dispatch. src/handlers/resource.js used naive `acceptHeader.includes('text/turtle')` substring matching to decide between JSON-LD and Turtle. An Accept header of `application/ld+json, text/turtle;q=0.1` returned Turtle even though JSON-LD was the higher-q preference. Replaced five substring sites (handleGet container-with-index, container listing, resource HTML data island; handleHead container) with the existing q-aware `selectContentType` from src/rdf/conneg.js. 2. HEAD/GET divergence on containers without index.html. handleHead hard-coded `application/ld+json` for that case regardless of Accept, so HEAD reported a content-type that didn't match what GET would emit. The mismatch confused tooling that read HEAD content-type and then GET'd the body. handleHead now runs the same q-aware Accept logic so HEAD content-type tracks GET. The auth-vs-anonymous mismatch in #325 doesn't reproduce on the current tip — likely fixed by the Turtle-converter work in #321. Both anon and auth paths now return identical content-type for the same Accept. Tests: new conneg.test.js block exercises q-weight respect, HEAD/GET parity across four Accept shapes, and auth-vs-anon parity. 470/470.
2 tasks
melvincarvalho
added a commit
that referenced
this pull request
May 2, 2026
* Phase 1: embed JSON-LD data island in mashlib HTML wrapper (#7) The mashlib HTML wrapper now carries the originating resource's JSON-LD bytes as a `<script type="application/ld+json" id="dataisland" data-uri="…">` block. Phase 1 is strictly additive: - mashlib still XHR-fetches as before (no behaviour change yet); - browsers ignore the script body since its MIME isn't JS; - but anything that knows to look for `application/ld+json` islands — search engines (Rich Results), archival crawlers, scrapers, static-site exporters, future LWS-aware tooling — now finds the data without a second HTTP request. This sets up Phase 2, where a small inline shim will patch mashlib's `fetcher` to read the island instead of refetching, eliminating the double-fetch that motivated the recent Vary/Cache work (#316, #325/#326). Format: JSON-LD (not Turtle), since: - JSS stores JSON-LD natively → zero server-side conversion; - `<script type="application/ld+json">` is the standardised way to embed structured data on the web; - aligns with our LWS / CID work — an HTML-only LWS verifier can parse the embedded JSON-LD and find the CID `service[]` entry without conneg. Sites: - src/mashlib/index.js: `generateDatabrowserHtml(url, cdn, {embedJsonLd})` emits the island when a payload is supplied. New `DATA_ISLAND_MAX_BYTES = 256 * 1024` size cap silently drops the island for oversize resources; the wrapper still works because mashlib falls back to its XHR path. - src/handlers/resource.js: passes the JSON-LD bytes when the stored content type is `application/ld+json` (resources) or whenever a container listing is generated. Security: - `</script>` substrings inside the JSON-LD body are escaped to `<\/script>` so a user-PUTted resource can't close the script tag prematurely and inject HTML. - `<!--` substrings are escaped to neutralise comment-escape tricks. - The `data-uri` attribute is HTML-entity-encoded (&, ", <, >) so attribute-quote injection isn't possible either. 9 new tests in test/data-island.test.js cover: emission shape, the back-compat omission case, the size cap, both `</script>` and `<!--` escapes, attribute encoding, and live HTTP integration for both resources and container listings — plus a negative test that the mashlib XHR path (Accept: application/ld+json) still gets RDF rather than the wrapper. Phase 1 of #7. 546/546 green. * Address Copilot round-1 on #343 Six points, all real: - src/mashlib/index.js: escapeForScriptBlock now encodes EVERY `<` byte as the JSON Unicode escape `<`. The previous narrow regex only caught literal `</script>` and `<!--`, but HTML parsers terminate a <script> element on the prefix `</script` regardless of what follows — `</script >`, `</script\n>`, `</SCRIPT>` and friends all close it. After this transform the body cannot contain a literal `<` at all, so no end-tag (or comment, or CDATA) can possibly start. JSON-LD semantics are preserved because JSON parsers decode `<` back to `<` natively. - src/mashlib/index.js: header comment rewritten to accurately describe the new strategy (the previous comment had been internally inconsistent and described a different approach). - src/mashlib/index.js: generateModuleDatabrowserHtml now also accepts opts.embedJsonLd and emits the data island. Previously module-mode mashlib deployments missed the feature entirely. - src/handlers/resource.js: both wrapper paths (CDN/local + module) now pass the JSON-LD content through. Module mode wraps consume the same opts shape. - src/handlers/resource.js: cap-aware short-circuit — when stats.size > DATA_ISLAND_MAX_BYTES the handler skips storage.read entirely. Previously a 10MB JSON-LD resource would load into memory on every HTML navigation only to have the island silently dropped by generateDatabrowserHtml. - test/data-island.test.js: replaced the single-variant </script> test with a parametrised loop covering exact/whitespace/ newline/uppercase/mixed-case end-tag forms plus <!--. All assert the same invariant: NO literal `<` survives in the script body. Added a module-mode emission test. Full suite: 551/551 (5 new tests). * Address Copilot round-2 on #343 (clarity / dedupe) Three points: - src/mashlib/index.js: escapeForScriptBlock comment now says literally "the six-character JSON escape sequence \\u003c" instead of "<", which had ambiguously rendered the literal escape as just `<` and made the security rationale read backwards. - src/mashlib/index.js: dataIsland's URI escaping now reuses the existing escapeHtml() helper from this file instead of an inline duplicate. Function declarations hoist, so the call site can precede the helper definition without reordering. - test/data-island.test.js: parametrised escape-test header comment rewritten to describe the actual transform (`\\u003c`). Pure clarity / refactor; no behaviour change. Full suite: 551/551. * Address Copilot round-3 on #343 (comment + buffer input) - src/mashlib/index.js: rewrote the escapeForScriptBlock comment to describe the transform unambiguously: "the JSON string-escape for U+003C — the six characters backslash-u-0-0-3-c". The previous attempt kept losing the literal `\\u003c` text to the rendering pipeline and reading as `<` in source, inverting the security rationale. - src/mashlib/index.js: escapeForScriptBlock now coerces input via `String(jsonLdString)`, so a Buffer (e.g. straight from `storage.read()`) passes through cleanly instead of throwing on `.replace`. Buffer.byteLength already accepted both, so dataIsland's size cap was already buffer-safe. - src/mashlib/index.js: JSDoc for `opts.embedJsonLd` now declares `string|Buffer` on both wrapper functions. - test/data-island.test.js: new regression test passing a Buffer payload directly, asserting the island emits with the expected body content. Full suite: 552/552. * Apply size cap to escaped body, not raw input (#343 round-4) DATA_ISLAND_MAX_BYTES was being checked against the *pre-escape* input. escapeForScriptBlock can expand input up to 6x (each `<` byte becomes the six-char escape `\\u003c`), so a `<`-heavy body just under the cap would balloon the HTML response well past it. Move the size check after the escape and gate on the bytes that will actually appear in the response. Comment updated accordingly. New regression test: a half-cap-size payload of pure `<` bytes — 6x expansion guarantees it would slip past the old check but is correctly rejected now. Full suite: 553/553. * Address PR #343 round-5 review (Copilot) Three small refinements: 1. dataIsland(): add cheap raw-byte pre-check before escaping. Since escapeForScriptBlock can only grow the body (each `<` becomes 6 bytes), a raw payload already over the cap is guaranteed to be over after escaping — skip the work. Post-escape check still guards the `<`-heavy expansion case. 2. mashlib/index.js dataIsland doc comment: clarify the 6x escape expansion using prose form for the escape sequence so it survives round-trips through tooling that might otherwise interpret the literal characters. 3. test/data-island.test.js post-escape cap test comment: same clarification. No behaviour change beyond the pre-check fast path; all 553 tests still pass. * Address PR #343 round-6 review (Copilot) Two genuine optimizations from this round (the other flags were re-runs of points already addressed in earlier rounds — pre-check, Buffer coercion, stats.size guard, module-mode wrapper — all already present in HEAD): 1. Container path: use compact JSON.stringify(jsonLd) for the embed instead of serializeJsonLd(), which pretty-prints with 2-space indent. The HTTP body still uses serializeJsonLd; only the inlined data island goes compact, so we don't waste bytes against DATA_ISLAND_MAX_BYTES on whitespace nothing reads. 2. Resource path: pass the storage.read() Buffer through to embedJsonLd directly. dataIsland() already coerces Buffer → string via String(jsonLdString), so the eager buf.toString('utf8') in the handler was redundant. All 553 tests still pass. * PR #343 round-7 review: test comment clarity Two doc-only fixes from this round; the other 7 inline comments re-flag points already addressed in earlier rounds (raw-byte pre-check, Buffer coercion, stats.size guard, module-mode wrapper, compact embed serialization). 1. Header comment: include `data-uri="..."` in the script tag example so it matches the actual emission shape the tests assert. 2. Inline comment on the escape-presence assertion: refer to the six-character escape sequence in prose form ("backslash-u-0-0-3-c") so the source intent survives tooling that converts the literal characters back to `<`. No code change; all 553 tests still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two conneg bugs reported in #325.
Bug 1: Q-weights ignored
src/handlers/resource.jsused naiveacceptHeader.includes('text/turtle')substring matching. AnAccept: application/ld+json, text/turtle;q=0.1header returned Turtle even though JSON-LD was the higher-q preference. Replaced five substring sites with the existing q-awareselectContentTypefromsrc/rdf/conneg.js:handleGet: container withindex.htmldata-islandhandleGet: container listing (no index.html)handleGet: resource with HTML data-island (.ttlURL hint preserved)handleHead: containerBug 2: HEAD/GET divergence
handleHeadhard-codedapplication/ld+jsonfor containers withoutindex.html, regardless of Accept. SoHEAD /me/andGET /me/reported different content-types for the same URL with the same Accept — tooling that read the HEAD label and then fetched the body got a mismatch.handleHeadnow runs the same q-aware Accept logic and tracks GET.Note on the "auth path mislabels content-type" symptom
Doesn't reproduce on current tip. Likely fixed by the Turtle-converter work in #321 (the converter previously dropped nested-node data, which produced confused responses on some auth-via-WAC paths). Both anon and auth paths now return identical content-type for the same Accept — the new tests pin that invariant.
Test plan
Accept: jsonld;q=1.0, turtle;q=0.1→ JSON-LD (q-weight respected)Accept: jsonld, turtle;q=0.5(downstream repro) → JSON-LDAccept: text/turtle→ Turtlecurl -iGET andcurl -IHEAD on/me/withAccept: application/ld+json;q=1.0, text/turtle;q=0.1both reportapplication/ld+jsonand the body is JSON-LD.Fixes #325.