Skip to content

Respect q-weights and align HEAD/GET content-type for conneg (#325) - #326

Merged
melvincarvalho merged 1 commit into
gh-pagesfrom
issue-325-conneg-qweight
Apr 27, 2026
Merged

Respect q-weights and align HEAD/GET content-type for conneg (#325)#326
melvincarvalho merged 1 commit into
gh-pagesfrom
issue-325-conneg-qweight

Conversation

@melvincarvalho

Copy link
Copy Markdown
Contributor

Summary

Fixes two conneg bugs reported in #325.

Bug 1: Q-weights ignored

src/handlers/resource.js used naive acceptHeader.includes('text/turtle') substring matching. An Accept: application/ld+json, text/turtle;q=0.1 header returned Turtle even though JSON-LD was the higher-q preference. Replaced five substring sites with the existing q-aware selectContentType from src/rdf/conneg.js:

  • handleGet: container with index.html data-island
  • handleGet: container listing (no index.html)
  • handleGet: resource with HTML data-island (.ttl URL hint preserved)
  • handleHead: container

Bug 2: HEAD/GET divergence

handleHead hard-coded application/ld+json for containers without index.html, regardless of Accept. So HEAD /me/ and GET /me/ reported different content-types for the same URL with the same Accept — tooling that read the HEAD label and then fetched the body got a mismatch. handleHead now runs the same q-aware Accept logic and tracks GET.

Note on the "auth path mislabels content-type" symptom

Doesn't reproduce on current tip. Likely fixed by the Turtle-converter work in #321 (the converter previously dropped nested-node data, which produced confused responses on some auth-via-WAC paths). Both anon and auth paths now return identical content-type for the same Accept — the new tests pin that invariant.

Test plan

  • Accept: jsonld;q=1.0, turtle;q=0.1 → JSON-LD (q-weight respected)
  • Accept: jsonld, turtle;q=0.5 (downstream repro) → JSON-LD
  • Accept: text/turtle → Turtle
  • No Accept → JSON-LD (native default)
  • HEAD content-type === GET content-type for four Accept shapes
  • Auth path content-type === anon path content-type for both turtle and jsonld
  • Full suite green: 470/470 (10 new tests)
  • Live smoke: curl -i GET and curl -I HEAD on /me/ with Accept: application/ld+json;q=1.0, text/turtle;q=0.1 both report application/ld+json and the body is JSON-LD.

Fixes #325.

Two related conneg bugs filed in #325:

1. Q-weights ignored in the dispatch.
   src/handlers/resource.js used naive `acceptHeader.includes('text/turtle')`
   substring matching to decide between JSON-LD and Turtle. An Accept
   header of `application/ld+json, text/turtle;q=0.1` returned Turtle
   even though JSON-LD was the higher-q preference. Replaced five
   substring sites (handleGet container-with-index, container listing,
   resource HTML data island; handleHead container) with the existing
   q-aware `selectContentType` from src/rdf/conneg.js.

2. HEAD/GET divergence on containers without index.html.
   handleHead hard-coded `application/ld+json` for that case regardless
   of Accept, so HEAD reported a content-type that didn't match what
   GET would emit. The mismatch confused tooling that read HEAD
   content-type and then GET'd the body. handleHead now runs the same
   q-aware Accept logic so HEAD content-type tracks GET.

The auth-vs-anonymous mismatch in #325 doesn't reproduce on the
current tip — likely fixed by the Turtle-converter work in #321.
Both anon and auth paths now return identical content-type for the
same Accept.

Tests: new conneg.test.js block exercises q-weight respect, HEAD/GET
parity across four Accept shapes, and auth-vs-anon parity. 470/470.
@melvincarvalho
melvincarvalho requested a review from Copilot April 27, 2026 06:28
@melvincarvalho
melvincarvalho merged commit 52f1556 into gh-pages Apr 27, 2026
4 checks passed
@melvincarvalho
melvincarvalho deleted the issue-325-conneg-qweight branch April 27, 2026 06:48
@melvincarvalho
melvincarvalho removed the request for review from Copilot April 27, 2026 06:49
melvincarvalho added a commit that referenced this pull request May 2, 2026
* Phase 1: embed JSON-LD data island in mashlib HTML wrapper (#7)

The mashlib HTML wrapper now carries the originating resource's
JSON-LD bytes as a `<script type="application/ld+json"
id="dataisland" data-uri="…">` block. Phase 1 is strictly additive:

- mashlib still XHR-fetches as before (no behaviour change yet);
- browsers ignore the script body since its MIME isn't JS;
- but anything that knows to look for `application/ld+json` islands
  — search engines (Rich Results), archival crawlers, scrapers,
  static-site exporters, future LWS-aware tooling — now finds the
  data without a second HTTP request.

This sets up Phase 2, where a small inline shim will patch mashlib's
`fetcher` to read the island instead of refetching, eliminating the
double-fetch that motivated the recent Vary/Cache work
(#316, #325/#326).

Format: JSON-LD (not Turtle), since:
- JSS stores JSON-LD natively → zero server-side conversion;
- `<script type="application/ld+json">` is the standardised way to
  embed structured data on the web;
- aligns with our LWS / CID work — an HTML-only LWS verifier can
  parse the embedded JSON-LD and find the CID `service[]` entry
  without conneg.

Sites:
- src/mashlib/index.js: `generateDatabrowserHtml(url, cdn, {embedJsonLd})`
  emits the island when a payload is supplied. New
  `DATA_ISLAND_MAX_BYTES = 256 * 1024` size cap silently drops the
  island for oversize resources; the wrapper still works because
  mashlib falls back to its XHR path.
- src/handlers/resource.js: passes the JSON-LD bytes when the
  stored content type is `application/ld+json` (resources) or
  whenever a container listing is generated.

Security:
- `</script>` substrings inside the JSON-LD body are escaped to
  `<\/script>` so a user-PUTted resource can't close the script
  tag prematurely and inject HTML.
- `<!--` substrings are escaped to neutralise comment-escape tricks.
- The `data-uri` attribute is HTML-entity-encoded (&, ", <, >) so
  attribute-quote injection isn't possible either.

9 new tests in test/data-island.test.js cover: emission shape, the
back-compat omission case, the size cap, both `</script>` and `<!--`
escapes, attribute encoding, and live HTTP integration for both
resources and container listings — plus a negative test that the
mashlib XHR path (Accept: application/ld+json) still gets RDF rather
than the wrapper.

Phase 1 of #7. 546/546 green.

* Address Copilot round-1 on #343

Six points, all real:

- src/mashlib/index.js: escapeForScriptBlock now encodes EVERY `<`
  byte as the JSON Unicode escape `<`. The previous narrow regex
  only caught literal `</script>` and `<!--`, but HTML parsers
  terminate a <script> element on the prefix `</script` regardless of
  what follows — `</script >`, `</script\n>`, `</SCRIPT>` and friends
  all close it. After this transform the body cannot contain a
  literal `<` at all, so no end-tag (or comment, or CDATA) can
  possibly start. JSON-LD semantics are preserved because JSON
  parsers decode `<` back to `<` natively.
- src/mashlib/index.js: header comment rewritten to accurately
  describe the new strategy (the previous comment had been internally
  inconsistent and described a different approach).
- src/mashlib/index.js: generateModuleDatabrowserHtml now also
  accepts opts.embedJsonLd and emits the data island. Previously
  module-mode mashlib deployments missed the feature entirely.
- src/handlers/resource.js: both wrapper paths (CDN/local + module)
  now pass the JSON-LD content through. Module mode wraps consume
  the same opts shape.
- src/handlers/resource.js: cap-aware short-circuit — when
  stats.size > DATA_ISLAND_MAX_BYTES the handler skips storage.read
  entirely. Previously a 10MB JSON-LD resource would load into
  memory on every HTML navigation only to have the island silently
  dropped by generateDatabrowserHtml.
- test/data-island.test.js: replaced the single-variant
  </script> test with a parametrised loop covering exact/whitespace/
  newline/uppercase/mixed-case end-tag forms plus <!--. All assert
  the same invariant: NO literal `<` survives in the script body.
  Added a module-mode emission test.

Full suite: 551/551 (5 new tests).

* Address Copilot round-2 on #343 (clarity / dedupe)

Three points:

- src/mashlib/index.js: escapeForScriptBlock comment now says
  literally "the six-character JSON escape sequence \\u003c" instead of
  "<", which had ambiguously rendered the literal escape as just
  `<` and made the security rationale read backwards.
- src/mashlib/index.js: dataIsland's URI escaping now reuses the
  existing escapeHtml() helper from this file instead of an inline
  duplicate. Function declarations hoist, so the call site can
  precede the helper definition without reordering.
- test/data-island.test.js: parametrised escape-test header comment
  rewritten to describe the actual transform (`\\u003c`).

Pure clarity / refactor; no behaviour change.

Full suite: 551/551.

* Address Copilot round-3 on #343 (comment + buffer input)

- src/mashlib/index.js: rewrote the escapeForScriptBlock comment to
  describe the transform unambiguously: "the JSON string-escape for
  U+003C — the six characters backslash-u-0-0-3-c". The previous
  attempt kept losing the literal `\\u003c` text to the rendering
  pipeline and reading as `<` in source, inverting the security
  rationale.
- src/mashlib/index.js: escapeForScriptBlock now coerces input via
  `String(jsonLdString)`, so a Buffer (e.g. straight from
  `storage.read()`) passes through cleanly instead of throwing on
  `.replace`. Buffer.byteLength already accepted both, so dataIsland's
  size cap was already buffer-safe.
- src/mashlib/index.js: JSDoc for `opts.embedJsonLd` now declares
  `string|Buffer` on both wrapper functions.
- test/data-island.test.js: new regression test passing a Buffer
  payload directly, asserting the island emits with the expected
  body content.

Full suite: 552/552.

* Apply size cap to escaped body, not raw input (#343 round-4)

DATA_ISLAND_MAX_BYTES was being checked against the *pre-escape*
input. escapeForScriptBlock can expand input up to 6x (each `<` byte
becomes the six-char escape `\\u003c`), so a `<`-heavy body just
under the cap would balloon the HTML response well past it.

Move the size check after the escape and gate on the bytes that
will actually appear in the response. Comment updated accordingly.

New regression test: a half-cap-size payload of pure `<` bytes —
6x expansion guarantees it would slip past the old check but is
correctly rejected now.

Full suite: 553/553.

* Address PR #343 round-5 review (Copilot)

Three small refinements:

1. dataIsland(): add cheap raw-byte pre-check before escaping. Since
   escapeForScriptBlock can only grow the body (each `<` becomes 6
   bytes), a raw payload already over the cap is guaranteed to be
   over after escaping — skip the work. Post-escape check still
   guards the `<`-heavy expansion case.

2. mashlib/index.js dataIsland doc comment: clarify the 6x escape
   expansion using prose form for the escape sequence so it survives
   round-trips through tooling that might otherwise interpret the
   literal characters.

3. test/data-island.test.js post-escape cap test comment: same
   clarification.

No behaviour change beyond the pre-check fast path; all 553 tests
still pass.

* Address PR #343 round-6 review (Copilot)

Two genuine optimizations from this round (the other flags were
re-runs of points already addressed in earlier rounds — pre-check,
Buffer coercion, stats.size guard, module-mode wrapper — all already
present in HEAD):

1. Container path: use compact JSON.stringify(jsonLd) for the embed
   instead of serializeJsonLd(), which pretty-prints with 2-space
   indent. The HTTP body still uses serializeJsonLd; only the inlined
   data island goes compact, so we don't waste bytes against
   DATA_ISLAND_MAX_BYTES on whitespace nothing reads.

2. Resource path: pass the storage.read() Buffer through to
   embedJsonLd directly. dataIsland() already coerces Buffer → string
   via String(jsonLdString), so the eager buf.toString('utf8') in the
   handler was redundant.

All 553 tests still pass.

* PR #343 round-7 review: test comment clarity

Two doc-only fixes from this round; the other 7 inline comments
re-flag points already addressed in earlier rounds (raw-byte
pre-check, Buffer coercion, stats.size guard, module-mode wrapper,
compact embed serialization).

1. Header comment: include `data-uri="..."` in the script tag
   example so it matches the actual emission shape the tests assert.

2. Inline comment on the escape-presence assertion: refer to the
   six-character escape sequence in prose form ("backslash-u-0-0-3-c")
   so the source intent survives tooling that converts the literal
   characters back to `<`.

No code change; all 553 tests still pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authenticated GET on a container returns Turtle body with application/ld+json content-type, ignoring Accept

1 participant