Skip to content

Commit 58e5f8b

Browse files
melvincarvalhomelvincarvalho
authored
Phase 1: embed JSON-LD data island in mashlib HTML wrapper (JavaScriptSolidServer#7) (JavaScriptSolidServer#343)
* Phase 1: embed JSON-LD data island in mashlib HTML wrapper (JavaScriptSolidServer#7) The mashlib HTML wrapper now carries the originating resource's JSON-LD bytes as a `<script type="application/ld+json" id="dataisland" data-uri="…">` block. Phase 1 is strictly additive: - mashlib still XHR-fetches as before (no behaviour change yet); - browsers ignore the script body since its MIME isn't JS; - but anything that knows to look for `application/ld+json` islands — search engines (Rich Results), archival crawlers, scrapers, static-site exporters, future LWS-aware tooling — now finds the data without a second HTTP request. This sets up Phase 2, where a small inline shim will patch mashlib's `fetcher` to read the island instead of refetching, eliminating the double-fetch that motivated the recent Vary/Cache work (JavaScriptSolidServer#316, JavaScriptSolidServer#325/JavaScriptSolidServer#326). Format: JSON-LD (not Turtle), since: - JSS stores JSON-LD natively → zero server-side conversion; - `<script type="application/ld+json">` is the standardised way to embed structured data on the web; - aligns with our LWS / CID work — an HTML-only LWS verifier can parse the embedded JSON-LD and find the CID `service[]` entry without conneg. Sites: - src/mashlib/index.js: `generateDatabrowserHtml(url, cdn, {embedJsonLd})` emits the island when a payload is supplied. New `DATA_ISLAND_MAX_BYTES = 256 * 1024` size cap silently drops the island for oversize resources; the wrapper still works because mashlib falls back to its XHR path. - src/handlers/resource.js: passes the JSON-LD bytes when the stored content type is `application/ld+json` (resources) or whenever a container listing is generated. Security: - `</script>` substrings inside the JSON-LD body are escaped to `<\/script>` so a user-PUTted resource can't close the script tag prematurely and inject HTML. - `<!--` substrings are escaped to neutralise comment-escape tricks. - The `data-uri` attribute is HTML-entity-encoded (&, ", <, >) so attribute-quote injection isn't possible either. 9 new tests in test/data-island.test.js cover: emission shape, the back-compat omission case, the size cap, both `</script>` and `<!--` escapes, attribute encoding, and live HTTP integration for both resources and container listings — plus a negative test that the mashlib XHR path (Accept: application/ld+json) still gets RDF rather than the wrapper. Phase 1 of JavaScriptSolidServer#7. 546/546 green. * Address Copilot round-1 on JavaScriptSolidServer#343 Six points, all real: - src/mashlib/index.js: escapeForScriptBlock now encodes EVERY `<` byte as the JSON Unicode escape `<`. The previous narrow regex only caught literal `</script>` and `<!--`, but HTML parsers terminate a <script> element on the prefix `</script` regardless of what follows — `</script >`, `</script\n>`, `</SCRIPT>` and friends all close it. After this transform the body cannot contain a literal `<` at all, so no end-tag (or comment, or CDATA) can possibly start. JSON-LD semantics are preserved because JSON parsers decode `<` back to `<` natively. - src/mashlib/index.js: header comment rewritten to accurately describe the new strategy (the previous comment had been internally inconsistent and described a different approach). - src/mashlib/index.js: generateModuleDatabrowserHtml now also accepts opts.embedJsonLd and emits the data island. Previously module-mode mashlib deployments missed the feature entirely. - src/handlers/resource.js: both wrapper paths (CDN/local + module) now pass the JSON-LD content through. Module mode wraps consume the same opts shape. - src/handlers/resource.js: cap-aware short-circuit — when stats.size > DATA_ISLAND_MAX_BYTES the handler skips storage.read entirely. Previously a 10MB JSON-LD resource would load into memory on every HTML navigation only to have the island silently dropped by generateDatabrowserHtml. - test/data-island.test.js: replaced the single-variant </script> test with a parametrised loop covering exact/whitespace/ newline/uppercase/mixed-case end-tag forms plus <!--. All assert the same invariant: NO literal `<` survives in the script body. Added a module-mode emission test. Full suite: 551/551 (5 new tests). * Address Copilot round-2 on JavaScriptSolidServer#343 (clarity / dedupe) Three points: - src/mashlib/index.js: escapeForScriptBlock comment now says literally "the six-character JSON escape sequence \\u003c" instead of "<", which had ambiguously rendered the literal escape as just `<` and made the security rationale read backwards. - src/mashlib/index.js: dataIsland's URI escaping now reuses the existing escapeHtml() helper from this file instead of an inline duplicate. Function declarations hoist, so the call site can precede the helper definition without reordering. - test/data-island.test.js: parametrised escape-test header comment rewritten to describe the actual transform (`\\u003c`). Pure clarity / refactor; no behaviour change. Full suite: 551/551. * Address Copilot round-3 on JavaScriptSolidServer#343 (comment + buffer input) - src/mashlib/index.js: rewrote the escapeForScriptBlock comment to describe the transform unambiguously: "the JSON string-escape for U+003C — the six characters backslash-u-0-0-3-c". The previous attempt kept losing the literal `\\u003c` text to the rendering pipeline and reading as `<` in source, inverting the security rationale. - src/mashlib/index.js: escapeForScriptBlock now coerces input via `String(jsonLdString)`, so a Buffer (e.g. straight from `storage.read()`) passes through cleanly instead of throwing on `.replace`. Buffer.byteLength already accepted both, so dataIsland's size cap was already buffer-safe. - src/mashlib/index.js: JSDoc for `opts.embedJsonLd` now declares `string|Buffer` on both wrapper functions. - test/data-island.test.js: new regression test passing a Buffer payload directly, asserting the island emits with the expected body content. Full suite: 552/552. * Apply size cap to escaped body, not raw input (JavaScriptSolidServer#343 round-4) DATA_ISLAND_MAX_BYTES was being checked against the *pre-escape* input. escapeForScriptBlock can expand input up to 6x (each `<` byte becomes the six-char escape `\\u003c`), so a `<`-heavy body just under the cap would balloon the HTML response well past it. Move the size check after the escape and gate on the bytes that will actually appear in the response. Comment updated accordingly. New regression test: a half-cap-size payload of pure `<` bytes — 6x expansion guarantees it would slip past the old check but is correctly rejected now. Full suite: 553/553. * Address PR JavaScriptSolidServer#343 round-5 review (Copilot) Three small refinements: 1. dataIsland(): add cheap raw-byte pre-check before escaping. Since escapeForScriptBlock can only grow the body (each `<` becomes 6 bytes), a raw payload already over the cap is guaranteed to be over after escaping — skip the work. Post-escape check still guards the `<`-heavy expansion case. 2. mashlib/index.js dataIsland doc comment: clarify the 6x escape expansion using prose form for the escape sequence so it survives round-trips through tooling that might otherwise interpret the literal characters. 3. test/data-island.test.js post-escape cap test comment: same clarification. No behaviour change beyond the pre-check fast path; all 553 tests still pass. * Address PR JavaScriptSolidServer#343 round-6 review (Copilot) Two genuine optimizations from this round (the other flags were re-runs of points already addressed in earlier rounds — pre-check, Buffer coercion, stats.size guard, module-mode wrapper — all already present in HEAD): 1. Container path: use compact JSON.stringify(jsonLd) for the embed instead of serializeJsonLd(), which pretty-prints with 2-space indent. The HTTP body still uses serializeJsonLd; only the inlined data island goes compact, so we don't waste bytes against DATA_ISLAND_MAX_BYTES on whitespace nothing reads. 2. Resource path: pass the storage.read() Buffer through to embedJsonLd directly. dataIsland() already coerces Buffer → string via String(jsonLdString), so the eager buf.toString('utf8') in the handler was redundant. All 553 tests still pass. * PR JavaScriptSolidServer#343 round-7 review: test comment clarity Two doc-only fixes from this round; the other 7 inline comments re-flag points already addressed in earlier rounds (raw-byte pre-check, Buffer coercion, stats.size guard, module-mode wrapper, compact embed serialization). 1. Header comment: include `data-uri="..."` in the script tag example so it matches the actual emission shape the tests assert. 2. Inline comment on the escape-presence assertion: refer to the six-character escape sequence in prose form ("backslash-u-0-0-3-c") so the source intent survives tooling that converts the literal characters back to `<`. No code change; all 553 tests still pass.
1 parent f09ef31 commit 58e5f8b

3 files changed

Lines changed: 333 additions & 11 deletions

File tree

‎src/handlers/resource.js‎

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ import {
1414
} from '../rdf/conneg.js';
1515
import { emitChange } from '../notifications/events.js';
1616
import { checkIfMatch, checkIfNoneMatchForGet, checkIfNoneMatchForWrite } from '../utils/conditional.js';
17-
import { generateDatabrowserHtml, generateModuleDatabrowserHtml, shouldServeMashlib } from '../mashlib/index.js';
17+
import { generateDatabrowserHtml, generateModuleDatabrowserHtml, shouldServeMashlib, DATA_ISLAND_MAX_BYTES } from '../mashlib/index.js';
1818

1919
/**
2020
* Live reload script - injected into HTML when --live-reload is enabled
@@ -238,9 +238,21 @@ export async function handleGet(request, reply) {
238238

239239
// Check if we should serve Mashlib data browser for containers
240240
if (shouldServeMashlib(request, request.mashlibEnabled, 'application/ld+json')) {
241+
// Phase 1 of #7: also embed the container's JSON-LD listing as a
242+
// data island so consumers that look for `<script
243+
// type="application/ld+json">` (search-engine rich-results,
244+
// archival crawlers, future mashlib zero-fetch path) get the data
245+
// without a second request. Use compact (no-whitespace) form for
246+
// the embed so we don't burn bytes against DATA_ISLAND_MAX_BYTES
247+
// on indentation that nothing will ever read.
248+
const embedJsonLd = JSON.stringify(jsonLd);
241249
const html = request.mashlibModule
242-
? generateModuleDatabrowserHtml(request.mashlibModule)
243-
: generateDatabrowserHtml(resourceUrl, request.mashlibCdn ? request.mashlibVersion : null);
250+
? generateModuleDatabrowserHtml(request.mashlibModule, resourceUrl, { embedJsonLd })
251+
: generateDatabrowserHtml(
252+
resourceUrl,
253+
request.mashlibCdn ? request.mashlibVersion : null,
254+
{ embedJsonLd }
255+
);
244256
const headers = getAllHeaders({
245257
isContainer: true,
246258
etag: stats.etag,
@@ -318,9 +330,31 @@ export async function handleGet(request, reply) {
318330
// Check if we should serve Mashlib data browser
319331
// Only for RDF resources when Accept: text/html is requested
320332
if (shouldServeMashlib(request, request.mashlibEnabled, storedContentType)) {
333+
// Phase 1 of #7: embed the resource's JSON-LD bytes as a data
334+
// island when it's already JSON-LD (the JSS-native format). Other
335+
// formats are out of Phase-1 scope; the wrapper still loads
336+
// correctly and mashlib XHR-fetches as before.
337+
//
338+
// Cap-aware short-circuit: skip the read entirely when the file is
339+
// already over the embed cap. The island would be dropped anyway,
340+
// and large JSON-LD resources would otherwise load into memory on
341+
// every HTML navigation.
342+
let embedJsonLd;
343+
if (storedContentType === 'application/ld+json' &&
344+
stats.size <= DATA_ISLAND_MAX_BYTES) {
345+
// dataIsland() in mashlib/index.js coerces Buffer → string itself,
346+
// so we hand it the Buffer directly instead of allocating a UTF-8
347+
// string copy on every navigation.
348+
const buf = await storage.read(storagePath);
349+
if (buf) embedJsonLd = buf;
350+
}
321351
const html = request.mashlibModule
322-
? generateModuleDatabrowserHtml(request.mashlibModule)
323-
: generateDatabrowserHtml(resourceUrl, request.mashlibCdn ? request.mashlibVersion : null);
352+
? generateModuleDatabrowserHtml(request.mashlibModule, resourceUrl, { embedJsonLd })
353+
: generateDatabrowserHtml(
354+
resourceUrl,
355+
request.mashlibCdn ? request.mashlibVersion : null,
356+
{ embedJsonLd }
357+
);
324358
const headers = getAllHeaders({
325359
isContainer: false,
326360
etag: stats.etag,

‎src/mashlib/index.js‎

Lines changed: 85 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,97 @@
44
* Generates HTML wrapper that loads SolidOS Mashlib from CDN.
55
* When a browser requests an RDF resource with Accept: text/html,
66
* we return this wrapper which then fetches and renders the data.
7+
*
8+
* Phase 1 of #7: when the originating resource is reasonably small
9+
* RDF, the JSON-LD bytes are embedded in the wrapper as a `<script
10+
* type="application/ld+json" id="dataisland" data-uri="…">` block.
11+
* Browsers ignore non-JS script bodies, so this is harmless to all
12+
* existing clients (mashlib still XHR-fetches today). It immediately
13+
* benefits anything that knows to look for `application/ld+json`
14+
* islands — search engine rich-results, archival crawlers, scrapers,
15+
* static-site exporters — and gives Phase 2 a zero-network fast path.
16+
*/
17+
18+
/**
19+
* Cap on how much JSON-LD we'll inline. A 256 KB resource fits any
20+
* realistic profile, type index, or container listing. Above that we
21+
* drop the island and let the existing XHR path handle it so we don't
22+
* make every navigation re-download a multi-megabyte resource.
23+
*/
24+
export const DATA_ISLAND_MAX_BYTES = 256 * 1024;
25+
26+
/**
27+
* Escape a JSON-LD body for safe inclusion inside `<script
28+
* type="application/ld+json">…</script>`.
29+
*
30+
* Browsers don't execute the script (wrong MIME), but the HTML parser
31+
* still scans the body for an end-of-script tag. The relevant rule:
32+
* any `</` followed by `script` (case-insensitive) terminates the
33+
* element regardless of what follows — `</script>`, `</script >`,
34+
* `</script\n>`, `</SCRIPT>` and friends all close it. Escaping just
35+
* the literal `</script>` token is too narrow.
36+
*
37+
* The robust fix is to replace every literal `<` byte in the body with
38+
* the JSON string-escape for U+003C — the six characters
39+
* backslash-u-0-0-3-c (the same form the implementation emits below).
40+
* JSON-LD is JSON, and a JSON parser decodes that escape back to a
41+
* literal `<` natively, so document semantics are preserved. After
42+
* this transform the body literally cannot contain a `<` byte — so no
43+
* end-tag (or comment, CDATA, etc.) can possibly start.
44+
*/
45+
function escapeForScriptBlock(jsonLdString) {
46+
return String(jsonLdString).replace(/</g, '\\u003c');
47+
}
48+
49+
/**
50+
* Build the data-island `<script>` block for the given JSON-LD payload.
51+
* Returns an empty string if the payload is missing or over the size
52+
* cap so callers can unconditionally interpolate `dataIsland(...)`.
53+
*
54+
* The cap applies to the *escaped* body — i.e. the bytes that will
55+
* actually appear in the HTTP response. `escapeForScriptBlock` can
56+
* expand input up to 6x (each literal `<` becomes the 6-byte JSON
57+
* escape sequence backslash-u-0-0-3-c), so checking the raw input
58+
* size alone could let an HTML response balloon past the cap.
59+
*
60+
* Two-stage check:
61+
* 1. Cheap raw-byte pre-check — escape can only grow the body,
62+
* so a raw payload already over the cap is guaranteed to be
63+
* over after escaping; drop without doing the work.
64+
* 2. Post-escape check — catches the rare case where input was
65+
* under the cap but expanded above it (`<`-heavy bodies).
766
*/
67+
function dataIsland(resourceUrl, jsonLdString) {
68+
if (!jsonLdString) return '';
69+
const raw = String(jsonLdString);
70+
if (Buffer.byteLength(raw, 'utf8') > DATA_ISLAND_MAX_BYTES) return '';
71+
const safeBody = escapeForScriptBlock(raw);
72+
if (Buffer.byteLength(safeBody, 'utf8') > DATA_ISLAND_MAX_BYTES) return '';
73+
const safeUri = escapeHtml(String(resourceUrl));
74+
return `<script type="application/ld+json" id="dataisland" data-uri="${safeUri}">${safeBody}</script>`;
75+
}
876

977
/**
1078
* Generate Mashlib databrowser HTML
1179
*
12-
* @param {string} resourceUrl - The URL of the resource being viewed (unused, kept for API compatibility)
80+
* @param {string} resourceUrl - The URL of the resource being viewed
1381
* @param {string} cdnVersion - If provided, load mashlib from unpkg CDN (e.g., "2.0.0")
82+
* @param {object} [opts]
83+
* @param {string|Buffer} [opts.embedJsonLd] - JSON-LD body to inline
84+
* as a `<script type="application/ld+json">` data island. Accepts a
85+
* UTF-8 string or a Buffer (coerced via `String()`). Honors a 256 KB
86+
* size cap; oversize payloads are silently dropped. Phase 1 of #7.
1487
* @returns {string} HTML content
1588
*/
16-
export function generateDatabrowserHtml(resourceUrl, cdnVersion = null) {
89+
export function generateDatabrowserHtml(resourceUrl, cdnVersion = null, opts = {}) {
90+
const island = dataIsland(resourceUrl, opts.embedJsonLd);
1791
if (cdnVersion) {
1892
// CDN mode - use script.onload to ensure mashlib is fully loaded before init
1993
// This avoids race conditions with defer + DOMContentLoaded
2094
const cdnBase = `https://unpkg.com/mashlib@${cdnVersion}/dist`;
2195
return `<!doctype html><html><head><meta charset="utf-8"/><title>SolidOS Web App</title>
2296
<link href="${cdnBase}/mash.css" rel="stylesheet"></head>
23-
<body id="PageBody"><header id="PageHeader"></header>
97+
<body id="PageBody">${island}<header id="PageHeader"></header>
2498
<div class="TabulatorOutline" id="DummyUUID" role="main"><table id="outline"></table><div id="GlobalDashboard"></div></div>
2599
<footer id="PageFooter"></footer>
26100
<script>
@@ -37,22 +111,27 @@ export function generateDatabrowserHtml(resourceUrl, cdnVersion = null) {
37111
// Local mode - use defer (reliable when served locally)
38112
return `<!doctype html><html><head><meta charset="utf-8"/><title>SolidOS Web App</title><script>document.addEventListener('DOMContentLoaded', function() {
39113
panes.runDataBrowser()
40-
})</script><script defer="defer" src="https://github.com/mashlib.min.js"></script><link href="https://github.com/mash.css" rel="stylesheet"></head><body id="PageBody"><header id="PageHeader"></header><div class="TabulatorOutline" id="DummyUUID" role="main"><table id="outline"></table><div id="GlobalDashboard"></div></div><footer id="PageFooter"></footer></body></html>`;
114+
})</script><script defer="defer" src="https://github.com/mashlib.min.js"></script><link href="https://github.com/mash.css" rel="stylesheet"></head><body id="PageBody">${island}<header id="PageHeader"></header><div class="TabulatorOutline" id="DummyUUID" role="main"><table id="outline"></table><div id="GlobalDashboard"></div></div><footer id="PageFooter"></footer></body></html>`;
41115
}
42116

43117
/**
44118
* Generate ES module-based databrowser HTML
45119
*
46120
* @param {string} moduleUrl - URL to the ES module entry point
121+
* @param {string} resourceUrl - The URL of the resource being viewed
122+
* @param {object} [opts]
123+
* @param {string|Buffer} [opts.embedJsonLd] - JSON-LD body for the
124+
* data island, same contract as `generateDatabrowserHtml`. Phase 1 of #7.
47125
* @returns {string} HTML content
48126
*/
49-
export function generateModuleDatabrowserHtml(moduleUrl) {
127+
export function generateModuleDatabrowserHtml(moduleUrl, resourceUrl = '', opts = {}) {
50128
const cssUrl = moduleUrl.replace(/\.js$/, '.css');
129+
const island = dataIsland(resourceUrl, opts.embedJsonLd);
51130
return `<!doctype html><html lang="en"><head><meta charset="utf-8"/>
52131
<meta name="viewport" content="width=device-width, initial-scale=1">
53132
<title>Solid Data Browser</title>
54133
<link rel="stylesheet" href="${cssUrl}"></head>
55-
<body><div id="mashlib"></div>
134+
<body>${island}<div id="mashlib"></div>
56135
<script type="module" src="${moduleUrl}"></script>
57136
</body></html>`;
58137
}

0 commit comments

Comments
 (0)