Add streaming HTML insertion methods - #12758

Reviewed at 1f7ee1a, together with web-platform-tests/wpt#61748 at 8c22eae. This covers the whole spec diff and all 53 test files. Each finding below was checked against the spec text and the test source. The spec diff has no new blockers. The test PR hasn't been updated since before the last three spec commits, and four of its tests are broken.

Blockers

• wpt domparsing/stream-html-reentrancy.html:128-157: since 1f7ee1a this test contradicts the spec.
  • Setup: the parser resumes from the queued task after slow.js. No write is in progress, so the inline script's abort() runs the sink's abort algorithm synchronously, and that sets fragment parser abort requested.
  • The tokenizer now stops when that flag is set (source:144213-144216), so <p id='b'> and the rest_parsed script are never parsed.
  • The test still expects ["a", "b"] and rest_parsed === true.
  • Before 1f7ee1a an abort during a write and an abort from the task both parsed the rest of the input. Now only the write case does.
  • Either update the test to expect ["a"], or confirm the asymmetry is intended. If it is, the note at source:128168-128171 should say parsing stops after the current token, not just that the abort is deferred.
• wpt domparsing/stream-html-parser-blocking-script.html:315-332 (D5) and 366-383 (D7) can't fail. The unclosed script was never prepared, so it's still parser-inserted. Appending text reaches the post-connection steps, which return early for parser-inserted scripts (source:68511-68518) whether or not already started is set. Consider inserting script.cloneNode(true) instead, as unclosed-script-already-started.html does: cloning copies already started, and the clone isn't parser-inserted.
• wpt domparsing/stream-html-parser-blocking-script.html:406-433 (F5) and stream-html-stylesheet-does-not-block-scripts.html:48-65 can't fail. A <link> made with createElement() never contributes a script-blocking style sheet, because it wasn't created by the document's parser (source:19484-19491). The tests would pass even in a browser that wrongly waited on the live document's sheets. Consider creating the pending sheet through a parser, e.g. an iframe with an unclosed document.open()/document.write('<link rel=stylesheet href=slow.css?pipe=trickle(d3)><div id=target>').
• wpt domparsing/stream-html-parser-blocking-script.html:522-543 (F12) can't fail. After iframe.remove(), parent is null in the iframe (source:98451-98454), so parent.iframe_blocked_ran = true would throw even if the script ran. Consider having the streamed script call a function defined in the parent's realm.

Suggestions

• source:151674-151677: the new "the script element is connected" condition also applies to document parsing. For example, <div id=d><script>d.remove()</script><svg><script>…</script></svg> no longer processes the SVG script. I think this matches engines, but it changes behavior outside streaming. Consider mentioning it in the commit message and adding a test.
• PR description: the SVG bullet is inverted. It says SVG scripts are processed only when the script target document "is the parser's Document", but the spec processes them when it is not. The bullet also leaves out the new connected condition.
• wpt: these behaviors from 39eb4dc, b07a79f and 1f7ee1a have no tests:
  • Adoption agency at the root after the reference node is displaced, e.g. <b><p>x then <script>ref.remove()</script></b>. The <p> should stay and the write should reject.
  • <template for> after the reference node is displaced.
  • A chunk whose only token after displacement is whitespace or a comment should error the stream.
  • close() after displacement with no further write should resolve, since only the write algorithm checks (source:128093-128096).
  • An SVG script with runScripts: true that isn't connected (disconnected target, or an ancestor removed by the sanitizer) must not run.
  • Displacement with a ShadowRoot target. The domintro promises an error for shadowRoot.streamPrependHTML(), but only element targets are tested.
  • Displacement for streamAfterHTML, which is left out of every displacement loop in stream-insertion-logic.html (lines 90 and 218).
• wpt domparsing/stream-insertion-logic.html:313-392: the parser now stops right after the first token following displacement. So the nested custom elements, <section> subtree and later <img>s in these two tests are never tokenized, and their assertions don't test what the test names say. Consider one test per kind of first token: a custom element, an <img>, and a child of a replaceWithChildrenElements element.
• wpt domparsing/stream-html-with-trusted-types-adopt.tentative.html:59-70 and stream-html-with-trusted-types-from-non-tt.tentative.html:51-56: these expect Trusted Types enforcement to follow the node document after adoption. The method steps use this's relevant global object, which adoption doesn't change. Either switch the spec to the node document's relevant global object (which existing Trusted Types adoption tests for setAttribute() assume), or flip these expectations.
• wpt domparsing/stream-sanitizer-resource-fetch.html:135: the control expects exactly one request. The spec now allows an extra speculative fetch for non-sanitizing fragment parsers (source:147032), and deduplication is left to caching rules that aren't specified. Consider asserting at least one request.
  • [Edit: It may be true that deduplication/caching isn't properly specified yet, but double-fetching I would say would be considered a bug. So I think the test should still assert one request.]
• wpt html/semantics/scripting-1/the-template-element/template-for/stream-append-has-invalidation.html:37-41: "Green (no span)" has no <?start name="content"?>, and </template> removes the consumed start marker. From the second iteration on, <template for=content> matches nothing, so only the first iteration tests anything. Consider prefixing that content with a new start marker.
• wpt domparsing/move-head-during-prepend-before-close-crash.html:9-15: per spec documentElement.streamPrependHTML() throws, so a conforming implementation always takes the body fallback and the head removal doesn't matter. Consider documentElement.streamHTML(), which is allowed, and class="test-wait" until close() settles.
  • [Edit: since this was a crash test, keeping the tested path seems valuable even if the spec requirements have changed. Could maybe change it to a regular test to expect the exception, and add a separate test for streamHTML()+close().]
• wpt domparsing/stream-html-abort-unblocks-rendering.html:33-49, 82-98: both streams start at once, so if the defer case fails to unblock rendering, the first test's requestAnimationFrame check fails too. Consider running them one after the other, or splitting them into two files.

Nits

• wpt domparsing/stream-fragment-context.html:188: >= 1 passes for Before and After even if nothing is inserted, since initialTd stays. Consider asserting the exact tr.innerHTML.
• wpt domparsing/stream-html-stylesheet-does-not-block-scripts.html:84-88: the comment says close() doesn't wait for the parser, but it does (source:128144-128151), so the step_wait isn't needed.
• wpt domparsing/stream-insertion-logic.html:145: if (await stream_insert_before(...)) is always true, so the if can go. Lines 115 and 135 pass "DOMException" as the description argument to promise_rejects_dom.
• wpt template-for/stream-append-has-invalidation.html:47 and stream-append-positional-pseudo-class-invalidation.html:81: the test names are empty strings.
• wpt domparsing/stream-html-sanitizer.html:305: in the test name, "effect" should be "affect".

Reviewed at 60f91db and again at b9dec97, together with web-platform-tests/wpt#61748 at 5b249c3 and 49e13af

All findings from the previous review are addressed. I've pushed fixes for the new findings below to both branches, one commit each, and self-reviewed them: lint.sh and ./wpt lint are clean. With these fixes, all 47 test files load in Chrome Canary with experimental features and no harness error. The remaining failures are cases where Chrome hasn't caught up with the spec. The new document-parsing SVG test (stream-html-run-scripts-option.html), plus a srcdoc variant, passes in Firefox Nightly, Chrome Canary and Safari TP.

Blockers

• wpt domparsing/stream-insertion-logic.html:599: the JS comment contains a literal </script>, which ends the test's <script> element early. The whole file has failed with a SyntaxError since web-platform-tests/wpt@8c22eae (my earlier review missed this). Fixed in web-platform-tests/wpt@b9f9b63.
• wpt domparsing/stream-insertion-logic.html:487: in the new adoption agency test, p.textContent is "xwindow.remove_ref();", because the second write's <script> is inserted inside the still-open <p>. So the test fails in a conforming implementation. Fixed in web-platform-tests/wpt@4c40ec8 by asserting p.firstChild.data.
• wpt domparsing/stream-html-run-scripts-option.html:153: the new test name contains a literal </script>, which ends the inline script. The file failed with a SyntaxError and none of its subtests ran. Fixed in web-platform-tests/wpt@2afe799.

Suggestions

• source:152171: before 60f91db, the first speculative fetch rule only applied when the speculative HTML parser met a base or meta element. 60f91db made it apply to every element created in a document with a browsing context. For example, an innerHTML <base href> on a detached div would change URLs for later speculative fetches, even though it's never inserted or processed. Fixed in b9dec97 by restoring the original "If the speculative HTML parser encounters one of the following elements" wording.
• wpt domparsing/move-head-during-prepend-before-close-crash.html:19-21: my earlier suggestion was off. streamHTML() removes head synchronously (source:127642-127643), so document.head?.remove() did nothing. Fixed in web-platform-tests/wpt@49e13af: the test writes <head></head> first, so there's a head to detach before close().
• wpt domparsing/stream-html-with-trusted-types-adopt.tentative.html and stream-html-with-trusted-types-from-non-tt.tentative.html: my earlier suggestion offered flipping these, but I hadn't checked engines. After adoption, Chrome Canary and Firefox Nightly enforce Trusted Types based on the node document for innerHTML and setHTMLUnsafe() (Chrome for the streaming methods too), even though HTML passes this's relevant global object for all of them. The existing trusted-types/Element-setAttribute-respects-Elements-node-documents-globals-CSP-after-adoption-* tests assume the node document too. Fixed in web-platform-tests/wpt@38f96d8 by reverting the flip, so these tentative tests match engines. The spec mismatch covers every Trusted Types sink, so a separate issue for it seems better than changing only the streaming methods here.

Nits

• source:147111: the note said descendants of elements the sanitizer replaces with their children get no speculative fetch. Those descendants are redirected into the real target (adjusted insertion location), so they do get one. Fixed in e0bc188 to say "descendants of removed or dropped elements".
• source:147007: inside the speculative parser, "the parser's Document" could be read as the isomorphic copy that parser writes into (source:152237-152240). That copy's list of speculative fetch URLs resets on every restart. Fixed in 58be8b0 by saying which Document is meant.
• wpt domparsing/stream-insertion-logic.html:430: if an implementation wrongly loads the dropped <img>, the load can happen after cleanup has deleted window.on_dropped_img_event. That gives a harness ERROR instead of a clean FAIL. Fixed in web-platform-tests/wpt@e8b1f5b by calling the handler with ?.().

Reviewed at b9dec97, together with web-platform-tests/wpt#61748 at 38f96d8

This was a fresh full pass: separate reviews of the spec's API layer, its tree construction changes, and the tests. I also ran all test files in Chrome Canary with experimental features, and in Firefox Nightly, which doesn't implement streaming, to catch tests that pass anyway. Line numbers below are at the reviewed commits.

I pushed a fix for each finding, one commit each, except where noted, and self-reviewed them: lint.sh and ./wpt lint are clean, and the changed test files run in Chrome Canary and Firefox Nightly with no harness errors. Of the two new tests, Chrome Canary passes the SVG one and fails the registry one, because it still uses the old document's definition.

Open questions

• @noamr: content inside an element the sanitizer replaces with its children is silently dropped when script moves the foster-parenting table it was stored against (see the source:147337 suggestion below). Do you want a fallback there, e.g. to the stream's own insertion point, or is dropping fine?

Blockers

• source:151682-151686: a streamed SVG script still ran after the target was moved to another document. With runScripts: true into a target in document A, if script moves the target into an iframe's document B, a later <svg><script> is created in B, is connected, and the script target document (A) isn't the parser's Document, so it was processed in B. HTML scripts are stopped by the intended-document check (source:68687). Fixed in 604b4c7 by also requiring the script target document to be null (document parsing) or the script element's node document. Test: web-platform-tests/wpt@41f8fb7.
• wpt domparsing/stream-html-abort-unblocks-rendering.html:72-99: the defer test couldn't fail, and my earlier suggestion caused this. Once the tests ran one after the other, the defer test started after <body> existed, so the document no longer allows adding render-blocking elements (source:12419-12421), and there was nothing for the abort to unblock. Fixed in web-platform-tests/wpt@b32d9bc by moving it into stream-html-abort-unblocks-rendering-defer.html.

Suggestions

• source:147034-147046: top-level elements got their registry from the parser's root html element. That root is created with the target's registry (source:153201-153204) and isn't updated when the target is adopted into another document. So after a move, new top-level elements were created in the new document with the old document's global registry. Fixed in 6e3cc21 by looking up the registry from the target in that case. Test: web-platform-tests/wpt@a052219.
• source:147337: for an element the sanitizer replaces with its children, the stored (targetParent, referenceChild) can be a foster-parenting table. If script moves the table, every later child of the replaced element is silently dropped, and the stream doesn't error, because the stream's own referenceChild is fine (source:128061-128062). Without the sanitizer, those children go into the element. Falling back to (redirectedParent, null) would append after the stream's referenceChild when redirectedParent is the target, so I didn't change the behavior (see the open question above). f4fb692 fixes the note at source:147274-147281, which said this case errors the stream.
• wpt: about 50 subtests passed in Firefox Nightly without the feature. The "throws TypeError" Unsafe cases passed because calling a missing method also throws a TypeError. Fixed in web-platform-tests/wpt@544374d by asserting the method exists first.
• wpt domparsing/stream-html-with-trusted-types-createParserOptions-order.html:40: for Append and Prepend the receiver was a div, which never reaches the html-element check (source:127726, source:127749). So the ordering wasn't tested for those two. Fixed in web-platform-tests/wpt@c8aaf82.
• wpt domparsing/stream-html-parser-blocking-script.html:35-56: both scripts pushed "slow", so the order wasn't observable. Fixed in web-platform-tests/wpt@d5ab8e7 with an optional tag parameter for slow.js.
• wpt domparsing/stream-html-parser-blocking-script.html: several cleanups, and waitForResource(), wait on promises that never settle when the behavior is missing or wrong. One failure then stalled the rest of the file with a harness timeout instead of failing. Fixed in web-platform-tests/wpt@676cdb1 by giving up after 3s.

Nits

• source:127294-127296: the note said scripts stop executing once the target moves to another document. That's only true while the target is in that document. If the target is disconnected at </script>, the script runs when the target is later inserted, as with document parsing. Fixed in 1cc0458.
• wpt domparsing/stream-html-run-scripts-option.html:113, 212: the queries could match elements left in #placeholder by the previous test. Fixed in web-platform-tests/wpt@af63853.

Reviewed at 604b4c7, together with web-platform-tests/wpt#61748 at f50b348

All findings from my previous review are addressed, and f50b348 covers the decision to drop children of a replaced element after its table moves. I've pushed a fix for each new finding below to both branches, one commit each, and self-reviewed them: lint.sh, the html-build build and ./wpt lint are clean. All 49 changed test files run in Chrome Canary (experimental features) and Firefox Nightly with no harness errors or timeouts. The remaining Chrome failures are places where Chrome doesn't match the spec yet. The new document-parsing SVG test passes in Chrome Canary and Firefox Nightly, and fails in Safari TP.

Open questions

• @noamr: Chrome Canary differs from the spec on two of the new tests. It still runs a streamed script when a microtask at that script's </script> checkpoint aborts the stream (see the source:152107 suggestion). And it doesn't run connectedCallback synchronously for a reconstructed formatting element (see the source:147550 suggestion). Are these worth Chromium bugs?
• Follow-up: in Document.parseHTML() with a sanitizer, associating a form control with the form pointer based on targetParent would match the unsanitized parse. The PR made that change, and I reverted it (see the source:147229 suggestion). If you want it, it seems better as a separate change with a test.

Blockers

• source:147295-147355, insert an element at the adjusted insertion location: the PR moved the structural early returns after the sanitizer step, which made Assert: if targetParent is a Document, then action is not "Replace with children" reachable. Example: a streamed script runs document.replaceChild(t, document.documentElement) on the foster-parenting table, then a replaceWithChildrenElements element follows. Fixed in f0e859b: the three checks from main come first again, and the abort and displacement checks stay after the sanitizer step. Test: web-platform-tests/wpt@6f78c59 (Chrome Canary puts the children inside the table, though its document parsing drops them).
• wpt domparsing/stream-insertion-logic.html:538-560: the whitespace and comment tests removed ref_node between writes, so the write algorithm rejected before parsing. The new early returns in insert a character and insert a comment were never reached. Fixed in web-platform-tests/wpt@c031e51 by displacing it in the same chunk.
• wpt domparsing/stream-fragment-context.html:327-341: the textarea test can't fail for popping, since a textarea's value follows its children either way. Fixed in web-platform-tests/wpt@3e855ed with a <style>, whose sheet is only created once it's popped.
• wpt domparsing/stream-insertion-logic.html:605-609: close() on an errored stream rejects with a TypeError (Streams' WritableStreamClose step 2), not HierarchyRequestError. Fixed in web-platform-tests/wpt@a8e5af7, which also checks that writer.closed rejects with HierarchyRequestError.

Suggestions

• source:147036-147047: children of an element the sanitizer replaces with its children took their registry from that element. It's never inserted, so it isn't updated when the target is adopted. My 6e3cc21 only fixed this for the root html element. Fixed in dc4940b by looking up from targetParent whenever the insertion is redirected. Test: web-platform-tests/wpt@54b8afb.
• source:152107-152113, abort a fragment parser: </script> performs a microtask checkpoint before the script is prepared. If a microtask there (e.g. a mutation observer) aborted the stream, the abort was only deferred and the script still ran. Fixed in 7bcc78b by marking the scripts on the stack of open elements as already started when deferring, as abort a parser does later anyway. Test: web-platform-tests/wpt@826ac2a.
• source:151690-151694: the rewritten SVG script condition only checked the node document when script target document is non-null. So document parsing still processed an SVG script that was moved to another document before its end tag, unlike HTML scripts. Fixed in 143c028. Test: web-platform-tests/wpt@5490fce. Chrome Canary and Firefox Nightly already don't run it; Safari TP does.
• source:147550, 147631 and 147659: insert a character, insert a comment and insert a processing instruction didn't check fragment parser abort requested, unlike elements. So text after an abort from a reaction in the same token was still inserted, e.g. after a reconstructed <b is=…> whose connectedCallback aborts. Fixed in 8aa768f, which also updates the abortAlgorithm note. Tests: web-platform-tests/wpt@d8948cc, plus web-platform-tests/wpt@f814dda for the existing element branch.
• source:147229: form association changed from intendedParent to targetParent. This step only runs for the document parser, so the change only affected Document.parseHTML() with a sanitizer, where it was unannounced and untested. Fixed in c02ebee by keeping intendedParent (see the follow-up above).
• wpt: several awaits on promises that only streamed scripts resolve, or on close() while a parser-blocking script is pending, had no timeout. One failure stalled the rest of the file. Fixed in web-platform-tests/wpt@b103b02. Also, stream-html-reentrancy.html tests could pass on globals left over from earlier ones. Fixed in web-platform-tests/wpt@1523d4e.
• wpt stream-html-with-trusted-types-runScripts-off.html and …-runScripts-cannot-be-turned-on.html: the safe-method cases couldn't fail, since safe methods always remove scripts. Fixed in web-platform-tests/wpt@4b13962 and web-platform-tests/wpt@b745071. The second file is renamed to …-runScripts-can-be-turned-on.html, which is what it tests.
• wpt domparsing/stream-html-run-scripts-option.html:141-154: the document-parsing SVG test isn't about streaming. Moved to svg/scripted/ in web-platform-tests/wpt@527d0ba.
• wpt coverage: streaming into an element in an XML document (web-platform-tests/wpt@2c7f8ce), streamHTML() replacing template contents (web-platform-tests/wpt@80373e6), and abort() while a write that aborts the parser is in flight, which reaches abortAlgorithm's aborted check (web-platform-tests/wpt@879e633). I didn't add a test for the realm of the rejection errors, since it seems low priority.

Nits

• source:127363-127374, 127528-127546: the ShadowRoot streamPrependHTML() domintros didn't cover an empty shadow root. Fixed in 3eee9c6.
• source:127631: the streaming section had no warning that the Unsafe methods don't sanitize by default. Fixed in 3a768ab.
• source:127285, 128169: the notes overstated what's still parsed. After displacement the rest of the chunk isn't parsed. After an abort from a script in the chunk, parsing only continues up to any parser-blocking script. Fixed in 8ea13dd.
• source:147392: the backup element queue note said it avoids chunk-boundary-dependent timing; it's the reverse. Fixed in 21a87b2.
• source:150319: the note said a streamed script only blocks with a connected target, but a disconnected target's open element can be moved into a document. Fixed in a07bada.
• source:69553 and 153654-153678: the noscript content model and the XML fragment parsing algorithm still linked to the renamed context dfn. Fixed in de4a74b and 6f7656e.
• I didn't change the "tasks that invoke the tokenizer" wording in execute the pending parsing-blocking scripts. It predates this PR and is harmless for fragment parsers.
• wpt: added streamReplaceWithHTML() to the sanitizer img test (web-platform-tests/wpt@4071ec8), removed a duplicate no-policy test (web-platform-tests/wpt@c69db15), replaced named access (web-platform-tests/wpt@37ad36b), dropped an assertion that couldn't fail (web-platform-tests/wpt@843232f), and dropped a streaming help link from a <template for> test (web-platform-tests/wpt@89a69f6). I also replaced a 20 ms wait for a dropped <img> with a control image (web-platform-tests/wpt@7e6b213).