Parsoid/MediaWiki DOM spec/Metadata tracking

Tracked in Phabricator
Task T419703

This is an experimental proposal for a future revision of the MediaWiki DOM spec.

For Parsoid selective update, we need to track the metadata created by each fragment of the page so that when that fragment is updated/replaced/removed we can recompute new metadata for the page as a whole.

Proposal

We propose to record metadata at top-level template and parser function invocations, in the form of a data-mw.metadata attribute which is a serialized ParserOutput (excluding HTML from the content holder), likely exposed via a ContentMetadataCollector::clone() method that returns an opaque JsonCodecable and an appropriate Hint to discourage abuse of this method for prohibited ContentMetadataCollector introspection. The clone() method will simply return the underlying ParserOutput to reuse ParserOutput::toJsonArray() for serialization (instead of inventing a new serialization format). The current JSON output for the ParserOutput is a little heavyweight, emitting property names even for properties with default or null values; we will tweak serialization to omit these continuing the work started in ParserOutput: Add default values for JSON deserialization (1193582). This allows us to leverage the robust back- and forward-compatibility testing done for the default ParserOutput serialization, instead of having to recreate this.

We will also emit the "top level" metadata for each <section>, which is the metadata for that section excluding all separately-recorded transclusions. The final metadata for the output is computed by computing the union of the top level metadata with the metadata for all of the top-level templates, which can be recomputed when either a top-level template is selectively updated (reusing top-level metadata), or when a section is selectively updated (reusing template metadata).

For input like:

foo [[Category:Foo]]

{{1x|bar [[Category:Bar]]}}

The output will look like:

<section data-mw-section-id="0" data-mw='{"metadata":{"Categories":{"Foo":""}}'>
    <p>foo </p>
    <link rel="mw:PageProp/Category" href="./Category:Foo"/>

    <p>
        <span about="#mwt1" typeof="mw:Transclusion" data-mw='{"metadata":{"Categories":{"Bar":""},"parts":[{"template":{"target":{"wt":"1x","href":"./Template:1x"},"params":{"1":{"wt":"bar [[Category:Bar]]"}},"i":0}}]}'>
            bar 
        </span>
        <link rel="mw:PageProp/Category" href="./Category:Bar" about="#mwt1"/>
    </p>
</section>

Note the data-mw.metadata on the top-level <section> doesn't include Category:Bar because that comes from a transclusion, and that the data-mw for the span[typeof="mw:Transclusion"] now includes a metadata key.

Alternatives

• Instead of recording the metadata as a global union, we could try to track metadata as reversible operations. This would avoid the need to track the top-level metadata separate from the final output; instead in order to update a template we would first "subtract" the template's old contribution from the final metadata, then "add" the new contribution. This gets complicated: if multiple templates add the same category, for example, we'd need to maintain a count of how many times the category had been "added" to the final output in order to be able to "subtract" an individual template's contribution properly. Operations which are order-dependent contribute further complications: in order to reverse an early template we need to be able to recompute its effect on later templates as well. The advantage would be theoretical O(1) update of metadata: we don't have to look at contributions from other templates when we update the global metadata. In practice, however, we have numerous O(N) steps already, including simply parsing the (entire) original HTML in order to extract the correct region to update, and so at present the added complexity of this approach isn't worth the trade-off.
• Update does currently rely on the union operation being order independent (T300979 Ensure ParserOutput can always be combined asynchronously/out-of-order) because of the way the top-level metadata is recorded. If we wished to support order-dependent metadata operations, we need to dump the top-level metadata state at the start of every template. For input like A {{B}} C we need to record the contributions of A, {{B}}, and C separately and replay them in order as A {{B'}} C when B is updated to B'. Sections are treated as separators like templates; for example A\n==B==\nC {{D}} E has to record separate metadata for A, ==B==\nC, {{D}}, and E. Template metadata can be recorded in the same way, but changes would be made to where and how the "top level" metadata is recorded. Although it would be possible to come up with a suitable representation the added complexity and increased bloat seems to discourage this approach for now.
• We could select different "selective update" boundaries. The current boundaries (sections, templates, and asynchronous parser functions) correspond to the first targets for the Parsoid selective update functionality. We could omit or add additional boundaries without fundamentally altering the approach described here. For example, metadata could be recorded on a paragraph level instead of section level to allow extremely fine-grained output. Most paragraphs would be expected to contribute nothing to the metadata so could have data-mw.metadata omitted, so the HTML overhead would still be low. However, we would need to traverse all the paragraphs to compute the global union after every update, instead of just all the sections and transclusions. At the current time section editing seems like an appropriate level of granularity for tracking changes.
• The metadata could be recorded in data-parsoid instead of data-mw. This would avoid any overhead in shipping the metadata to read-views clients, at the cost of limiting the information available to downstream analysis consumers. #Contents of transcluded blocks in the MW DOM spec describes potentially stripping <link> and <meta> tags in transcluded HTML and using the metadata described here as the primary source for that information: that would have to be reconsidered or adjusted if the information were made Parsoid-internal.