Parsoid/Internals/Tracking edited nodes

Parsoid has several data attributes that capture semantic and syntactic information about (a) the wikitext source (b) the document represented by the wikitext source (c) parsoid-specific internal details that are not relevant to other clients. The attributes include data-mw, data-parsoid, and other Rich Attributes which are present in tokens, DOM nodes, and in the serialized HTML.

Parsoid's canonical representation of a title/revision is the HTML string / DOM document which are equivalent given HTML/XHTML serialization and HTML5 parsing spec algorithms. So, given a MediaWiki DOM Spec-compliant HTML5 string, a MediaWiki DOM Spec-compliant DOM document can be constructed using a HTML5 parser. And, given a MediaWiki DOM Spec-compliant DOM document, a MediaWiki DOM Spec-compliant HTML5 string can be obtained with a XHTML/HTML serialization of the DOM.

Implementation options for read/write access of data-* attributes

v2 of Parsoid's Mediawiki DOM spec is not the most efficient representation possible. See previous discussions of structured-value attributes and rich attributes. If a client wants to operate on the full structured value of an attribute, it either has to operate on the string representation of the attribute which is very error-prone, or it has to first convert the string representation to a richer internal representation of programming-language-specific objects representing the various data-* attributes. Overall, there are at least three modes of working with a MediaWiki DOM Spec DOM’s attributes.

• String manipulation: work with data-* HTML5 attribute strings
• Load-and-store on every read/write: convert the string to an object, use it, and convert it back to a string if modified.
• Lazy-loading: on an as-needed basis per-node, load a specific data-* attribute as an in-memory class-backed object.
• Eager-loading: load-and-prepare the DOM where all data-* attributes of all nodes in the DOM are converted to in-memory class-backed objects.

The choice usually depends on the client and the specific needs. String manipulation is an exceptional scenario, although still often used in test cases and in legacy parts of mediawiki-core. Load-and-store on every read/write works but is very inefficient. (This was the original implementation in Parsoid’s early days before we implemented eager loading.) Any analysis that needs to inspect all nodes on the DOM, and also all attributes of those nodes benefits from eager-loading. Pretty much all of Parsoid’s internal DOM processing falls in this category. And, so thus far, eager-loading was the solution of choice. However, OutputTransformPipeline passes only inspect a small fraction of nodes and so, eager-loading is almost always very inefficient. Lazy-loading is a better fit for this scenario.

For now, let us focus on Parsoid itself. Parsoid has several transformation modes: wt->html, html->wt, html->html (without going through wt). In all these modes, it frequently access the data-* attributes, and if one doesn't exist, creates a fresh object for the attribute. However, this creation of a fresh object can cause problems in some of the transformations if we aren't careful. Let us look at this a bit more closely.

wikitext to html

Parsoid transforms wt->html and in this process, it creates tokens with data-* objects, builds a DOM out of this, runs many transformations on this DOM, and then serializes this DOM to HTML5. It is clear that in this transformation process, it is most efficient to always work with in-memory objects till it is time to serialize. In this wt->html mode, it is almost always the right thing to create a fresh object if one doesn't exist, since the most common pattern is to fetch and then immediately mutate data-parsoid for a node. However, during serialization, one question to answer is what happens to "empty" objects, which are almost always nodes which have never had ::getDataParsoid() called on them (since we almost always immediately mutate a data-parsoid we obtain). Do we emit them as {} or do we skip them? We can answer this better when we look at the other transformation modes.

It is worth briefly noting that our codebase does not currently have a way to check data-parsoid without creating it by default. In current PHP, we could use a method like ::getDataParsoidCheck($node)?->some_property to avoid creating empty data-parsoid nodes unnecessarily, but for largely historical reasons this method doesn’t exist. If it were available, it would likely largely be used in non-wikitext-to-html parts of the codebase; as noted above the vast majority of wikitext-to-html code is going to mutate a data-parsoid object immediately after fetching it.

html to wikitext

One of Parsoid's crucial selling points is its ability to convert edited Parsoid HTML to wikitext with only the occasional "dirty diff". In addition, wiki editors often have the expectation that newly created content, when represented as wikitext, is emitted in some sort of "canonical wikitext" form. The "no-dirty-diff" and "emit-canonical-wikitext" requirements don't play well with each other. Consider two wikilinks in the edited HTML. One existed in the original wikitext, and the second one was newly inserted in an edit session. The original wikilink should be emitted as is without any canonicalization whereas the new wikilink should be emitted in canonical form. This requires Parsoid to be able to distinguish between original and newly-added wikilinks (as represented in the DOM). Here are the possible options:

1. Editing clients are responsible for flagging all newly inserted content.
2. Parsoid relies on DOM diffs to compare the original and edited DOM and flag all newly inserted content.
3. Parsoid flags all nodes in the original HTML so that all newly created nodes can be detected.

Option 1 was not pursued because requiring all editors to flag new content was considered onerous and error-prone. So, that leaves us with options 2 & 3. Note that option 3 is a mirror image of option 1, but can (in theory) be done on the entire content before handing it off to the editor, and enforcing that the editor not add certain attributes is easier (and can be mechanically checked) than trying to enforce that all newly-created nodes contain a certain attribute.

Parsoid currently relies on both options 2 and 3. In selser (selective serialization) mode, Parsoid diffs original and edited DOM to flag nodes in the edited DOM as modified, inserted, or deleted when compared to the original. As you can imagine, this is best-effort and in complex edits, nodes can be marked as new or modified when they are merely part of the original content which hasn’t been properly associated. But, it is good enough to reduce the vast majority of dirty diffs. However the diff process also compares data-parsoid blobs and because, as noted above, there is no way to look at data-parsoid without creating a new object, could create and initialize new data-parsoid attributes that weren’t present in the original document.

DOM comparison is difficult or impossible in testing and other non-selser cases, and as mentioned above it can sometimes fail to associate moved or rearranged content, so we still need another mechanism to reliably detect newly created nodes.

Given all that, right now, Parsoid employs option 3. The way that Parsoid flags nodes in the original HTML is by relying on the presence / absence of data-parsoid attributes. During serialization of a document after a wikitext-to-html conversion, Parsoid emits a data-parsoid blob for "all" nodes even if it is empty. The caveat is that for transcluded content—output of templates, parser functions, extensions—Parsoid only guarantees that the roots of the DOM trees representing transcluded content have a data-parsoid attribute, and for extension-generated content, delegates handling of content within the transcluded DOM tree to the extension itself. Some extensions (eg, Cite) may therefore use data-parsoid marking for content as Parsoid itself does, but extensions are not required to.

Because Parsoid's helper methods that fetch data-parsoid attributes auto-initialize a new object if one doesn't exist, in html-to-wikitext transformations, we can’t actually detect whether the data-parsoid attribute was originally present on the node or not. As a workaround, the getDataParsoid helper method sets a flag on newly created data-parsoid objects in certain circumstances, currently controlled by a flag named markNew.  Historically, this avoided the need to deal with null data-parsoid objects in certain places in the code base, although recent versions of PHP make this less painful than it would have been at the time.

Another interesting case is when the editor (VisualEditor for purposes of discussion) does a copy or cut-and-paste operation. In theory we want to preserve data-parsoid from a copy-and-paste, so that the original wikitext styling of a moved section is preserved—ideally selser should recognize this as well and preserve the source string literally, but we don't currently detect moved sections well, so only the style information preserved in data-parsoid is preserved.

When the original content is duplicated instead of moved, things get more complicated, especially when pagebundle format is used (T256687). We will either end up with multiple nodes having the same id attribute, which violates HTML expectations for the id attribute, or else one of the copies will lose its data-parsoid, causing a dirty diff. (Similarly, if the content is copied out of one document into another, the pagebundle format will not preserve the data-parsoid for the inserted content, and the ids may not be unique in the destination document.) We have not yet resolved these issues.

html to html (directly without going through wikitext)

There are a lot of places where Parsoid converts the html to a DOM, analyzes / mutates the DOM, and writes the DOM back to html. Sometimes, these html-to-html transforms happen during the wikitext-to-html transform (typically during processing of attribute-embedded docs, although this is deprecated and only used by the Cite extension), sometimes they happen during html-to-wikitext transforms (convertOffsets, processing of attribute-embedded docs (again, deprecated and only Cite), other pre-processing like domdiffs, normalization), sometimes they are standalone passes unrelated to html-to-wikitext and html-to-wikitext (ex: version downgrades, convertOffsets).

In these codebases, the habit of automatically creating data-parsoid attributes that didn’t exist in the source is a hazard which makes it easy to create “dirty” passes which modify the input unexpectedly.