Build/Test Tools: Compare style attributes by their declarations - #14783

The corpus docblock now names @rmenke/css-tokenizer-tests, the package the test cases derive from.

CSS preprocessing replaces NUL with U+FFFD before tokenization, so an unquoted url() containing NUL is a url-token, not a bad-url-token. The source byte stays in the token for offsets; the value decodes it.

Ported from WordPress/php-toolkit bb1c7069 with its CSSPreprocessingTest cases.

Ident sequences and unquoted url values advance over runs of bytes that need no special handling with one strspn() call instead of one PHP loop iteration per byte. Tokenizing 2,000 copies of a three-declaration style string (url, dimension, custom property) drops from 54 ms to 14 ms.

Ported from WordPress/php-toolkit 8e411e7c.

decode_range() and its escape helpers take the CSS source as an argument, so code that holds the source and a token's offsets can decode the token after the cursor has moved on.

get_token() returns the current token as an object that keeps its values after the processor moves on: type, flag, value, unit, start, length, and whether it ended at its closing delimiter. The value is decoded from the source on the first call to get_value(). The processor stays a cursor; get_token_value() reads the same object, so the value decoding is in one place.

The token processor reads CSS; the new class writes it. serialize_string() and serialize_ident() move unchanged, with their tests.

WP_CSS_Token_Serializer::serialize() writes tokens read by the processor as CSS that reads as the same tokens. Comments are dropped, and each whitespace run, including one broken by comments, becomes one space. An empty comment, /**/, separates each token pair the §9 table marks when no whitespace is between them. Tokens are escaped again from their decoded values: a `\` delim keeps its newline, a hash is escaped by its type flag, and a unit that reads as an exponent has its `e` escaped. Strings and urls cut off by the end of the input are written closed, and blocks still open at the end are closed in reverse order.

Bad-string and bad-url tokens have no value; they are written as a quote and a newline, and as `url(()`.

serialize_string() still normalizes newlines in its argument. Token values are quoted without that step, so an escaped CR or FF in a string keeps its code point.

CSS Syntax 3 §9 allows a tokenizer to preserve comments when they have no effect on parsing. The processor emits them so callers can find comments by byte offset.

serialize_string() replaced CR, CRLF and FF in its argument with LF, as if the argument were CSS source. Its output is escaped and never preprocessed, so the replacement changed the value: a string token written from "a\D b" read back with a newline. CR and FF are now written as \D and \C, so the string reads back as the same value; NULL still becomes U+FFFD. A caller that wants normalized line endings normalizes before calling.

serialize() writes string tokens with serialize_string() again, and the private quote_string() is removed.

The CSS Syntax 3 §9 table compares token types in pairs. It misses two sequences that the tokenizer reads as a CDO or CDC token: `<`, `!` and a token that starts with `--` spell `<!--`, and an ident `--` followed by `>` spells `-->`. So `<!/**/--x` was written as `<!--x` (CDO, ident `x`) and `--/**/>` as `-->`. An empty comment is now written before the last token of each, so the output reads as the same tokens.

A token list can hold tokens that have no valid CSS form, so the public way to write CSS will be at the declaration level, where the style attribute processor exposes only valid declarations. WP_CSS_Token_Serializer is marked @access private; its methods stay public so other core classes can call them.

serialize_ident() and serialize_string() had no caller outside the serializer. They become escape_ident(), public for property names, and the private escape_string(); url values use the private escape_url(), which now returns the whole url token. CSSOM defines "serialize an identifier" and "serialize a string" with a different escape set, so the new names avoid suggesting those algorithms.

The string escaping tests now run through serialize(). The class documentation states what the output guarantees: it parses to the same CSS as the tokens. It is not always valid CSS: bad-string and bad-url tokens are written as the same bad tokens.

escape_ident() repeated escape_name()'s loop with three changes for the first code point. It now handles a lone `-` and a leading digit, or `-` and a digit, and escapes the rest with escape_name(). Output is unchanged: 80,565 generated values of up to five code points escape the same as before.

Hash flag round-trips get a data provider, with the expected flag of each input checked on both sides. serialize() is tested with a generator, since it accepts any iterable, and with a DEL delim.

String tokens wrote U+0001 to U+001F and U+007F raw, except LF, CR and FF. They now escape every code point in that range, as CSSOM's "serialize a string" does, so a raw tab is written `\9 `. Identifiers, hash names and urls already escaped these code points.

CSSOM's "escape a character as code point" writes the hex digits in lowercase, so `<` is written `\3c ` instead of `\3C `. CSS reads either case, so the output parses as before.

CSS Syntax limits the non-ASCII code points an identifier may contain to those HTML allows in custom element names. The tokenizer, like Chrome, still reads every code point above U+007F as an identifier code point, and escape_name() wrote them all raw: `a\a0 b` was written with a raw no-break space, which a tokenizer that follows the spec reads as an ident, a delim and an ident. escape_name() now hex-escapes the code points outside the set, so identifiers, hash names and dimension units read the same under both rules.

set_value() and append_declaration() trimmed CSS whitespace bytes from the
end of the value. When the last token is an escape of a space or tab, or a
backslash delimiter followed by a newline, the trim removed the byte that
completed it, so the `;` written after the value became part of an escape:
append_declaration( '--x', "a\\ " ) wrote `--x: a\;`, whose value is the
ident `a;`. set_value() rejected the same values because its replacement
check saw the declaration absorb the next one.

Trim only whitespace tokens, and keep one newline after a final backslash
delimiter.

The processor kept an array per token with the type, the decoded value
and unit, and the offsets, and get_value_tokens() returned those arrays.
It now stores the WP_CSS_Token snapshots from
WP_CSS_Token_Processor::get_token() and returns them, so values are
decoded when read rather than for every token during parsing, and
consumers read the flag and termination that the arrays did not carry.

Declarations stay private arrays, and the processor uses the token
processor rather than extending it.

get_eof_repair() decided whether a comment or url token at the end of
the style was closed by reading bytes: a comment counted as closed when
the style ended in `*/`, and a url token was re-tokenized with a
sentinel appended. WP_CSS_Token::is_terminated() records the answer, so
use it and delete is_url_token_closed().

The byte check counted `/*/` as a closed comment. Appending after it
still failed, because the parse check rejects a declaration absorbed by
the comment; the new test pins that result, which is the same before
and after this change.

Code that reads a declaration and needs to write it back, such as a
filter that keeps some declarations and drops others, had no public way
to do so: WP_CSS_Token_Serializer is internal, and the processor wrote
text only through mutations.

serialize_declaration() returns the declaration at the cursor as
`name: value;` or `name: value !important;`, as CSSOM writes a
declaration, or null when the cursor is not on a declaration, like the
other getters. The name is escaped as an identifier and the value is
written by WP_CSS_Token_Serializer, so comments are dropped, whitespace
runs become one space, and blocks left open at the end of the style are
closed. The name follows WP_HTML_Processor::serialize_token().

The private formatter it shares with set_value() and
append_declaration() is renamed format_declaration().

Every successful mutation parses the whole style text again, so removing
declarations one at a time takes time that grows with the square of the
style's length. Measured on this branch, removing every declaration of a
style takes 12, 189 and 744 ms for 50, 200 and 400 declarations, while
reading them all and joining serialize_declaration() takes 0.3, 1.2 and
2.4 ms. Say so on remove_declaration() and in the spec, and point to the
alternative.

set_value() and append_declaration() checked that a supplied value parses
as one declaration value, then wrote the caller's bytes: comments,
whitespace runs, raw control characters inside strings and any escape
spelling reached the style. They now write the declaration as
serialize_declaration() does: the name is escaped as an identifier and
the value is written by WP_CSS_Token_Serializer from its tokens, with the
whitespace and comments around it removed. set_value() writes the
decoded name, so `c\6f lor` becomes `color`.

trim_value() and the raw_name field have no other use and are removed.

CSS closes a string or a comment that the end of the input cuts off, and
so does WP_CSS_Token_Serializer::serialize(), but get_eof_repair()
closed only blocks and urls: append_declaration() and set_important()
returned false after `font-family: "a` or `color: red /* x`. The repair
now writes the string's own quote or `*/` before the block closers. A `\`
at the end of the style still makes the append fail, because no appended
text can end the escape it starts.

…ree()

build_visual_html_tree() split a style attribute on every `;` and `:`, so
`url(https://example.com/a.png)` and `url(https://example.com/b.png)` both
became `url(https`, and a `;` or `:` inside a string or function cut the
value short. It also kept whitespace inside values, so `rgb(1, 2, 3)` and
`rgb(1,2,3)` differed.

build_style_attribute_declarations() lists the declarations that
WP_HTML_Style_Attribute_Processor reads, each written from its tokens
with whitespace and comments ignored. The tree writes that list, or the
value as it is when it has text that is not a declaration.

Tests that check a style attribute value compare exact bytes, so a change
to how declarations are written, such as `name: value;` instead of
`name:value;`, changes every expectation even when the declarations are
the same.

assertEqualStyleAttribute() compares the declarations that
build_style_attribute_declarations() lists for each value: the same
property names, priorities and value tokens in the same order, with
whitespace and comments ignored. It fails when either value has text that
is not a declaration, since that text has no canonical form to compare.

…-style-attr-processor

…al-style-attribute