KSES: Parse declarations in safecss_filter_attr() - #14784

The corpus docblock now names @rmenke/css-tokenizer-tests, the package the test cases derive from.

CSS preprocessing replaces NUL with U+FFFD before tokenization, so an unquoted url() containing NUL is a url-token, not a bad-url-token. The source byte stays in the token for offsets; the value decodes it.

Ported from WordPress/php-toolkit bb1c7069 with its CSSPreprocessingTest cases.

Ident sequences and unquoted url values advance over runs of bytes that need no special handling with one strspn() call instead of one PHP loop iteration per byte. Tokenizing 2,000 copies of a three-declaration style string (url, dimension, custom property) drops from 54 ms to 14 ms.

Ported from WordPress/php-toolkit 8e411e7c.

decode_range() and its escape helpers take the CSS source as an argument, so code that holds the source and a token's offsets can decode the token after the cursor has moved on.

get_token() returns the current token as an object that keeps its values after the processor moves on: type, flag, value, unit, start, length, and whether it ended at its closing delimiter. The value is decoded from the source on the first call to get_value(). The processor stays a cursor; get_token_value() reads the same object, so the value decoding is in one place.

The token processor reads CSS; the new class writes it. serialize_string() and serialize_ident() move unchanged, with their tests.

WP_CSS_Token_Serializer::serialize() writes tokens read by the processor as CSS that reads as the same tokens. Comments are dropped, and each whitespace run, including one broken by comments, becomes one space. An empty comment, /**/, separates each token pair the §9 table marks when no whitespace is between them. Tokens are escaped again from their decoded values: a `\` delim keeps its newline, a hash is escaped by its type flag, and a unit that reads as an exponent has its `e` escaped. Strings and urls cut off by the end of the input are written closed, and blocks still open at the end are closed in reverse order.

Bad-string and bad-url tokens have no value; they are written as a quote and a newline, and as `url(()`.

serialize_string() still normalizes newlines in its argument. Token values are quoted without that step, so an escaped CR or FF in a string keeps its code point.

CSS Syntax 3 §9 allows a tokenizer to preserve comments when they have no effect on parsing. The processor emits them so callers can find comments by byte offset.

serialize_string() replaced CR, CRLF and FF in its argument with LF, as if the argument were CSS source. Its output is escaped and never preprocessed, so the replacement changed the value: a string token written from "a\D b" read back with a newline. CR and FF are now written as \D and \C, so the string reads back as the same value; NULL still becomes U+FFFD. A caller that wants normalized line endings normalizes before calling.

serialize() writes string tokens with serialize_string() again, and the private quote_string() is removed.

The CSS Syntax 3 §9 table compares token types in pairs. It misses two sequences that the tokenizer reads as a CDO or CDC token: `<`, `!` and a token that starts with `--` spell `<!--`, and an ident `--` followed by `>` spells `-->`. So `<!/**/--x` was written as `<!--x` (CDO, ident `x`) and `--/**/>` as `-->`. An empty comment is now written before the last token of each, so the output reads as the same tokens.

A token list can hold tokens that have no valid CSS form, so the public way to write CSS will be at the declaration level, where the style attribute processor exposes only valid declarations. WP_CSS_Token_Serializer is marked @access private; its methods stay public so other core classes can call them.

serialize_ident() and serialize_string() had no caller outside the serializer. They become escape_ident(), public for property names, and the private escape_string(); url values use the private escape_url(), which now returns the whole url token. CSSOM defines "serialize an identifier" and "serialize a string" with a different escape set, so the new names avoid suggesting those algorithms.

The string escaping tests now run through serialize(). The class documentation states what the output guarantees: it parses to the same CSS as the tokens. It is not always valid CSS: bad-string and bad-url tokens are written as the same bad tokens.

escape_ident() repeated escape_name()'s loop with three changes for the first code point. It now handles a lone `-` and a leading digit, or `-` and a digit, and escapes the rest with escape_name(). Output is unchanged: 80,565 generated values of up to five code points escape the same as before.

Hash flag round-trips get a data provider, with the expected flag of each input checked on both sides. serialize() is tested with a generator, since it accepts any iterable, and with a DEL delim.

String tokens wrote U+0001 to U+001F and U+007F raw, except LF, CR and FF. They now escape every code point in that range, as CSSOM's "serialize a string" does, so a raw tab is written `\9 `. Identifiers, hash names and urls already escaped these code points.

CSSOM's "escape a character as code point" writes the hex digits in lowercase, so `<` is written `\3c ` instead of `\3C `. CSS reads either case, so the output parses as before.

CSS Syntax limits the non-ASCII code points an identifier may contain to those HTML allows in custom element names. The tokenizer, like Chrome, still reads every code point above U+007F as an identifier code point, and escape_name() wrote them all raw: `a\a0 b` was written with a raw no-break space, which a tokenizer that follows the spec reads as an ident, a delim and an ident. escape_name() now hex-escapes the code points outside the set, so identifiers, hash names and dimension units read the same under both rules.

The hook now matches the form of wp_kses_force_legacy_parser: one bool argument, default false. The CSS string is no longer passed to the filter.

safe_style_css controls the property list, not values, so it does not
replace safecss_filter_attr_allow_css. No filter does.

filter is in the default property list, but no filter function was
allowed, so only url() values passed. The legacy function has the same
gap (#65871); hue-rotate() passes there only because its rotate-function
pattern matches inside the name.

Plugins attached safecss_filter_attr_allow_css callbacks to allow these
functions; with the hook deprecated, the policy has to allow them. All ten
are Filter Effects 1, take numbers, lengths, angles and colors, and ship
in Blink, Gecko and WebKit.

The legacy function accepts calc(100% - abs(-10px)) and
calc(sin(45deg) * 1turn) because its regex removes calc() with everything
inside it. The parser checks every nested function, so without these
names it rejected values the legacy function kept.

All seventeen are CSS Values 4, take numbers, dimensions, percentages and
nested calculations, and ship in Blink, Gecko and WebKit.

cubic-bezier(), steps() and linear() are the values of the timing
function properties. None of those properties is in the default list, but
a plugin that adds them should not need the deprecated allow_css hook to
use their values. All three take numbers, percentages and keywords and
ship in Blink, Gecko and WebKit.

CSS Values 4 defines src() so that var() can supply a URL, which url()
cannot express. The filter cannot check a URL that a variable supplies,
and it already required one string argument, so the only form it allowed
was a second spelling of url("..."). No browser ships src().

A function is added when its result depends only on its arguments and all
three browser engines ship it unprefixed. Spec maturity is not a test: the
allowlist already mixes Candidate Recommendation and Working Draft
functions, and browsers ship from Working Drafts.

Also list the legacy differences the new functions and the hook
deprecation introduce.

The filter's output contains no `&`. The serializer escapes it in
strings, URLs, identifiers and hashes, and no standard property value
uses a bare `&` delimiter, so a declaration with one is rejected.

`<`, `>`, `=`, `<!--` and `-->` are kept. Encoding the output for HTML
is the caller's task, and both core encoders encode `<` and `>`. The
legacy function also kept `<`, `>`, `<!--` and `-->`; it rejected `=`.

The filter wrote each value with its own serializer, which turned every
comment run into a space. That added a whitespace token: in Chrome,
`calc(1px/**/+/**/2px)` is invalid, because `+` needs whitespace on both
sides, and the filter made it `calc(1px + 2px)`, which is valid. CSS
Syntax Level 3 §9 drops comments and writes an empty comment only between
two tokens that would otherwise read as other tokens.

Join the WP_HTML_Style_Attribute_Processor::serialize_declaration() text
of the accepted declarations instead, and delete
_safecss_filter_attr_serialize_value() and
_safecss_filter_attr_serialize_hash_value(). The checks read the
WP_CSS_Token objects the processor now returns.

Output changes: declarations are written `name: value;`, where the
filter wrote `name:value;`; an empty comment stays between marked pairs;
url tokens are written unquoted, with escapes; a hash keeps its type
flag, so `#\31 a` stays `#\31 a` where it was written `#1a`, a hash of
the other type.

When the input ended inside a comment, string or url token, the filter
dropped the last accepted declaration. CSS Syntax Level 3 returns a string
or url token at the end of the input with a parse error (§4.3.5, §4.3.6)
and ends a comment there, so a browser keeps the declaration: Chrome keeps
every case. The legacy function kept `color: red; /*` and
`font-family: "abc`, which the rule dropped.

Check the declaration like any other and let the serializer write the
token closed. Delete _safecss_filter_attr_ends_inside_token(), which
tokenized the input a second time to find the case.

The filter rejected a declaration whose value had a block (a function,
`(`, `[` or `{`) still open at the end of the input, on the grounds that
the legacy function rejected an unbalanced parenthesis. CSS Syntax Level 3
closes a simple block or function at the end of the input, so a browser
applies `width: calc(3em + 10px` as `width: calc(3em + 10px)`. The legacy
function kept an unclosed `[` or `{` as it was.

Check the declaration like any other and let the serializer write the
block closed. Delete _safecss_filter_attr_value_has_open_block(). A
`url(` function with a string argument may now end at the end of the
input, as `calc(` may, so `url("a.png"` is kept as `url(a.png` already was.

WP_Theme_JSON::is_safe_css_declaration() keeps the legacy CSS filter until
WP_Theme_JSON builds its declarations with the CSS API. A closure on
wp_kses_force_legacy_css_parser is added for the call and removed after it,
so other callers keep the parser-based filter.

The docblock of _safecss_filter_attr_declarations() still listed the `<`,
`>` and `=` delimiters and the `<!--` and `-->` tokens, which the filter
keeps. It rejects only a bare `&` delimiter.

A `safe_style_css` callback that returns a value that is not an array, such
as `null` from a callback with no `return`, made safecss_filter_attr()
return its input unfiltered. Only an empty array disables filtering now;
any other value that is not an array is replaced with the default list, on
both the parser-based and the legacy path.

The default list moves into a variable so it can be used after the filter
runs. The list itself is unchanged.

The parser-based filter lowercased every entry of the allowed list, so an
entry that was an array threw a TypeError and a null entry raised a
deprecation notice. Entries that are not strings are now ignored, so a list
with no string entry allows no property. The legacy filter compares entries
strictly and gives the same results.

The serializer writes a delimiter as its code point, so a control
character delimiter appeared raw in the output: `color: red` in a
style attribute became `color:red` followed by U+0001. No property value
uses a control character as a delimiter, and CSSOM escapes control
characters when it serializes CSS. A declaration with a delimiter in
U+0001 to U+001F or U+007F is now rejected, at any depth, as for `&`.
Whitespace control characters are whitespace tokens, and U+0000 is read
as U+FFFD, so neither is a delimiter.

A backslash followed by a newline outside a string is a `\` delimiter.
CSS Syntax requires the serializer to write it as `\` and a newline,
since a backslash before any other code point starts an escape, so the
output contained a raw newline: `color: red \` and a line feed became the
same text plus `;`. No standard property accepts a `\` delimiter, and the
legacy filter rejected every backslash. A declaration with a `\` delimiter
at any depth is now rejected. Escapes are unchanged.

The filter joined the serialize_declaration() text of the accepted
declarations with nothing between them. CSSOM joins the declarations of
a declaration block with a space, so `color: red; width: 1px` is now
written `color: red; width: 1px;` instead of `color: red;width: 1px;`.

…processor-foundations

…-style-attr-processor

…al-style-attribute

…css-filter-attr

# Conflicts:
#	src/wp-includes/kses.php
#	tests/phpunit/tests/kses.php