What changes were proposed in this pull request?

Add a built in normalize(str[, form]) scalar function that performs Unicode normalization using the JVM's java.text.Normalizer.

API surface added:

• SQL: normalize(str) and normalize(str, form)
• Scala DataFrame: functions.normalize(col) and functions.normalize(col, form)
• PySpark, classic and Spark Connect: pyspark.sql.functions.normalize(str, form=None)

Key design choices:

• The default form is NFC. Supported forms are NFC, NFD, NFKC, and NFKD, case insensitive, matching the ANSI SQL standard and PostgreSQL, Trino, and BigQuery.
• The form is a foldable string argument, consistent with how other Spark functions take a mode string (for example date_trunc). This keeps the change small and idiomatic and avoids adding new grammar. An invalid form raises a clear INVALID_PARAMETER_VALUE.NORMALIZE_FORM error.
• Implemented as a RuntimeReplaceable expression backed by a StaticInvoke into ExpressionImplUtils, so there is no hand written codegen, following the recent hmac function.

Why are the changes needed?

Unicode normalization is a common need in text processing and data cleaning, for example to compare strings that look identical but differ in code point composition. Spark has no built in way to do this, so users fall back to a UDF, which cannot be optimized by Catalyst. This function is part of the SQL standard and is already available in PostgreSQL, Trino, and Google BigQuery, so it also improves parity with the engines that Spark users migrate from.

Does this PR introduce any user-facing change?

Yes. It adds a new built in SQL function normalize and the corresponding Scala and PySpark DataFrame API entries. No existing behavior changes.

Example:

spark-sql> SELECT normalize('fi', 'NFKC');
fi

How was this patch tested?

Added unit tests in ExpressionImplUtilsSuite and StringExpressionsSuite covering all four forms (NFC, NFD, NFKC, NFKD), a case insensitive form name, null propagation, empty input, a supplementary (surrogate pair) character, and the invalid form error. Added a DataFrame API test in StringFunctionsSuite, a PySpark test in test_functions.py, and SQL golden tests in string-functions.sql with regenerated results. Also regenerated sql-expression-schema.md.

Was this patch authored or co-authored using generative AI tooling? No