tools: build the Unicode tables from the character database - #7188

src/unicase.h and mrbgems/mruby-regexp/src/re_cased.h are generated tables. Both generators gather them by asking the host CRuby to case every codepoint:

(0x80..0x10FFFF).each do |cp|
  next if cp.between?(0xD800, 0xDFFF)
  c = begin; cp.chr("UTF-8"); rescue RangeError; next; end
  l = c.downcase
  u = c.upcase
  t = c.capitalize          # one character capitalized is its title case
  f = c.downcase(:fold)

and stamp the header with whichever Unicode that Ruby happens to carry:

** Generated by tools/gen_unicase.rb from Unicode 17.0.0
** as carried by ruby 4.0.6. Do not edit by hand.

So which Unicode mruby answers by is decided by the machine a table was last regenerated on, both headers are reproducible only on a machine carrying the same Ruby, and a regeneration run on two hosts can leave core and the regexp gem holding different Unicodes.

This PR reads the Unicode Character Database instead.

One reading for both generators

tools/unicode/case_data.rb reads UnicodeData.txt, SpecialCasing.txt and CaseFolding.txt, and hands out lower, upper, title, fold and swap as {source => the codepoints it answers with}, holding a source only where the answer differs from the source itself. Both generators take theirs from it, so how a field is read is stated once and what each of them makes of the mappings stays where it is.

They read the files out of a directory named on the command line, defaulting to the one beside case_data.rb:

$ ruby tools/gen_unicase.rb src [UCDDIR]
$ ruby mrbgems/mruby-regexp/tools/gen_cased.rb mrbgems/mruby-regexp/src [UCDDIR]

That directory is named after the version, and which version is asked for is VERSION in case_data.rb rather than whatever the directory happens to hold.

Which bytes that version is

CHECKSUMS, beside VERSION, records the digest of each of the three files, and a file whose digest is not the recorded one is refused rather than generated from:

CHECKSUMS = {
  'UnicodeData.txt'   => '2e1efc1dcb59c575eedf5ccae60f95229f706ee6d031835247d843c11d96470c',
  'SpecialCasing.txt' => 'efc25faf19de21b92c1194c111c932e03d2a5eaf18194e33f1156e96de4c9588',
  'CaseFolding.txt'   => 'ff8d8fefbf123574205085d6714c36149eb946d717a0c585c27f0f4ef58c4183',
}.freeze

Reading the version off a first line would answer for two of the three. Their first lines:

UnicodeData.txt     0000;<control>;Cc;0;BN;;;;;N;NULL;;;;
SpecialCasing.txt   # SpecialCasing-17.0.0.txt
CaseFolding.txt     # CaseFolding-17.0.0.txt

UnicodeData.txt is data from its first byte and names no version, and it is the file most of a table is read out of: the simple lower, upper and title mappings, the general category, and the decomposition the swap case rule rides on. A directory holding it from another release, with the other two from this one, would generate that release's mappings under a header naming this one.

What a digest cannot say is that the bump itself fetched what Unicode published. It pins the bytes that were fetched, and since a published release never changes, those are the bytes every regeneration after the bump has to read. rake unicode:download prints the digest of each file it got, as sha256sum spells it, because a bump reaches a release before anything can know its digests:

$ rake unicode:download
downloading https://www.unicode.org/Public/17.0.0/ucd/CaseFolding.txt
  ff8d8fefbf123574205085d6714c36149eb946d717a0c585c27f0f4ef58c4183  CaseFolding.txt

What the mappings are read from

The simple mappings are fields 12 to 14 of UnicodeData.txt, the full ones the unconditional entries of SpecialCasing.txt, which replace the simple mapping rather than sit beside it, and folding the C and F entries of CaseFolding.txt.

A SpecialCasing.txt entry carrying a condition is one only a caller that knows the language or the surrounding text can apply, which is neither what these tables are asked for nor what CRuby answers off them, so it is skipped here as it is there, and "ΟΔΟΣ".downcase stays "οδοσ".

Two answers are in none of the three files, and both are stated where the tables are built:

• The Georgian Mtavruli capitals, U+1C90 to U+1CBF, title to their lower case rather than to themselves. Nothing in the database says so; CRuby spells the same rule as a codepoint range in enc/unicode.c.
• A title case character swaps to neither of its cases: each piece of what it decomposes to swaps on its own, so U+01C5, decomposing to D and ž, swaps to "dŽ".

The database is not in the repository

.gitignore covers /tools/unicode/data. The three files are read when a table is regenerated and at no other time, they weigh 2.3 MB against the 53 KB they produce, and the license they carry would be a fourth exception in LEGAL for something nothing compiles. CRuby does not track them either: .gitignore there covers /enc/unicode/data.

Regenerating

tasks/unicode.rake is what a Unicode version bump comes to: change VERSION, fetch the database it names, record the digests the fetch prints, regenerate every table at once. Each file of the database is a task that fetches it, so asking for a table before anything was downloaded fetches it first.

That the committed tables are what the generators write is a claim nothing states, so a header edited by hand, or left behind by a bump that ran one generator, reads as current. unicode:verify regenerates into a temporary directory and compares what comes out with the headers as they are committed, naming every one that is not what the database generates.

A build reaches none of this. The tables stay committed and the tasks are run by hand.

What changes in the tables

Nothing that is compiled. Both regenerated headers are identical to the committed ones apart from the lines naming where they came from:

-** Generated by tools/gen_unicase.rb from Unicode 17.0.0
-** as carried by ruby 4.0.6. Do not edit by hand.
+** Generated by tools/gen_unicase.rb from the Unicode 17.0.0 character
+** database. Do not edit by hand.

re_cased.h also names its generator where it stands. The line read tools/gen_cased.rb, which is where core's generator lives and not where this one does, so anyone following it to regenerate the table reached the wrong file.

Testing

Every mapping case_data.rb reads was compared against what the host CRuby answers for the same codepoint, over the 1,111,936 codepoints of U+0080 to U+10FFFF the old generators walked. All five agree everywhere, swap included, which is the one the table stores as a difference from a rule rather than outright.

rake unicode:verify passes on the committed tables, and reports src/unicase.h stale when a line of it is changed by hand. rake unicode:download over a deleted CaseFolding.txt fetches it again and prints the digest CHECKSUMS already holds for it.

A UnicodeData.txt with one line appended, in a directory whose other two files are untouched, is refused:

$ ruby tools/gen_unicase.rb src /tmp/mixed
/tmp/mixed/UnicodeData.txt is not the file Unicode 17.0.0 was generated from:
  recorded 2e1efc1dcb59c575eedf5ccae60f95229f706ee6d031835247d843c11d96470c
  read     a63cab158dd62133b84aafe012f2372d8de9a28ca6ff0ed70691126007dcbfc4

A check that read the version off the first lines instead accepted that same directory, exit 0, and wrote a table of 185 lower case runs against 184 and 25 folding runs against 24 under a header saying Unicode 17.0.0.

rake -m test over ci/gcc-clang, all four builds green, 0 KO, 0 crash, no new warnings:

No test accompanies the change. Nothing compiled is different, and what the generators write is pinned by unicode:verify against the database rather than by a test against a build.

Environment

Summary by CodeRabbit

• Build Improvements

  • Added automated tasks to download, generate, and verify Unicode data tables.
  • Added validation to detect outdated generated Unicode tables.

• Maintenance

  • Updated Unicode case-mapping data to Unicode 17.0.0.
  • Improved consistency and reproducibility of Unicode case conversion and folding behavior.
  • Added clearer guidance when required Unicode data is unavailable.
  • Improved support for maintaining and regenerating Unicode-related data.