tools: give the generated Unicode headers an include guard - #7192
Conversation
The tables were gathered by asking the host CRuby to case every codepoint, which put whichever Unicode version that Ruby happens to carry into `unicase.h` and `re_cased.h`, left both headers reproducible only on a machine carrying the same one, and let a regeneration run on two hosts leave core and the regexp gem holding different Unicodes. `tools/unicode/case_data.rb` reads `UnicodeData.txt`, `SpecialCasing.txt` and `CaseFolding.txt` and hands out the mappings, and both generators take theirs from it, so how a field is read is stated once. They read them out of a directory named on the command line, defaulting to the one beside `case_data.rb`, which is named after the version. Which version is asked for is `VERSION` there rather than whatever the directory happens to hold, and `CHECKSUMS` beside it says which bytes that version is: a file whose digest is not the recorded one is refused rather than generated from. Reading the version off a first line would answer for two of the three files. `UnicodeData.txt` carries no such line, and it is the file most of a table is read out of, so a directory holding it from another release would generate the simple mappings of that release, and the swap answers riding on its decompositions, under a header naming this one. What a digest cannot say is that the bump itself fetched what Unicode published; it pins the bytes that were fetched, which is what every regeneration after it has to match. The files are not in the repository. They are read when a table is regenerated and at no other time, they weigh 2.3 MB against the 53 KB they produce, and the license they carry would be a fourth exception in LEGAL for something nothing compiles. CRuby does not track them either: `.gitignore` there covers `/enc/unicode/data`. The simple mappings come from the `UnicodeData.txt` fields, the full ones from the unconditional `SpecialCasing.txt` entries, and folding from the C and F entries of `CaseFolding.txt`. A conditional entry is one only a caller that knows the language or the surrounding text can apply, which is why it is skipped here as it is there, and why `"ΟΔΟΣ".downcase` stays `"οδοσ"`. Two answers are in none of the files. The Georgian Mtavruli capitals title to their lower case, which the database nowhere says and which CRuby spells as a codepoint range in `enc/unicode.c`. And a title case character swaps to neither of its cases: each piece of what it decomposes to swaps on its own, so `U+01C5`, decomposing to D and ž, swaps to "dŽ". For the refusals `re_cased.h` holds, the single counterpart a folding of several characters may still leave keeps coming from the lower case mapping, rather than from the S entries sitting beside the F ones. `U+1FD3`, `U+1FE3` and `U+FB05` each have an S entry naming a codepoint they fold with, but core's simple folding reads the folding difference over the lower case mapping and reaches none of the three; a counterpart the engine cannot use is not one to record. Both readings put the same codepoints in the table, so this is about what the pass claims and not about what it writes. `re_cased.h` also names its generator where it stands. The line read `tools/gen_cased.rb`, which is where core's generator lives and not where this one does, so anyone following it to regenerate the table reached the wrong file. ### Verified Both regenerated headers are identical to the committed ones apart from the lines naming where they came from, so nothing compiled changes. Every mapping was compared against what the host CRuby answers for the same codepoint over `U+0080` to `U+10FFFF`, and they agree everywhere. The refusals come out of them as 1,483 pairs and 76 skipped sources over 2,986 cased codepoints, and reading the S entries instead leaves that cased set where it is. A `UnicodeData.txt` with one line added to it, in a directory whose other two files are untouched, is refused with both digests named. A check that read the version off the first lines instead accepted that same directory, and wrote a table of 185 lower case runs against 184 and 25 folding runs against 24 under a header saying Unicode 17.0.0. `MRUBY_CONFIG=ci/gcc-clang rake -m test`, KO 0, Crash 0 and Warning 0.
Fetching the character database and regenerating the tables were steps written down nowhere, and each generator had to be run by hand, so a version bump could update one table and leave the other where it was. `rake unicode:download` fetches the database `VERSION` names into the directory the generators read, and `rake unicode:generate` runs both of them. Asking for a table before anything was downloaded fetches it first, since each file of the database is a task of its own. The download prints the digest of each file it got, as `sha256sum` spells it, so that the bump has them to record in `CHECKSUMS`. It prints rather than checks: a bump fetches a release before anything can know its digests, and a download that checked them would have nothing to check against. ### Verified `rake unicode:generate` with the database in place rewrites `src/unicase.h` and `mrbgems/mruby-regexp/src/re_cased.h` byte for byte. `rake unicode:download` over a deleted `CaseFolding.txt` fetches it again and prints the digest `CHECKSUMS` already holds for it.
That the committed tables are what the generators write is a claim nothing states, so a header edited by hand, or left behind by a bump that ran one generator, reads as current. `rake unicode:verify` regenerates into a temporary directory and compares what comes out with `src/unicase.h` and `mrbgems/mruby-regexp/src/re_cased.h` as they are committed, naming every header that is not what the database generates. ### Verified `rake unicode:verify` passes on the committed tables, and reports `src/unicase.h` stale when a line of it is changed by hand.
`src/unicase.h` and `mrbgems/mruby-regexp/src/re_cased.h` are what the two generators write, and neither carried an include guard. Their neighbours have one: `src/value_array.h` beside the first and `mrbgems/mruby-regexp/include/re_internal.h` beside the second, which is also where the `MRB_` spelling comes from. Each header is included from one file, so no build changes. What the guard buys is the second include point: without it, reading either header twice in a translation unit redefines the objects it holds, the `uni_*` arrays in one and `re_cased_ranges` in the other. The amalgamated build puts every core source and every gem source into a single translation unit, so include points that sit far apart in the tree sit next to each other there. It also keeps the two out of a reading they do not fit. A header with no guard is an X-macro table in this tree, `mruby/ops.h` and `mrbgems/mruby-compiler/include/mrc_ops.h` being the ones that are, and `lib/mruby/amalgam.rb` sorts a gem's headers by that very test. These two expand the same way wherever they are read. The generators write the guards, since they write the headers, and `rake unicode:verify` keeps the two in step.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughThe PR adds a verified Unicode 17.0.0 case-data loader, updates core and regexp table generators to use it, and adds Rake tasks for downloading, generating, and verifying generated headers. ChangesUnicode case-table generation
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk: ⚪ Minimal · up to This localized change adds include guards to generated Unicode headers and reports successful build, amalgamation, and Unicode verification checks; no actionable merge-blocking risk remains. Sequence Diagram(s)sequenceDiagram
participant Developer
participant UnicodeRakeTasks
participant UnicodeCaseData
participant TableGenerators
participant GeneratedHeaders
Developer->>UnicodeRakeTasks: run unicode:generate or unicode:verify
UnicodeRakeTasks->>UnicodeCaseData: load verified Unicode data
UnicodeCaseData-->>TableGenerators: return composed case mappings
TableGenerators->>GeneratedHeaders: generate core and regexp tables
UnicodeRakeTasks->>GeneratedHeaders: compare regenerated output
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Stacked on #7188, whose three commits are the first three here. The last commit is this PR's own change, and the diff to read is
git diff <third commit>..HEAD. Merging #7188 first leaves this one a single commit.src/unicase.handmrbgems/mruby-regexp/src/re_cased.hare whattools/gen_unicase.rbandmrbgems/mruby-regexp/tools/gen_cased.rbwrite, and neither carries an include guard. The header beside each one does:src/value_array.hhasMRB_VALUE_ARRAY_H__, andmrbgems/mruby-regexp/include/re_internal.hhasMRB_RE_INTERNAL_H, which is where the spelling used here comes from.Each of the two is read from a single file,
src/unicase.candmrbgems/mruby-regexp/src/re_utf8.c, so no build changes. What the guard buys is the second include point. Both headers define objects, theuni_*arrays in one andre_cased_rangesin the other, so reading either twice in a translation unit is a redefinition rather than a no-op. Amalgamation puts every core source and every gem source into one translation unit, which is where include points that sit far apart in the tree come to sit next to each other.It also keeps the two out of a classification they do not fit. A header with no guard is an X-macro table in this tree,
mruby/ops.handmrbgems/mruby-compiler/include/mrc_ops.hbeing the ones that are, andlib/mruby/amalgam.rbsorts a gem's headers by exactly that test, inlining a guardless one afresh at every include point instead of deduplicating it. Neither of these two expands differently by site.What the generators write
mrbgems/mruby-regexp/src/re_cased.hgets the same three lines underMRB_RE_CASED_H. Both are regenerated here, sorake unicode:verifystays green.Checked
full-debug,bintest,cxx_abi,byte-stringMRUBY_CONFIG=ci/gcc-clang rake -m test, KO 0 in all fourfull-debugamalgamrake amalgam, thengcc -c -I<dir> mruby.candgcc -c -I<dir> mruby_compiler.crake unicode:verifyreports the Unicode 17.0.0 tables up to date, which is what says the committed headers are what the generators now write.Summary by CodeRabbit
Enhancements
Chores