Skip to content

tools: give the generated Unicode headers an include guard - #7192

Merged
matz merged 4 commits into
mruby:masterfrom
takumin:build/unicode-header-guards
Aug 15, 2026
Merged

tools: give the generated Unicode headers an include guard#7192
matz merged 4 commits into
mruby:masterfrom
takumin:build/unicode-header-guards

Conversation

@takumin

@takumin takumin commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Stacked on #7188, whose three commits are the first three here. The last commit is this PR's own change, and the diff to read is git diff <third commit>..HEAD. Merging #7188 first leaves this one a single commit.

src/unicase.h and mrbgems/mruby-regexp/src/re_cased.h are what tools/gen_unicase.rb and mrbgems/mruby-regexp/tools/gen_cased.rb write, and neither carries an include guard. The header beside each one does: src/value_array.h has MRB_VALUE_ARRAY_H__, and mrbgems/mruby-regexp/include/re_internal.h has MRB_RE_INTERNAL_H, which is where the spelling used here comes from.

Each of the two is read from a single file, src/unicase.c and mrbgems/mruby-regexp/src/re_utf8.c, so no build changes. What the guard buys is the second include point. Both headers define objects, the uni_* arrays in one and re_cased_ranges in the other, so reading either twice in a translation unit is a redefinition rather than a no-op. Amalgamation puts every core source and every gem source into one translation unit, which is where include points that sit far apart in the tree come to sit next to each other.

It also keeps the two out of a classification they do not fit. A header with no guard is an X-macro table in this tree, mruby/ops.h and mrbgems/mruby-compiler/include/mrc_ops.h being the ones that are, and lib/mruby/amalgam.rb sorts a gem's headers by exactly that test, inlining a guardless one afresh at every include point instead of deduplicating it. Neither of these two expands differently by site.

What the generators write

/* src/unicase.h */
** See Copyright Notice in mruby.h
*/

+#ifndef MRB_UNICASE_H
+#define MRB_UNICASE_H
+
/* One run of sources start, start+stride, ... (count entries), each
...
#define UNI_FOLD_MAX 0x0FB17
+
+#endif /* MRB_UNICASE_H */

mrbgems/mruby-regexp/src/re_cased.h gets the same three lines under MRB_RE_CASED_H. Both are regenerated here, so rake unicode:verify stays green.

Checked

build what ran
full-debug, bintest, cxx_abi, byte-string MRUBY_CONFIG=ci/gcc-clang rake -m test, KO 0 in all four
full-debug amalgam rake amalgam, then gcc -c -I<dir> mruby.c and gcc -c -I<dir> mruby_compiler.c

rake unicode:verify reports the Unicode 17.0.0 tables up to date, which is what says the committed headers are what the generators now write.

Summary by CodeRabbit

  • Enhancements

    • Updated Unicode case-mapping data to Unicode 17.0.0.
    • Improved consistency and reliability of case conversion and regular-expression behavior across environments.
  • Chores

    • Added automated tasks to download, generate, and verify Unicode data.
    • Added integrity checks to detect missing, invalid, or outdated generated data.

The tables were gathered by asking the host CRuby to case every codepoint,
which put whichever Unicode version that Ruby happens to carry into
`unicase.h` and `re_cased.h`, left both headers reproducible only on a
machine carrying the same one, and let a regeneration run on two hosts leave
core and the regexp gem holding different Unicodes.

`tools/unicode/case_data.rb` reads `UnicodeData.txt`, `SpecialCasing.txt` and
`CaseFolding.txt` and hands out the mappings, and both generators take theirs
from it, so how a field is read is stated once. They read them out of a
directory named on the command line, defaulting to the one beside
`case_data.rb`, which is named after the version.

Which version is asked for is `VERSION` there rather than whatever the
directory happens to hold, and `CHECKSUMS` beside it says which bytes that
version is: a file whose digest is not the recorded one is refused rather than
generated from.

Reading the version off a first line would answer for two of the three files.
`UnicodeData.txt` carries no such line, and it is the file most of a table is
read out of, so a directory holding it from another release would generate the
simple mappings of that release, and the swap answers riding on its
decompositions, under a header naming this one. What a digest cannot say is
that the bump itself fetched what Unicode published; it pins the bytes that
were fetched, which is what every regeneration after it has to match.

The files are not in the repository. They are read when a table is
regenerated and at no other time, they weigh 2.3 MB against the 53 KB they
produce, and the license they carry would be a fourth exception in LEGAL for
something nothing compiles. CRuby does not track them either: `.gitignore`
there covers `/enc/unicode/data`.

The simple mappings come from the `UnicodeData.txt` fields, the full ones
from the unconditional `SpecialCasing.txt` entries, and folding from the C
and F entries of `CaseFolding.txt`. A conditional entry is one only a caller
that knows the language or the surrounding text can apply, which is why it is
skipped here as it is there, and why `"ΟΔΟΣ".downcase` stays `"οδοσ"`.

Two answers are in none of the files. The Georgian Mtavruli capitals title to
their lower case, which the database nowhere says and which CRuby spells as a
codepoint range in `enc/unicode.c`. And a title case character swaps to
neither of its cases: each piece of what it decomposes to swaps on its own,
so `U+01C5`, decomposing to D and ž, swaps to "dŽ".

For the refusals `re_cased.h` holds, the single counterpart a folding of
several characters may still leave keeps coming from the lower case mapping,
rather than from the S entries sitting beside the F ones. `U+1FD3`, `U+1FE3`
and `U+FB05` each have an S entry naming a codepoint they fold with, but
core's simple folding reads the folding difference over the lower case
mapping and reaches none of the three; a counterpart the engine cannot use is
not one to record. Both readings put the same codepoints in the table, so
this is about what the pass claims and not about what it writes.

`re_cased.h` also names its generator where it stands. The line read
`tools/gen_cased.rb`, which is where core's generator lives and not where
this one does, so anyone following it to regenerate the table reached the
wrong file.

### Verified

Both regenerated headers are identical to the committed ones apart from the
lines naming where they came from, so nothing compiled changes. Every mapping
was compared against what the host CRuby answers for the same codepoint over
`U+0080` to `U+10FFFF`, and they agree everywhere. The refusals come out of
them as 1,483 pairs and 76 skipped sources over 2,986 cased codepoints, and
reading the S entries instead leaves that cased set where it is.

A `UnicodeData.txt` with one line added to it, in a directory whose other two
files are untouched, is refused with both digests named. A check that read the
version off the first lines instead accepted that same directory, and wrote a
table of 185 lower case runs against 184 and 25 folding runs against 24 under
a header saying Unicode 17.0.0.

`MRUBY_CONFIG=ci/gcc-clang rake -m test`, KO 0, Crash 0 and Warning 0.
Fetching the character database and regenerating the tables were steps
written down nowhere, and each generator had to be run by hand, so a version
bump could update one table and leave the other where it was.

`rake unicode:download` fetches the database `VERSION` names into the
directory the generators read, and `rake unicode:generate` runs both of them.
Asking for a table before anything was downloaded fetches it first, since
each file of the database is a task of its own.

The download prints the digest of each file it got, as `sha256sum` spells it,
so that the bump has them to record in `CHECKSUMS`. It prints rather than
checks: a bump fetches a release before anything can know its digests, and a
download that checked them would have nothing to check against.

### Verified

`rake unicode:generate` with the database in place rewrites `src/unicase.h`
and `mrbgems/mruby-regexp/src/re_cased.h` byte for byte.

`rake unicode:download` over a deleted `CaseFolding.txt` fetches it again and
prints the digest `CHECKSUMS` already holds for it.
That the committed tables are what the generators write is a claim nothing
states, so a header edited by hand, or left behind by a bump that ran one
generator, reads as current.

`rake unicode:verify` regenerates into a temporary directory and compares
what comes out with `src/unicase.h` and `mrbgems/mruby-regexp/src/re_cased.h`
as they are committed, naming every header that is not what the database
generates.

### Verified

`rake unicode:verify` passes on the committed tables, and reports
`src/unicase.h` stale when a line of it is changed by hand.
`src/unicase.h` and `mrbgems/mruby-regexp/src/re_cased.h` are what the two
generators write, and neither carried an include guard. Their neighbours
have one: `src/value_array.h` beside the first and
`mrbgems/mruby-regexp/include/re_internal.h` beside the second, which is
also where the `MRB_` spelling comes from.

Each header is included from one file, so no build changes. What the guard
buys is the second include point: without it, reading either header twice
in a translation unit redefines the objects it holds, the `uni_*` arrays in
one and `re_cased_ranges` in the other. The amalgamated build puts every
core source and every gem source into a single translation unit, so include
points that sit far apart in the tree sit next to each other there.

It also keeps the two out of a reading they do not fit. A header with no
guard is an X-macro table in this tree, `mruby/ops.h` and
`mrbgems/mruby-compiler/include/mrc_ops.h` being the ones that are, and
`lib/mruby/amalgam.rb` sorts a gem's headers by that very test. These two
expand the same way wherever they are read.

The generators write the guards, since they write the headers, and
`rake unicode:verify` keeps the two in step.
@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3b4da6c1-d158-420c-9eec-c6f97b4b10ba

📥 Commits

Reviewing files that changed from the base of the PR and between c02991b and 6a471c8.

📒 Files selected for processing (8)
  • .gitignore
  • Rakefile
  • mrbgems/mruby-regexp/src/re_cased.h
  • mrbgems/mruby-regexp/tools/gen_cased.rb
  • src/unicase.h
  • tasks/unicode.rake
  • tools/gen_unicase.rb
  • tools/unicode/case_data.rb

📝 Walkthrough

Walkthrough

The PR adds a verified Unicode 17.0.0 case-data loader, updates core and regexp table generators to use it, and adds Rake tasks for downloading, generating, and verifying generated headers.

Changes

Unicode case-table generation

Layer / File(s) Summary
Unicode case-data loader
tools/unicode/case_data.rb
Loads and verifies Unicode files, parses case mappings, and composes lower, upper, title, fold, and swap maps.
Core and regexp generators
tools/gen_unicase.rb, mrbgems/mruby-regexp/tools/gen_cased.rb
Use Unicode::CaseData, support an optional UCD directory, and record the Unicode database version in generated output.
Unicode Rake tasks
Rakefile, tasks/unicode.rake, .gitignore
Adds download, generation, and byte-for-byte verification tasks. The Unicode data directory is ignored.
Generated headers
src/unicase.h, mrbgems/mruby-regexp/src/re_cased.h
Updates provenance metadata and adds include guards.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 6a471

This localized change adds include guards to generated Unicode headers and reports successful build, amalgamation, and Unicode verification checks; no actionable merge-blocking risk remains.

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant UnicodeRakeTasks
  participant UnicodeCaseData
  participant TableGenerators
  participant GeneratedHeaders
  Developer->>UnicodeRakeTasks: run unicode:generate or unicode:verify
  UnicodeRakeTasks->>UnicodeCaseData: load verified Unicode data
  UnicodeCaseData-->>TableGenerators: return composed case mappings
  TableGenerators->>GeneratedHeaders: generate core and regexp tables
  UnicodeRakeTasks->>GeneratedHeaders: compare regenerated output
Loading

Possibly related PRs

  • mruby/mruby#7058: Extends Unicode regexp case-folding tables and replaces host-Ruby-derived data.
  • mruby/mruby#7182: Updates the core Unicode case-table generator and generated header.
  • mruby/mruby#7183: Consolidates shared Unicode case-table generation across core and regexp code.

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding include guards to the generated Unicode headers.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@matz
matz merged commit 9156475 into mruby:master Aug 15, 2026
21 checks passed
@takumin
takumin deleted the build/unicode-header-guards branch August 15, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants