unicase.c: unfold an ASCII character without reading a table by takumin · Pull Request #7225 · mruby/mruby

takumin · 2026-08-17T04:00:55Z

Compiling a pattern under /i walks the Unicode case tables once per
character and once per character class span, and a pattern of nothing but
ASCII, which is most patterns, walks them for an answer that is not in them.

mrb_uni_case_unfold() (src/unicase.c:238 on master) answers the case
counterparts of one character. It reads ASCII inline from two rules and then
scans both tables anyway:

  /* The folded form is itself a member of the class. */
  if (folded != cp && n < max) out[n++] = folded;

  /* ASCII sources are in no table, so the upper case letter that folds into a
     lower case one is added here. */
  if (folded >= 'a' && folded <= 'z' && folded - 32 != cp && n < max) {
    out[n++] = folded - 32;
  }

  n = unfold_from(&case_tables[MRB_CASE_KIND_FOLD], cp, folded, out, max, n);
  n = unfold_from(&case_tables[MRB_CASE_KIND_LOWER], cp, folded, out, max, n);

That is 24 + 184 runs, each unpacked from six packed bytes by the loop in
case_run_at(), and each surviving candidate re-checked through
mrb_uni_case_fold(), which is two more binary searches.
mrb_uni_case_unfold_range() (src/unicase.c:359) ends the same way over a
span.

Both are on the /i compile path for ASCII. emit_char()
(mrbgems/mruby-regexp/src/re_compile.c:1097) asks for the counterparts of
every ASCII letter a pattern spells, and the closure compile_charclass()
runs over a finished class asks for the counterparts of every run of set bits
in its ASCII bitmap.

Why the answer is not in the tables

The generator drops every source below 0x80: ASCII is what the tables are the
rest of, and the callers fold it inline from the two rules above. What the
scans can still find for an ASCII source is a source above ASCII that folds
down into it, and over the Unicode 17 database there are two, U+017F into
's' and U+212A into 'k'. Nothing else in either table folds below 0x80.
They are the same two re_internal.h already spells out as RE_FOLD_LONG_S
and RE_FOLD_KELVIN for a build with no table at all.

Fix

tools/gen_unicase.rb emits that list as UNI_FOLD_TO_ASCII, an X-macro over
the same two maps the tables themselves are built from:

#define UNI_FOLD_TO_ASCII \
  X(0x0017F, 0x00073)  /* to 's' */ \
  X(0x0212A, 0x0006B)  /* to 'k' */

Measured over the maps rather than written out beside them for the reason
UNI_CASE_MAX_BYTES is measured rather than declared: a Unicode version that
adds a third source gets a third entry here without anyone having to notice
that it had to, and rake unicode:verify fails if the committed header and the
database disagree.

mrb_uni_case_unfold() then answers an ASCII source from the list and returns,
and mrb_uni_case_unfold_range() answers the ASCII part of its span from the
list and hands the walks only what is left above 0x80.

mrb_uni_case_fold_range() keeps both of its walks. It reports the folds of
the sources in a span, and an ASCII source is in no table, so there is no list
for it to read; the gem folds its ASCII bitmap inline already and never asks
that function about a span below 0x80.

Instruction count

callgrind, Ir(2N) - Ir(N) to cancel interpreter start-up. gcc 13.3.0 at
-O3, full-core.

	master	this PR	ratio	`MRB_USE_ASCII_CASE`
`Regexp.new("[a-z0-9_]", //i)` x20,000	1,056,511,426	461,287,040	2.29x	410,565,952
`Regexp.new("hello world", //i)` x20,000	1,797,022,034	447,367,788	4.02x	433,127,448
`/[a-z0-9_]/i =~ "hello"` x100,000	5,385,099,698	2,408,977,326	2.24x	2,154,671,750
`Regexp.new("[\u{80}-\u{2FFF}]", //i)` x2,000	881,046,557	841,504,178	1.05x	`RegexpError`

The last column is a build that drops the Unicode case tables altogether, as
the floor an ASCII-only pattern could reach. The three ASCII rows land within
3% to 12% of it; what remains is the closure the gem runs over a finished
class, not the tables. Under callgrind, mrb_uni_case_unfold_range() and
everything below it is 1.88% of the first row after this change.

Matching itself is untouched. The third row is a plain literal in a loop, which
mruby recompiles on every pass, which is why it moves with the other two.

Size

.text of bin/mruby, build_config/ci/gcc-clang.rb, from a clean build
directory.

build	master	this PR	delta
`byte-string`	1,245,750	1,245,750	0
`ascii-case`	1,266,102	1,266,102	0
`bintest`	1,277,446	1,277,398	-48
`cxx_abi`	1,302,025	1,302,105	+80
`full-debug` (`-O0`)	1,873,206	1,873,398	+192

The first two compile none of this: byte-string has no MRB_UTF8_STRING and
ascii-case defines MRB_USE_ASCII_CASE, and the whole file is behind that
pair. bintest is the build that pays for the tables, and it shrinks: the two
scans lose a call site each. The cxx_abi and full-debug figures are the two
new branches not being folded away, at -O0 for the latter.

Verification

No test is added. What the fast path has to preserve is already pinned, on
every build, by Regexp - /i folds an ASCII letter's class whole
(mrbgems/mruby-regexp/test/regexp_syntax.rb:218), which asserts that
[a-z], [j-l] and [k] under /i all reach U+212A and that [^k] does
not, and by Regexp - Unicode case folding under /i
(mrbgems/mruby-regexp/test/unicode_case.rb:5), which pins the
single-character forms. Both pass.

Beyond that, two differential checks:

Against the two functions themselves. A harness carrying master's
mrb_uni_case_unfold() and mrb_uni_case_unfold_range() verbatim, linked
against the patched libmruby.a, comparing the two as sets, since the walks
may report the same source twice and in spans of a different shape:

every codepoint from U+0000 to U+10FFFF, 1,114,112 of them: 0 differences
221,125 spans: every [lo, hi] inside U+0000..U+0250, every table entry
and its neighbourhood, and 20,000 random spans over the whole range: 0
differences
the same run under ASan + UBSan: clean, 0 differences

Against the binary. 15,277 patterns, each compiled under /i and run
against 118 subjects chosen to cover ASCII, both fold-to-ASCII sources, the
three-way sigma class, dotted and dotless i, and the U+10400 and U+1E900
blocks. Single characters and their positive and negated classes over
U+0000..U+02FF and a stride through the rest, 34 ranges, the ASCII-only
classes the fast path answers whole, and mixed classes. Output of the master
binary and the patched one is byte-identical.

rake test on build_config/ci/gcc-clang.rb, all five builds plus bintest:

build	total	KO	crash
`bintest` (bintest suite)	117	0	0
`full-debug`	2,266	0	0
`bintest`	2,336	0	0
`cxx_abi`	2,336	0	0
`byte-string`	2,332	0	0
`ascii-case`	2,336	0	0

rake test on build_config/asan.rb: 2,336 total, 0 KO, 0 crash, plus its 79
bintests, with no sanitizer report.

rake unicode:verify against the Unicode 17.0.0 database: the committed
tables, including the new list, are what the database generates.

Environment

Details


OS	Ubuntu 24.04, Linux x86_64
gcc	13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)
valgrind	callgrind 3.27.1, for the instruction counts
CRuby	4.0.6, for `tools/gen_unicase.rb`

Compile lines for src/unicase.c in the builds quoted above, paths shortened:

# -O3 full-core, the Ir rows
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/perf/include" -o "build/perf/src/unicase.o" "src/unicase.c"

# -O3 full-core, MRB_USE_ASCII_CASE, the last Ir column
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/perf-ascii/include" -o "build/perf-ascii/src/unicase.o" "src/unicase.c"

# ci/gcc-clang bintest
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK -I"include" -I"build/bintest/include" -o "build/bintest/src/unicase.o" "src/unicase.c"

# ci/gcc-clang full-debug
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/full-debug/include" -o "build/full-debug/src/unicase.o" "src/unicase.c"

# ci/gcc-clang cxx_abi
gcc -MMD -c -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/cxx_abi/include" -o "build/cxx_abi/src/unicase.o" "src/unicase.c"

# ci/gcc-clang byte-string
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/byte-string/include" -o "build/byte-string/src/unicase.o" "src/unicase.c"

# ci/gcc-clang ascii-case
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/ascii-case/include" -o "build/ascii-case/src/unicase.o" "src/unicase.c"

# asan
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -fsanitize=address,undefined -g3 -O0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"build/asan/include" -o "build/asan/src/unicase.o" "src/unicase.c"

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dhp4MAkemDdpxUSPQWZHZ3

Summary by CodeRabbit

Performance
- Improved Unicode case folding for ASCII inputs and ranges, reducing unnecessary processing.
- Accelerated handling of non-ASCII characters that fold to ASCII equivalents.
Compatibility
- Added support for additional Unicode characters folding to ASCII “s” and “k” equivalents.

Compiling a pattern under `/i` asks `mrb_uni_case_unfold()` for the case counterparts of every character it holds, and `mrb_uni_case_unfold_range()` for the counterparts of every span a character class holds. Both finish with two full scans, over the 24 runs of the folding difference and the 184 runs of the lowercase mapping, each run unpacked from six packed bytes and each candidate re-checked through `mrb_uni_case_fold()`, which is two more binary searches. A pattern of nothing but ASCII, which is most patterns, makes those scans once per character and once per span, and collects almost nothing. Almost, rather than nothing, because ASCII is not in the tables at all: the generator drops every source below `0x80`, since the callers fold ASCII inline from the two rules above the scans. What the scans can still find for an ASCII source is a source above ASCII that folds down into it, and there are two of them, `U+017F` into `'s'` and `U+212A` into `'k'`. Nothing else in either table folds below `0x80`. So `tools/gen_unicase.rb` emits that list, `UNI_FOLD_TO_ASCII`, measured over the same two maps the tables are built from rather than written out beside them, for the reason `UNI_CASE_MAX_BYTES` is measured rather than declared: a Unicode version that adds a third gets a third here without anyone noticing it had to. Both entry points then answer an ASCII source, and the ASCII part of a span, from the list, and hand the scans only the range they have anything to say about. Nothing else changes. `mrb_uni_case_fold_range()` keeps both of its walks: an ASCII source is what the tables are the rest of, so its half of the closure has no list to read, and the gem folds its bitmap inline already. Measured with gcc 13.3.0 at `-O3`, `full-core`, Linux x86_64, as callgrind `Ir(2N) - Ir(N)` so that interpreter start-up cancels: | | master | this | `MRB_USE_ASCII_CASE` | |----------------------------------------------|--------------:|--------------:|---------------------:| | `Regexp.new("[a-z0-9_]", //i)` x20,000 | 1,056,511,426 | 461,287,040 | 410,565,952 | | `Regexp.new("hello world", //i)` x20,000 | 1,797,022,034 | 447,367,788 | 433,127,448 | | `/[a-z0-9_]/i =~ "hello"` x100,000 | 5,385,099,698 | 2,408,977,326 | 2,154,671,750 | | `Regexp.new("[\u{80}-\u{2FFF}]", //i)` x2,000| 881,046,557 | 841,504,178 | `RegexpError` | That is 2.29x, 4.02x and 2.24x on the three rows an ASCII pattern was paying for, landing within 3% to 12% of what a build that drops the Unicode tables altogether spends, and 1.05x on the row that genuinely wants them. The remainder above the last column is the general closure the gem runs, not the tables. `.text` of `bin/mruby`, `build_config/ci/gcc-clang.rb`, gcc 13.3.0, against master: byte-string and ascii-case are unchanged, since neither compiles this file; bintest is 48 bytes smaller, cxx_abi 80 larger and full-debug 192.

coderabbitai · 2026-08-17T04:01:21Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ed01d1d8-575c-4351-8c1f-88b2e3835e92

📥 Commits

Reviewing files that changed from the base of the PR and between 34867cf and f9a6eb3.

📒 Files selected for processing (3)

src/unicase.c
src/unicase.h
tools/gen_unicase.rb

Included review availability: Your plan includes up to 8 reviews per rolling hour; 1 remains after this review.

📝 Walkthrough

Walkthrough

The generator now emits non-ASCII-to-ASCII folding mappings. Unicode unfolding uses these mappings directly, returns early for ASCII inputs, and limits mixed-range scans to non-ASCII codepoints.

Changes

ASCII case unfolding

Layer / File(s)	Summary
Generate ASCII fold mappings `tools/gen_unicase.rb`, `src/unicase.h`	The generator derives single-character folds to ASCII and emits them through the `UNI_FOLD_TO_ASCII` X-macro.
Use mappings during unfolding `src/unicase.c`	Single-codepoint and range unfolding use the generated mappings, return early for ASCII-only inputs, and scan tables only from codepoint 128 for mixed ranges.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to f9a6e

This localized optimization changes how ASCII Unicode case counterparts are derived while preserving matching behavior; no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

mruby/mruby#7058: Adds generated ASCII-target fold mappings and optimized unfolding for regexp case folding.
mruby/mruby#7182: Introduces the Unicode case-folding tables and generator extended by this change.
mruby/mruby#7183: Modifies the same Unicode case-folding implementation, header, and generator.

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: adding a table-free fast path for ASCII case unfolding.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

takumin requested a review from matz as a code owner August 17, 2026 04:00

github-actions Bot added core tools labels Aug 17, 2026

takumin mentioned this pull request Aug 17, 2026

/i pattern compilation is up to 25x slower now that Unicode folding is the default, and an ASCII-only pattern pays the most takumin/mruby#149

Closed

matz merged commit 03daa66 into mruby:master Aug 17, 2026
21 checks passed

takumin deleted the unicase-ascii-unfold branch August 17, 2026 04:30

claude Bot mentioned this pull request Aug 17, 2026

Upstream PR scheduling: conflict graph, phases, waves and operating rules takumin/mruby#104

Open

Navigation Menu

Uh oh!

unicase.c: unfold an ASCII character without reading a table - #7225

unicase.c: unfold an ASCII character without reading a table#7225
matz merged 1 commit into
mruby:masterfrom
takumin:unicase-ascii-unfold

takumin commented Aug 17, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 17, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

takumin commented Aug 17, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why the answer is not in the tables

Fix

Instruction count

Size

Verification

Environment

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

takumin commented Aug 17, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 17, 2026 •

edited

Loading