mruby-regexp: classify a POSIX bracket by Unicode above ASCII - #7278

The POSIX brackets held their ASCII members and no character above them, so on any text that is not ASCII both polarities answered wrongly: [[:alpha:]] missed every letter of every other script, and [[:^alpha:]] took them all.

"あ" =~ /[[:alpha:]]/          # CRuby: 0     mruby before: nil   mruby after: 0
"aあx" =~ /[[:^alpha:]]x/       # CRuby: nil   mruby before: 1     mruby after: nil
"１２３" =~ /[[:digit:]]+/      # CRuby: 0     mruby before: nil   mruby after: 0
"あ" =~ /[[:word:]]/           # CRuby: 0     mruby before: nil   mruby after: 0
"あ" =~ /\w/                   # CRuby: nil   mruby before: nil   mruby after: nil
"ā" =~ /[[:upper:]]/i          # CRuby: 0     mruby before: nil   mruby after: 0

A build reading its strings as characters knows that "あ" is one character; what it lacked was a table saying what kind. This PR carries that table in the gem, generated from the Unicode Character Database the case tables already come from, and reads it wherever a bracket is asked about a character above ASCII. \d, \w and \s are ASCII in Ruby's syntax and stay so.

The types

The types are the ones CRuby's engine gives the brackets:

tools/unicode/ctype_data.rb reads the files and spells each type as the properties it is the union of; mrbgems/mruby-regexp/tools/gen_ctype.rb packs the answer into mrbgems/mruby-regexp/src/re_ctype.h, and rake unicode:generate runs it beside the case generators. PropList.txt and DerivedCoreProperties.txt join the files tools/unicode/ucd.rb names and checksums; the tables are regenerated together and read one release between them.

The table

The types are held together rather than one range list each. Every codepoint has one set of answers, so the codepoint space above ASCII is cut into the 3,468 runs over which the set does not change, and a run is one 32-bit entry: the codepoint it starts at in the high 21 bits and the set in the low 11. One binary search answers all the types at once, and the runs are a fifth of what the types would take as separate lists (the letters alone are 759 ranges). cntrl has no bit: above ASCII it is the C1 controls and nothing else, which two numbers answer, and leaving it out is what lets the set fit beside a codepoint. ASCII is not in the table; the compiler sets those bits from the list it always had.

The table is compiled on the condition the case table is, MRB_UTF8_STRING without MRB_USE_ASCII_CTYPE, under a RE_UNICODE_CTYPE defined beside RE_UNICODE_CASE: a build that asked to leave the case table behind is counting its bytes and wants this one no more. Without it, and where strings are read as bytes, a bracket holds its ASCII and no character above it, and its negation everything above, which is what every build answered before.

In the class

A bracket does not spell its type out as members. [[:alpha:]] would put some 760 ranges into a class and have every character read through them one by one; instead re_charclass keeps two masks, the types its positive brackets name (ctype_yes) and the types its negated ones name (ctype_no), and class_match() reads the type of a character above ASCII once its ranges have said nothing. A character is in through a bit its type has, or a bit it lacks: [[:^alpha:][:^upper:]] holds "ā" and not "Ā". A byte that is no character, from a byte-indexed subject, has no type and is in through a negated bracket alone, which is what CRuby answers for an ASCII-8BIT subject.

Under /i a member the class holds by bit or by range is closed under folding at compile time as before, and a type is closed at match time instead: the type read is that of the character and of every character sharing its folding (mrb_uni_case_unfold()), so [[:upper:]] holds "ā" through "Ā" and "Dž" through "DŽ", and [[:^upper:]] holds "Ā" through "ā". The ASCII counterparts are left out of that reading, because the closure over the bitmap has already reached across the boundary from them: "ſ" is in [[:upper:]] under /i once s is, through the ranges. The reading lives in re_utf8.c (mrb_re_class_ctype_match()), out of line of class_match(), so that the four sites the matcher is inlined into do not each carry it.

Commits

1. tools: keep which database the tables read apart from what is read of it: tools/unicode/ucd.rb takes the release, the files, their digests and the check out of case_data.rb, so a second reader can name the same database. rake unicode:verify finds the committed tables up to date.
2. mruby-regexp: classify a POSIX bracket by Unicode above ASCII: the table and everything above.

Size

bin/mruby, full-core, gcc 13.3.0 -g -O3 (the toolchain default), size -A, master and this branch built at the same path:

The read-only data is the 13,872 bytes of the table and 16 of alignment. The .text of the classifying build is mrb_re_class_ctype_match() (431), mrb_re_ctype() (90), the bracket parser posix_class_bits() (+146), the masks in compile_charclass() (+67, inlined into compile_seq()), the class matcher's call at its four sites (+256 in exec_range(), +80 in bt_match()) and the first-set walk (+8). The MRB_USE_ASCII_CTYPE build's .text moves without a byte of the change being compiled there: posix_class_bits() gained an argument it never writes on that build, and the compiler now inlines it into compile_charclass() and leaves that out of compile_seq(), where it had them the other way round. The byte build's sections are the same size as master's.

Testing

Every codepoint above ASCII against every bracket in both polarities ([[:x:]], [[:^x:]], [^[:x:]]), and under /i for upper, lower, alpha, alnum, word, punct and graph: 63 patterns, each run by String#scan over the whole codepoint space on this build and on CRuby 4.0.6 (Unicode 17.0.0), comparing the count and a checksum of the codepoints matched. All 63 agree. The one disagreement on the way was word, off by two until Join_Control (U+200C, U+200D) joined it.

New test files, selected by mrbgem.rake on the same condition as the case pair: mrbgems/mruby-regexp/test/unicode_ctype.rb on a build with the table (members and non-members of every bracket above ASCII in three polarities, runs, a lookbehind, brackets beside members and ranges, the /i closure including the title case letter and the two ASCII-reaching foldings, and a byte-indexed subject) and mrbgems/mruby-regexp/test/ascii_ctype.rb on a build without it (every bracket holding nothing above ASCII and its negation everything). Every assertion in unicode_ctype.rb was run under CRuby 4.0.6 as well and agrees. The [:word:] comments in regexp_syntax.rb say what now depends on the build. README: the brackets under Pattern Syntax and a paragraph under Configuration; doc/guides/mrbconf.md: what MRB_UTF8_STRING and MRB_USE_ASCII_CTYPE now cover.

Full suite green at every commit.

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK src/string.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# ascii-ctype
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CTYPE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c

Summary by CodeRabbit

• New Features

  • Regular expressions now support Unicode-aware POSIX character classes, including alpha, digit, word, space, punctuation, and related classes.
  • Case-insensitive matching works with Unicode POSIX classes when enabled.
  • ASCII-only configurations consistently restrict POSIX classes to ASCII characters.

• Documentation

  • Updated configuration and regexp documentation to explain Unicode and ASCII behavior, supported classes, and build-specific limitations.

• Tests

  • Added comprehensive coverage for Unicode classification, negation, case folding, indexing, and ASCII fallbacks.