mruby-regexp: keep the word class inside ASCII under /i - #7265

Closing a character class under /i folded every ASCII bit alike, whichever form of the class had set it. The bits \w, [:word:] and [:ascii:] set were folded across the ASCII boundary with the rest, so k reached U+212A and s reached U+017F through them, and the negated forms rejected the two characters they have to accept:

"ſ".match?(/[^\w]/i)   # CRuby: true    mruby before: false   mruby after: true
"K".match?(/[^\w]/i)   # CRuby: true    mruby before: false   mruby after: true
/[\w]/i =~ "ſ"         # CRuby: nil     mruby before: 0       mruby after: nil
/[\w]/i =~ "K"         # CRuby: nil     mruby before: 0       mruby after: nil
/[[:ascii:]]/i =~ "ſ"  # CRuby: nil     mruby before: 0       mruby after: nil

Each of the three is a set ASCII defines: \w is [a-zA-Z0-9_] and no more, so a fold that leaves ASCII leaves the set. CRuby keeps them out of the class it folds across the boundary from and folds everything else in the class, the other POSIX brackets included; there those brackets hold every Unicode member, so what the fold adds to them is a member already.

Fix

compile_charclass() now holds what \w, \W, [:word:] and [:ascii:] add in a class of its own until the closure under folding has run, and merges its bitmap and utf8_any into the class after. Each of them holds both cases of every letter it holds, so the ASCII part of the closure had nothing to add to them, and nothing beyond the two foldings above folds to what they hold, so holding them apart changes nothing else. posix_class_bits() reports which names the rule covers.

A letter written out beside the shorthand still folds, since the class then holds it by name as well, and the other POSIX brackets fold as before:

/[\ws]/i =~ "ſ"              # CRuby: 0     mruby before: 0     mruby after: 0
/[\wS]/i =~ "ſ"              # CRuby: 0     mruby before: 0     mruby after: 0
/[k\w]/i =~ "K"              # CRuby: 0     mruby before: 0     mruby after: 0
/[\wk]/i =~ "ſ"              # CRuby: nil   mruby before: 0     mruby after: nil
/[[:lower:]]/i =~ "K"        # CRuby: 0     mruby before: 0     mruby after: 0
"ſ".match?(/[^[:alpha:]]/i)  # CRuby: false mruby before: false mruby after: false
/\w/i =~ "ſ"                 # CRuby: nil   mruby before: nil   mruby after: nil

Every build carries the two foldings (U+212A to k, U+017F to s), so every build was affected the same way; the ascii-case build takes the same fix through the #else branch of the closure.

What stays as it was: this gem's POSIX brackets are ASCII sets, so [[:word:]] and [[:alpha:]] do not hold ſ where CRuby's do; that is a separate divergence, and /i no longer changes the answer for [[:word:]] either way.

Testing

New block Regexp - /i keeps the word class inside ASCII in mrbgems/mruby-regexp/test/regexp_syntax.rb, next to the existing block about the two ASCII-result foldings. It runs on every build that reads its strings as characters (the answers do not depend on the Unicode table, so it is not in unicode_case.rb), and covers: [\w], [^\w], [[:ascii:]], [^[:ascii:]], [\W], [^\W], \w, \W against both U+212A and U+017F; [[:word:]] giving the same answer with and without /i; a letter beside the shorthand in either case and either order ([\ws], [\wS], [k\w], [\wa-z], [^\ws]); naming one letter folding only that letter ([\wk], [\ws]); a member above ASCII still reaching the letter's other case through the fold ([\wſ] matching S); and [[:lower:]] / [[:alpha:]] / [^[:alpha:]] still folding. Every assertion in the block was run under CRuby 4.0.6 as well and agrees.

README: one sentence in the case folding section.

Full suite green at every commit (single commit).

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK src/string.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# ascii-case
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c

Summary by CodeRabbit

• Bug Fixes
  • Corrected case-insensitive regular expression behavior for ASCII-only classes such as \w, \W, [:word:], and [:ascii:].
  • Prevented these classes from incorrectly matching non-ASCII Unicode characters while preserving expected folding for explicitly written letters and other POSIX classes.
• Documentation
  • Clarified the behavior of ASCII-only character classes when using the /i flag.