string.c: let a build index by character and convert case by ASCII - #7190

Follows #7183, which left core with one Unicode case table and mruby-regexp reading it rather than carrying a copy.

MRB_UTF8_STRING buys two things at once. One is that a string indexes, slices and iterates by character; the other is that its case converts by Unicode, which is a table and the walks over it. A target that wants the first and is counting its bytes has no way to leave the second behind. MRB_USE_ASCII_CASE is that way out.

What the option leaves out

The table and the walks are not compiled, and mrb_str_case_convert_unicode() answers that there was nothing to walk. Every caller of it already converts the ASCII of a string in a loop of its own and reaches for the walk only where the string holds more, so that answer hands each of them back to the loop it kept: String#downcase, #upcase, #capitalize, #swapcase and #casecmp? map 'A' to 'Z' and hand every other character back as it stands.

That is where a character indexed build stood before core learned Unicode case, so the option asks for what such a build already had rather than for something new.

/i folds what the build folds

MRB_UNICODE_CASE was the honest arrangement while mruby-regexp carried a case table of its own: a build that never wrote /Ā/i had no reason to ship 4KB it would not read. #7183 took that copy away and /i reads core's table, so the option bought only the two walks over a table the build was carrying anyway, and it paid for them by splitting in two something Ruby has one of:

"Ā".downcase   #=> "ā"
/Ā/i           # RegexpError, on that same build

So the option is turned around and folded into this one. /i folds what the build's own case conversion folds, by Unicode where the build converts by Unicode:

/Ā/i =~ "ā"    #=> 0

and by ASCII where it converts by ASCII, whether it was asked to or reads its strings as bytes and has no character to fold in the first place:

conf.cc.defines << 'MRB_USE_ASCII_CASE'
/Ā/i           # RegexpError: /i needs Unicode case folding for this character

A build that reads its strings as bytes answers as it did. A build that reads characters folds /i by Unicode where it had to ask for MRB_UNICODE_CASE before, and can narrow the whole of its case handling with the new option.

What the refusal says

The message named the option to define. There are now two ways to reach the refusal, defining MRB_USE_ASCII_CASE or reading strings as bytes, so naming either one would be wrong under the other. It names the thing that is missing instead: /i needs Unicode case folding for this character.

The foldings follow the gem

The four foldings /i reads are compiled for mruby-regexp and for nothing else: core's own case conversion calls mrb_uni_case_map() and none of them. They sit in the same object as that mapping, and every build defining MRB_UTF8_STRING calls that, so the linker takes the object whole and a build without the gem carries four functions nothing in it can reach.

So the gem says it is here, the way mruby-encoding already says it, and unicase.c reads that:

spec.build.defines << 'HAVE_MRUBY_REGEXP_GEM'

The declarations in mruby/internal.h go behind the same question, so a caller reaching for one of them where it was not compiled is a compile error rather than a link one.

Tests

build_config/ci/gcc-clang.rb drops the define full-debug carried for the sake of compiling mruby-regexp/test/unicode_case.rb somewhere, that file being what every UTF-8 build compiles now, and gains an ascii-case build for the other side. That build is the only home the refusal has: test/ascii_case.rb compiles into byte-string as well, but a build reading its strings as bytes has no character to refuse and skips those assertions there. It is also what compiles core's ASCII conversion at all.

The Unicode case assertions were guarded by UTF8STRING, which is now the wrong question: such a build indexes by character and may still convert case by ASCII. They ask what they actually need instead, a constant beside it:

UNICODECASE = "Ä".downcase == "ä"

Size

bin/mruby, gcc 13.3.0 -O3, the text column of size, against master. The table is read only data rather than code, so .text alone does not show it:

The second row is master under MRB_UNICODE_CASE against the default here, which is the option going away rather than its effect: a build folding /i by Unicode measures what it measured under the option, and the 5,304 in the first row is that folding becoming what a build carrying the case table does.

MRB_USE_ASCII_CASE takes 11,592 bytes off that and lands 6,288 below what master's default measures. Of the 11,592, .text proper accounts for 5,936 and the read only sections for the rest, the table living in .rodata.

The last two rows are what the second commit is for. A build reading its strings as bytes grows by 32 bytes, which is the two refusal messages spelling out what is missing rather than naming a macro. A build carrying the table but not mruby-regexp measures what master measures; the first commit alone would have left it 4,408 bytes larger, that being the foldings compiled where the gem that reads them is absent.

Verified

MRUBY_CONFIG=ci/gcc-clang rake -m test from a clean build directory on both commits: full-debug, bintest, cxx_abi, byte-string and the new ascii-case, plus the bintests, with KO 0, Crash 0 and Warning 0 throughout.

What the ascii-case build answers was read off the binary as well as the assertions:

"ÄB".downcase      #=> "Äb"
"ä".casecmp?("Ä")  #=> false
"Ā".length         #=> 1
/K/i =~ "k"        #=> 0
Regexp.new("Ā", Regexp::IGNORECASE)
#=> RegexpError: /i needs Unicode case folding for this character: /Ā/

The string still indexes by character, the case methods answer for ASCII, and the two foldings whose result is an ASCII letter go on working without a table.

Summary by CodeRabbit

• New Features

  • Added an ASCII-only case-handling option for UTF-8 strings.
  • UTF-8 indexing remains available while case conversion, comparison, and case-insensitive matching can be limited to ASCII.
  • Regular expressions now report when Unicode case folding is unavailable.

• Documentation

  • Updated guidance on string, regular-expression, configuration, and limitation behavior.

• Tests

  • Added coverage for builds with and without Unicode case-conversion support.