string.c: write the case walk's answer into the buffer it is building - #7208

A string with one character above ASCII in it is converted a character at a time through an append, and that is what the whole string costs.

What it costs

str_case_convert_utf8() builds the answer beside the string, since a mapping changes how many bytes a character takes, and it appended what it made of every character with mrb_str_cat(). An append from anywhere has questions to ask that an append of one's own bytes does not: it checks the length for overflow, works out whether the source overlaps the string, modifies the string, and carries the coderange across. Every character paid all of that, and every character was read through mrb_utf8_decode() first, ASCII included.

String#downcase, #upcase, #capitalize and #swapcase each keep an ASCII loop of their own and reach this walk only where the string holds more, so a string of ASCII with one character above it in the middle is walked whole. Over 100,000 characters that is 194 instructions per character, where the ASCII loop those methods kept costs 9.5.

What changes

The walk writes into the buffer and holds the length beside it, so what it asks the string for is room for one more character's mapping, and only where fewer than that many bytes are left. A run of ASCII is converted where it stands, needing neither the decoder nor the tables.

The result is the same string. The coderange it records, the nil the bang forms answer with when nothing changed, and the ArgumentError for bytes that spell no character are all where they were.

Speed

Instructions per call, callgrind on bin/mruby, as Ir(2N) - Ir(N) so the startup cancels. Every build is a clean one. perf is not available on this machine and its wall clock moves by up to 80% between runs of the same binary, so the instruction count is the measurement rather than a check on one. Every string is 100,000 characters but the last:

The first row is the loop this walk is not reached from, which is what says the walk is where the cost was. The two long non-ASCII rows keep the walk for every character and save the append alone.

Generated code

.text over every .o, each side built from an empty build directory, for the five builds ci/gcc-clang makes. src/string.o is the only object that moves in any of them:

The 3,272 bytes are an inlining decision rather than code. The walk is smaller than the one it replaces, and gcc now inlines it into the callers where mode reaches it as a constant, which is what the bytes are:

The builds that carry it are the ones already carrying the case tables. A build whose strings index by byte, and one that converts case by ASCII, compile none of this and measure to the byte what master measures.

A version that writes one character per turn of the loop rather than a run of them costs 128 bytes on bintest instead of 3,272 and lands at 3,548,010 instructions for the second row above rather than 1,748,005. It is the same fix at half the recovery; this PR is the other choice.

Testing

rake -m test over ci/gcc-clang from an empty build directory, all five builds green, 0 KO, 0 crash, no new warnings:

rake -m test over build_config/asan.rb is green too, address and undefined sanitizers both: 2313 tests, 2310 OK, 3 skip, plus 79 bintests. Writing into a buffer the walk grows itself is what to run it for.

The new test is for what building beside the string costs. A buffer that leaves an object carries over as many bytes as the object says it holds, and the walk holds its length apart from the string, so it has to say how much it has written or the bytes written so far are dropped where the buffer moves. U+0390 is two bytes and upper cases to six, so a string short enough to live inside its own object converts to one that cannot:

assert_equal "Ϊ́" * 4, ("ΐ" * 4).upcase
assert_equal 24, ("ΐ" * 4).upcase.bytesize
1.upto(12) do |n|
  assert_equal "Ϊ́" * n, ("ΐ" * n).upcase
end

That assertion fails on the version of this commit that does not record the length, which is how the case was found in the first place.

Against master over 2,000 randomly built strings, each put through downcase, upcase, capitalize and swapcase and both of the bang forms, the answers are identical: the bytes, the length, valid_encoding? and whether the bang form said it changed anything. The strings are 0 to 300 characters, spanning the lengths on both sides of what fits inside an object, and are drawn from ASCII, Latin-1, Latin Extended, Greek, Cyrillic, the Roman numerals, the ligatures whose mapping spells several characters, the fullwidth forms, Deseret and Hiragana. Identical under the sanitizers as well.

Environment

MRuby::Build.new('perf-utf8') do |conf|
  conf.toolchain :gcc
  conf.gembox 'full-core'
  conf.cc.flags = %w(-O3 -g -std=gnu99 -Wall -Wundef)
  conf.enable_test
end

# perf-utf8, the build the instruction counts and symbol sizes come from
gcc -O3 -g -std=gnu99 -Wall -Wundef -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK

# ci/gcc-clang, cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, ascii-case
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# build_config/asan.rb
clang -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -Wzero-length-array -fsanitize=address,undefined -g3 -O0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

Summary by CodeRabbit

• Bug Fixes

  • Improved UTF-8 Unicode uppercase conversion for strings requiring expanded output.
  • Invalid UTF-8 input is now rejected consistently during case conversion.
  • Improved handling of ASCII and mixed Unicode text during uppercase conversion.

• Tests

  • Added regression coverage for uppercase conversions that expand string length across multiple input sizes.