mruby-string-ext: read a String#casecmp? operand, not a copy of it - #7189

Stacked on #7187, whose three commits are the first three here and where str_single_byte_p() comes from. The fourth is this PR's own change, and the diff to read is git diff cb767edb0..HEAD. Merging #7187 first leaves this one a single commit. Every number below is given against master and against that branch both, the master column so the whole stack can be read at once and the middle one so this PR's own change can be.

String#casecmp? folds both sides before it compares them, and it decides whether the fold tables have anything to say about a string by reading the coderange off it:

static mrb_bool
str_folds_beyond_ascii(mrb_value str)
{
  struct RString *s = mrb_str_ptr(str);
  return RSTR_CODERANGE(s) != MRB_STR_CODERANGE_7BIT && !RSTR_BINARY_P(s);
}

A string nothing has read yet records nothing, so it stands at MRB_STR_CODERANGE_UNKNOWN and reads here as a string that folds beyond ASCII, however plain its bytes are. What that answer sends it down is the folding path, which copies both operands, folds the copies and throws them away:

if (str_folds_beyond_ascii(self) || str_folds_beyond_ascii(other)) {
  mrb_value a = mrb_str_dup(mrb, self);
  mrb_value b = mrb_str_dup(mrb, other);
  if (mrb_str_case_convert_unicode(mrb, a, MRB_CASE_FOLD) < 0) str_fold_ascii(mrb, a);
  ...
}

The walk that would have said "nothing but ASCII" is made on the copy, and it goes out with the copy. So the next comparison of the same string starts at UNKNOWN again and pays for the same two copies, and a pair of plain ASCII strings, which is what the method is mostly handed, never stops paying for the walk that would have said so.

Ask the bytes, and leave the answer on the string

What the predicate spells out is what str_single_byte_p() in string.c already answers: it asks the bytes where the string does not say, and a string is single byte where it holds nothing but ASCII and where it is read as bytes, which are the two cases named here by hand. Publish it as mrb_str_single_byte_p() and ask it:

static mrb_bool
str_folds_beyond_ascii(mrb_state *mrb, mrb_value str)
{
  return !mrb_str_single_byte_p(mrb, str);
}

The reading is left on the string it was made about rather than on a copy, so the first comparison walks and every one after it reads a flag. A pair holding nothing but ASCII then orders by its bytes the way it did before this method learned to fold.

A string read as bytes still costs nothing to answer for. The walk is made by mrb_str_valid_encoding_p(), which hands such a string back before it reads a byte of it.

The declaration goes inside the MRB_UTF8_STRING guard rather than beside mrb_str_valid_encoding_p(), which stands outside one. That check answers on both sides because mruby-regexp and mruby-string-ext call it without a guard; this one is asked from inside a guard on either side of it, and a build indexing by byte has no second way to arrive at a single byte string for the answer to be about.

Measurements

The bench-utf8 build under Environment at the end, whose compile line is gcc -O3, the three binaries run against each other five times over and the best of each taken:

a = "abcdefgh"; b = "ABCDEFGH"
2_000_000.times { a.casecmp?(b) }

c = "a" * 4000; d = "A" * 4000
100_000.times { c.casecmp?(d) }

e = "aあbcde"; f = "Aあbcde"
500_000.times { e.casecmp?(f) }

xs = Array.new(100_000) { "aあbcde".dup }
ys = Array.new(100_000) { "Aあbcde".dup }
100_000.times { |i| xs[i].casecmp?(ys[i]) }

gs = Array.new(10_000) { ("a" * 3000 + "あ" * 300).dup }
hs = Array.new(10_000) { ("A" * 3000 + "あ" * 300).dup }
10_000.times { |i| gs[i].casecmp?(hs[i]) }

Each loop is written out as a while in the script that was run, so what is timed is the call rather than a block around it.

The last column is against master.

The first two rows are one flag read standing in for a pair of copies, and what separates them is the length of the pair: the longer the ASCII string, the more of it was being walked and copied for nothing.

The last three are where the reading cannot pay for itself. A pair holding a character above ASCII is folded as it was before, and the bottom two never read a flag back, since no pair in them is compared twice. Those are the rows to read for whether this is paid for elsewhere.

The two left columns are the same work timed twice: master and #7187 both fold, since neither has a flag to read, and the coderange goes out with the copy on both. What separates them is not this method. #7187 changes no function casecmp? calls, and its first two commits leave the second row where master has it; the third, the one that adds str_substr()'s walk to src/string.c and 720 of the 1,232 bytes that file's object gains, is where the middle column parts from the left one.

Generated code

.text over every .o, against the same objects built from master and from this PR's base, for each of the four builds ci/gcc-clang makes. Their compile lines are under Environment at the end: full-debug is -O0, the other three are -O3. Each figure in brackets is against the column to its left.

The two objects that move, and the only two, over the whole stack:

The middle column is #7187's str_substr() walk, which is all of what the stack adds to src/string.o before this PR reaches it and none of what it adds to the gem.

A published function needs one copy of itself to be called through where three inlined flag reads needed none, so src/string.o grows by the call it now has to be reachable through, and the gem falls by the two reads it spelled out. full-debug is the build where this PR's own change is a saving: at -O0 the function was already a call, so publishing it costs nothing there and the gem still drops its two reads. byte-string is unchanged throughout, since what either PR touches stands inside MRB_UTF8_STRING.

Testing

rake -m test over ci/gcc-clang, all four builds green, 0 KO, 0 crash, no new warnings:

rake -m test over build_config/asan.rb is green too, address and undefined sanitizers both: 2313 tests, 2310 OK, 3 skip, plus 79 bintests.

No test comes with this. The answers do not move; what moves is which of them is read off a flag and which is walked for, and there is nothing a script can ask a string that tells those two apart. What stands in for one is the comparison against the base binary, over 3480 cases: 58 operands spanning empty, short and long ASCII, Latin above ASCII, ß and ff and the dotted and dotless i, Greek final sigma, Cyrillic, Japanese, an emoji, bytes that spell no character and bytes below space, taken as every ordered pair and each pair asked casecmp? and casecmp twice over, once while both operands are fresh and once after the first pair of calls has left its reading on them, plus every operand against a Symbol and an Integer. The answers are identical throughout, the ArgumentError that an operand spelling no character raises included. The same 3480 run identically on the byte-string build, and under the sanitizers, which report nothing.

Environment

MRuby::Build.new('bench-utf8') do |conf|
  toolchain :gcc
  conf.cc.flags << '-O3'

  conf.gembox 'default'
  conf.gem :core => 'mruby-encoding'
end

# bench-utf8, the build every timing was taken on
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -O3 -DMRB_USE_SET -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_COMPLEX -DMRB_USE_BIGINT -DMRB_USE_DEBUG_HOOK -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING

# ci/gcc-clang, full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_UNICODE_CASE -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK

# ci/gcc-clang, cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# build_config/asan.rb
clang -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -Wzero-length-array -fsanitize=address,undefined -g3 -O0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

Summary by CodeRabbit

• Bug Fixes
  • Improved UTF-8 string indexing, including negative indexes, bounded lengths, and out-of-range handling.
  • Corrected UTF-8-aware case-insensitive comparisons and string searching for single-byte content.
  • Fixed cached string character information after chomp! and chop! remove non-ASCII characters.
  • Improved substring operations to return nil consistently for invalid positions or lengths.