string.c: follow an ASCII run only as far as the index asks - #7185

mrb_str_char_to_byte() walks a string to find the byte a character index names. Where the byte under the cursor is ASCII it hands the run to search_nonascii() and steps over the whole of it at once, since every ASCII byte stands for a character of its own:

while (p<e && i<idx) {
  if ((*p & 0x80) == 0) {
    const char *np = search_nonascii(p, e);
    ptrdiff_t alen = np - p;
    if (idx < i+alen) {
      p += idx-i;
      i=idx;
    }
    else {
      p = np;
      i += alen;
    }
  }

The limit it hands over is e, the end of the string, so the run is followed to wherever it ends and the first arm then cuts it back to idx when it went too far. Where a string is ASCII apart from something near its end, the run ends only at that something, and asking for the character just past the head reads as many bytes as asking for the last one does.

idx is the point past which the run cannot matter. This PR hands that over as the limit instead:

const char *lim = (e - p) > (idx - i) ? p + (idx - i) : e;
const char *np = search_nonascii(p, lim);
i += np - p;
p = np;

alen is now at most idx - i, so the arm that cut an overshooting run back cannot be reached and the two fold into the one that remains. The walk stops at the same byte it stopped at before, having read only the bytes before it.

Who asks

Every caller that names a character index near the head of a long string. String#split("") is the one that pays worst, because it asks for one character at a time from wherever it has got to:

end = mrb_str_char_to_byte(mrb, str, idx, 1);

On master each of those calls reads the ASCII run out to the end of the string, so splitting a string of n ASCII characters reads n²/2 bytes. Here each call reads one.

The others are String#index with an offset, String#[] and String#slice, String#[]=, String#rindex with a non-negative offset, String#slice!, and Regexp#match with a position.

Measurements

The bench-utf8 build under Environment at the end, whose compile line is gcc -O3, best of 5 runs of best of 5:

s = "a" * 100000 + "é"
20000.times { s.index("a", 1) }
20000.times { s[0, 2] }

a = "a" * 20000 + "é";  20.times { a.split("") }
b = "a" * 100000 + "é"; 20.times { b.split("") }

s[0, 2] halves rather than better because str_substr() counts the whole string with mrb_str_char_len() before it asks, so one of its two reads is elsewhere. The split("") ratio grows with the string because what it drops is the quadratic term.

A string whose bytes the walk has to read anyway reads the same as it did, and a single byte string never enters the loop: mrb_str_char_to_byte() answers idx at its head.

Generated code

.text over every .o, against the same objects built from master, for each of the four builds ci/gcc-clang makes. Their compile lines are under Environment at the end: full-debug is -O0, the other three are -O3.

byte-string is the build without mruby-encoding, where mrb_str_char_to_byte() is the other definition, the one that answers idx and has no walk in it.

Testing

rake -m test over ci/gcc-clang, all four builds green, 0 KO, 0 crash, no new warnings:

rake -m test over build_config/asan.rb is green too, address and undefined sanitizers both, tests and bintests.

Against master over 8789 cases of String#[] and 6720 of String#index and String#rindex, spanning ASCII, multi-byte, broken and binary receivers with offsets on both sides of every edge, the answers are identical byte for byte, under the sanitizers as well.

No test accompanies the change. The walk answers what it answered, so there is nothing new to pin.

Environment

MRuby::Build.new('bench-utf8') do |conf|
  toolchain :gcc
  conf.cc.flags << '-O3'

  conf.gembox 'default'
  conf.gem :core => 'mruby-encoding'
end

# bench-utf8, the build every timing was taken on
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -O3 -DMRB_USE_SET -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_COMPLEX -DMRB_USE_BIGINT -DMRB_USE_DEBUG_HOOK -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING

# ci/gcc-clang, full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_UNICODE_CASE -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK

# ci/gcc-clang, cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# ci/gcc-clang, byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER

# build_config/asan.rb
clang -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -Wzero-length-array -fsanitize=address,undefined -g3 -O0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER