string.c: read String#chop! back from the last byte - #7122

Problem

String#chop! finds the start of the last character by walking the whole string
from the front, one character at a time. That is O(n) for a single call and
quadratic for a loop that chops a string down. An ASCII receiver pays it too,
since the walk is not skipped for one.

const char* t = RSTR_PTR(s), *p = t;
const char* e = p + RSTR_LEN(s);
while (p<e) {
  mrb_int clen = mrb_utf8len(p, e);
  if (p + clen>=e) break;
  p += clen;
}
len = p - t;

Change

mrb_utf8_char_head() answers the same question from the other end, reading back
at most three bytes, since nothing longer than four bytes spells a character.

const char* t = RSTR_PTR(s);
const char* e = t + RSTR_LEN(s);
len = mrb_utf8_char_head(t, e-1, e) - t;

str_char_rindex() and mrb_str_rindex_m() already read a boundary this way.
The byte-indexed branch and the \r\n pair are untouched.

Why the two agree

Every byte after a lead byte inside a multi-byte character is a continuation
byte, since that is what mrb_utf8len() requires before it answers more than 1.
So the front walk can step over neither a lead byte nor a stray continuation
byte: it lands on each of them, and those are exactly the boundaries
mrb_utf8_char_head() reads back to. Whether a lead byte reaches a given
continuation byte is mrb_utf8len()'s answer on both sides, so an overlong
sequence, a surrogate, and a lead byte the string end cuts short are read the
same way by either.

Checked as well as argued. With both functions lifted into a standalone harness,
the old and new cuts agree on all 137,560 strings of length 1 to 4 over 19 bytes
chosen at the interesting boundaries (0x80, 0x8F, 0x90, 0x9F, 0xA0, 0xBF, 0xC0,
0xC1, 0xE0, 0xED, 0xF0, 0xF4, 0xF5, 0xFF among them) and on 2,000,000 random
byte strings of length 1 to 12. The same cases were then run through builds of
bin/mruby from before and after, byte for byte.

Measurements

gcc 13.3.0, -O3, x86_64, full-core gembox with MRB_UTF8_STRING.

The first row still carries the copy chop makes, which is most of what is
left of it.

Tests

String#chop! had no case ending in a byte that spells no character, so one is
added in the commit before the change and passes on both sides of it: a stray
continuation byte, a string of nothing else, an overlong sequence and a
surrogate RFC 3629 forbids, and a truncated lead byte, next to a four-byte
character and the \r\n pair.

rake test with the config above: 2255 examples, 0 failures, 0 crashes. The
default host config passes its unit tests too (2061 examples, 0 failures); the
bintest failure it shows there (uninitialized constant digits::other_excl,
from mruby-range-ext) is present on an untouched tree and unrelated.

Summary by CodeRabbit

• Bug Fixes

  • Improved String#chop! handling for UTF-8 strings, including invalid, truncated, and multibyte character sequences.
  • Preserved correct byte-based behavior for binary strings and non-UTF-8 configurations.
  • Ensured CRLF endings continue to be handled correctly.

• Tests

  • Added coverage for UTF-8 edge cases and standalone continuation bytes.