mruby-regexp: refuse a subject whose bytes are not UTF-8 - #7110

CRuby raises ArgumentError when a search is given a subject holding a byte
that spells no character. mruby answered for it, so the same program took a
result CRuby would not have produced:

"あ\x80b" =~ /b/         # CRuby: ArgumentError, mruby: 2
"あ\x80b".scan(/./)      # CRuby: ArgumentError, mruby: ["あ", "\x80", "b"]
"\xC0\xBC" =~ /[^<]/     # CRuby: ArgumentError, mruby: 0

Refuse the search instead, for =~, match, match?, ===, index,
rindex, byteindex, byterindex, [], []=, sub, gsub, scan,
split, partition, rpartition, start_with? and slice!. String#scrub is how a
subject like this becomes matchable.

Two commits

The first moves the tests that ask what the engine does with a byte standing
for no character onto a byte-indexed subject. What each of them pins belongs to
the pattern, and a byte-indexed subject puts the same question to the engine,
so they keep asking it after the second commit refuses the UTF-8 subject they
used to go through. That commit changes no behavior and is green on its own.

The second is the refusal itself, which leaves it holding the check, the tests
whose answer it changes, and nothing else.

What goes through untouched

A binary string is indexed by byte from end to end, so its bytes make no claim
that could be broken.

A quoted String pattern is exempt on the grounds CRuby exempts it: a literal is
searched for byte by byte, and the subject is read as UTF-8 nowhere along the
way.

"あ\x80b".sub("b", "!")  #=> "あ\x80!" in both, where /b/ is refused in both
"あ\x80b".scan("b")      # ArgumentError in both: `scan` refuses a literal too

sub, sub!, gsub and gsub! quote a String pattern into a Regexp before
they search, so each carries the fact that it was a literal down to the search
rather than letting the quoting hide it.

A string derived from a binary one does not carry MRB_STR_BINARY with it, so
b + "" and b.sub(...) are refused where b itself is not. That gap predates
this change and is left as it is.

Where the check runs

re_check_encoding() walks the whole subject, so it runs once per search a
method makes rather than once per match it finds.

The entry points that take a subject from Ruby check it themselves:
Regexp#match, #=~, #===, #match?, and the class methods __search,
__search_p, __sub_str, __gsub_str and __scan, the last three of which
loop in C over a subject already checked.

The two entry points an mrblib loop drives leave the check to that loop, which
runs it once on the subject it holds fixed. Regexp.__check_encoding at the top
of gsub with a block, of split and of byteindex covers their
__byte_search calls, and __regexp_rsearch, which backs rindex,
byterindex and rpartition, runs it and passes the flag that says so to
__search.

Checking inside those loops instead walks the subject once per match, which is
quadratic in the number of matches: ("あ," * 20000).split(/,/) took 8 times as
long that way.

Cost

What remains is one walk per search, which a whole subject pays as well. The
flag String#length leaves behind cannot stand in for the walk, since a string
of stray bytes has one byte per character too, so a subject searched twice is
walked twice.

The walk skips a run of ASCII a word at a time and decodes only where a byte
leaves that range, so the cost follows how much of the subject is multi-byte.
Measured at -O3 on a full-core build, three runs each:

One known difference

An out of range positive pos still answers nil rather than raising, since
re_char_to_byte() rejects it before the check runs. CRuby raises there for
Regexp#match; for index, rindex and byteindex it answers nil as this
does.

Tests

What a match position inside a whole character is remains pinned on subjects
that are whole UTF-8. A build without MRB_UTF8_STRING reads no encoding for
the bytes to break, so the cases that turn on one answer positions there and
branch on __ENCODING__.

The engine reads a byte-indexed subject through a branch of its own, so the
tests the first commit moved stop covering the UTF-8 one. Regexp - truncated UTF-8 at subject end is the case where that mattered: it was a fuzz-derived
regression test for a read past the end of the string buffer, and the code path
it covered is no longer reachable from Ruby under this rule.

Verified

• rake test on a full-core build (MRB_UTF8_STRING through mruby-encoding):
  2246 tests at the first commit and 2247 at the second, all green
• rake test on the default gembox (no MRB_UTF8_STRING): 2057 and 2058 tests,
  all green
• 30 String-argument methods compared against CRuby 4.0.6 on a broken subject.
  The two remaining differences are split("b"), which CRuby refuses and the
  aliased C __split still answers, and slice!("b"), which is wrong on whole
  UTF-8 too and is not this change
• prek run --all-files passes, except that markdownlint could not install
  locally (npm engine mismatch); no Markdown is touched here

Summary by CodeRabbit

• Bug Fixes
  • Improved UTF-8 validation across regular expression search, matching, splitting, scanning, and substitution operations.
  • Invalid UTF-8 subjects now raise ArgumentError consistently in UTF-8 builds.
  • Preserved byte-oriented behavior for literal string operations and binary data.
  • Improved handling of malformed, truncated, and overlong UTF-8 sequences, including character-class matching.