mruby-regexp: refuse a subject whose bytes are not UTF-8 in the C searches - #7119

Third of the four pieces #7110 was split into, on top of #7115 and #7116.

CRuby raises ArgumentError when a search is given a subject holding a byte
that spells no character. mruby answered for it instead, so the same program
took a result CRuby would not have produced:

"あ\x80b" =~ /b/         # CRuby: ArgumentError, mruby: 2
"あ\x80b".scan(/./)      # CRuby: ArgumentError, mruby: ["あ", "\x80", "b"]
"\xC0\xBC" =~ /[^<]/     # CRuby: ArgumentError, mruby: 0

Refuse the search instead, wherever the search itself runs in C: =~, match,
match?, ===, index, rindex, byterindex, [], []=, sub, sub!,
gsub with a replacement, gsub!, scan, partition, rpartition,
start_with? and slice!. String#scrub is how a subject like this becomes
matchable.

What is left for the fourth piece

gsub with a block, split and byteindex keep answering for now:

"あ\x80b".split(/b/)        # ["あ\x80"], where "あ\x80b" =~ /b/ raises here
"あ\x80b".byteindex(/b/)    # 4
"あ\x80b".gsub(/b/) { "!" } # "あ\x80!"

Each drives Regexp.__byte_search from a loop in mrblib that searches the same
subject once per match, so a check in the search walks the whole subject again
for every match it finds. What they need is one check at the entry to the
method, which is the next piece, measured there at 105 ms against 1404 ms for
("あ," * 20000).split(/,/).

__regexp_rsearch, which backs rindex, byterindex and rpartition, is a
loop of that kind too. It refuses here, because each of its searches checks;
the same piece hoists the check out of it.

What goes through untouched

A binary string is indexed by byte from end to end, so its bytes make no claim
that could be broken.

A quoted String pattern is exempt on the grounds CRuby exempts it: a literal is
searched for byte by byte, and the subject is read as UTF-8 nowhere along the
way.

"あ\x80b".sub("b", "!")  #=> "あ\x80!" in both, where /b/ is refused in both
"あ\x80b".scan("b")      # ArgumentError in both: `scan` refuses a literal too

sub, sub!, gsub and gsub! quote a String pattern into a Regexp before
they search, so each carries the fact that it was a literal down to the search
rather than letting the quoting hide it.

A string derived from a binary one does not carry MRB_STR_BINARY with it, so
b + "" and b.sub(...) are refused where b itself is not. That gap predates
this change and is left as it is.

Where the check runs

re_check_encoding() walks the whole subject, so it runs once per search a
method makes rather than once per match it finds. Regexp#match, #=~, #===
and #match? check what arrives from Ruby, and so do the class methods
__search, __search_p, __sub_str, __gsub_str and __scan, the last three
of which loop in C over a subject already checked.

Cost

What remains is one walk per search, which a whole subject pays as well. The
flag String#length leaves behind cannot stand in for the walk, since a string
of stray bytes has one byte per character too, so a subject searched twice is
walked twice.

The walk skips a run of ASCII a word at a time and decodes only where a byte
leaves that range, so the cost follows how much of the subject is multi-byte.
Measured at -O3 on a full-core build, the fastest of seven runs each:

The last two are inside the noise of the machine they were taken on, which is
what the second one answering faster after the change says.

One known difference

An out of range positive pos still answers nil rather than raising, since
re_char_to_byte() rejects it before the check runs. CRuby raises there for
Regexp#match; for index and rindex it answers nil as this does.

Tests

The tests whose subject this refuses are the ones #7115 and #7116 gave a
byte-indexed subject to ask through, so each keeps its question and loses only
the subject that no longer reaches the engine. What a match position inside a
whole character is stays pinned on subjects that are whole UTF-8.

Regexp - truncated UTF-8 at subject end is worth calling out: it was a
fuzz-derived regression test for a read past the end of the string buffer, and
the UTF-8 half of the walk it covered is no longer reachable from Ruby under
this rule. The byte-indexed half #7116 added still walks to the end of the
buffer and is what remains.

The one case that turns on what mruby knows about the string rather than on the
engine keeps both answers and branches on __ENCODING__, since a build without
MRB_UTF8_STRING reads no encoding for the bytes to break.

Verified

• rake test on a full-core build (MRB_UTF8_STRING through mruby-encoding):
  2254 tests, all green
• rake test on the default gembox (no MRB_UTF8_STRING): 2061 tests, all
  green
• 34 calls on a broken subject compared against CRuby 4.0.6. All agree except
  the three the next piece takes (byteindex, gsub with a block, split),
  split("b"), which CRuby refuses and the aliased C __split still answers
  and which is not this change, and String#inspect on a byte-indexed string,
  which prints the bytes rather than escaping them
• prek run --all-files passes, except that markdownlint could not install
  locally (npm engine mismatch); no Markdown is touched here

Summary by CodeRabbit

• Bug Fixes
  • Added UTF-8 validation for regular expression operations on text subjects.
  • Invalid UTF-8 now raises ArgumentError instead of producing unreliable matches.
  • Preserved byte-oriented matching for literal strings and byte-indexed data.
  • Improved consistency across matching, scanning, substitution, and global substitution operations.