mruby-regexp: refuse a subject whose bytes are not UTF-8 - #7126

CRuby raises ArgumentError when a search is given a subject holding a byte that spells no character. mruby answers for it instead, so the same program takes a result CRuby would not have produced:

"あ\x80b" =~ /b/         # CRuby: ArgumentError, mruby: 2
"あ\x80b".scan(/./)      # CRuby: ArgumentError, mruby: ["あ", "\x80", "b"]
"\xC0\xBC" =~ /[^<]/     # CRuby: ArgumentError, mruby: 0

This follows CRuby wherever a search reads the subject: =~, match, match?, ===, index, rindex, byteindex, byterindex, [], []=, sub, sub!, gsub, gsub!, scan, split, partition, rpartition, start_with? and slice!. String#scrub is how a subject like this becomes matchable.

A binary string goes through untouched, since it is indexed by byte from end to end and its bytes make no claim that could be broken.

What this leaves for the next PR

A quoted String pattern is refused here as well, where CRuby answers for it:

"あ\x80b".sub("b", "!")  # CRuby: "あ\x80!", here: ArgumentError

CRuby searches for a literal byte by byte and reads the subject as UTF-8 nowhere along the way. Exempting it means carrying the fact that the pattern was a literal from sub down to the search it makes, which is a change of its own; it is the PR that follows this one.

Where the check goes, and what it costs

The check goes in each search rather than at the entry to the methods that drive one, because core remembers a string it has read as valid UTF-8. Most entry points search once for what arrives from Ruby, but __byte_search, which the mrblib loops of gsub, split and byteindex drive, is called once per match, and __regexp_rsearch steps __search over every match position. Those turns read the flag core left on the subject after the first one, so what they cost is a flag test and not a walk. A gsub block that modifies the subject clears the flag and the next turn walks what is left; mruby allows such a block where CRuby raises RuntimeError: string modified, and this leaves that as it is.

The walk skips a run of ASCII a word at a time and decodes only where a byte leaves that range, so what a subject nothing has walked yet pays follows how much of it is multi-byte. At -O3 on a full-core build, the fastest of fifteen runs each, with mb for "あ" * 100000 and cs for "あ," * 20000:

The first row builds a subject for every search so that every search walks one. The rest reuse a subject, which is why the loop rows show what the flag costs rather than what the walk does, and they and the ASCII row are inside the noise of the machine they were taken on.

What changed since #7110

#7110 was all of this in one PR. Its test moves are in master already (#7115, #7116, #7120, #7121), so what is left here is the refusal itself, and the quoted String pattern it exempted is the PR that follows this one. Three things it said do not hold, which is why the shape is different:

• It ran the check at the entry to each mrblib loop, through a new Regexp.__check_encoding class method and a flag passed into __search out of __regexp_rsearch, on the grounds that checking inside __byte_search walks the subject once per match, quadratic in the number of matches, and that ("あ," * 20000).split(/,/) took 8 times as long that way. Measured again on master it does not: the flag core sets makes every turn after the first a flag test, and neither placement is distinguishable from the other or from master. The class method and the hoist are gone with it, and __byte_search holds the check.
• It said that a subject searched twice is walked twice, so a walk fell on every search. The flag survives the search and a copy of the string, so the walk falls on the string.
• It said an out of range pos answers nil for index, rindex and byteindex. rindex clamps a position past the end to the end rather than rejecting it, so it reaches the check and refuses, which is what CRuby answers too.

Positions

A pos outside the subject still answers nil rather than raising, since re_char_to_byte() rejects it before the check runs. String#index answers nil for one in CRuby too; Regexp#match raises there, where this answers nil. rindex clamps a position past the end to the end rather than rejecting it, so it reaches the check and refuses, which is what CRuby answers as well.

What the lookbehind tests ask now

#7125 measures a lookbehind in the characters its bytes spell, and its tests put the question to a subject whose bytes spell no character, read as UTF-8:

"\x80ab" =~ /(?<=\x80a)b/          # 2
"\xE3\x81ab" =~ /(?<=\xE3\x81a)b/  # 3

A search like that is refused here, so the block branches on __ENCODING__ and keeps both answers: a UTF-8 build asserts the refusal, and a build that reads no encoding for the bytes to break asserts the widths. The byte rewind the same block asserts is reached in both, since a byte-indexed subject is exempt.

That leaves the character measurement reachable only where MRB_UTF8_STRING is off. A subject that spells characters throughout never matches a lookbehind holding a byte that spells none, at either width, so no other subject puts the question.

Checked

• rake test on the default gembox: 2064 assertions, 0 failures; bintest 105, 0 failures
• rake test on a full-core build (MRB_UTF8_STRING): 2259 assertions, 0 failures
• rake test on a build without MRB_UTF8_STRING, holding mruby-regexp and mruby-string-ext: 1060 assertions, 0 failures
• 20 calls over a broken subject compared against CRuby 4.0.6, and the position cases above

Summary by CodeRabbit

• Bug Fixes

  • Regular expression operations now reject malformed UTF-8 subjects with ArgumentError.
  • Validation consistently applies to matching, searching, replacements, scanning, splitting, and related operations.
  • Binary strings remain supported without UTF-8 validation.
  • Valid UTF-8 behavior and position handling remain unchanged.

• Tests

  • Added coverage for malformed UTF-8 across regular expression and string APIs, including lookbehind and mutation operations.