mruby-regexp: a match may not end inside a character - #7070

A pattern is compiled byte by byte: a literal character becomes a run of
RE_CHAR, and RE_CHAR consumes exactly one byte of the subject. For a
pattern that is valid UTF-8 that is harmless, since its bytes form whole
characters and a run that matches ends where a character ends. A pattern
holding a byte that no character reaches has no such property, and the match
then ends in the middle of one.

"ĵ" is C4 B5. A pattern of the single byte C4 matches its lead byte and
stops there:

j = "ĵ"                                  # C4 B5
j.match(Regexp.new("\xc4"))[0].bytes     # mruby: [196],     CRuby: RegexpError
j.gsub(Regexp.new("\xc4"), "!").bytes    # mruby: [33, 181], CRuby: RegexpError
j.match(Regexp.new("\xc4."))[0].bytes    # mruby: [196, 181] by way of the same start

What comes back is a String that is not valid UTF-8: [196] is half a
character, and the gsub leaves the orphaned continuation byte behind.
Nothing raises at any point. CRuby 4.0.6 answers RegexpError: invalid multibyte character to all three at Regexp.new time.

The engines already refuse to start a match inside a character, in
pike_vm(), backtrack_exec() and literal_exec(). The byte C4 is a lead
byte, so it is a legal start, and no test looked at where the match ends. The
two halves of the same character therefore answered differently:

j =~ Regexp.new("\xb5")   # nil: B5 is inside "ĵ", so no match starts there
j =~ Regexp.new("\xc4")   # 0:   C4 starts "ĵ", and the match ends inside it

Fix

Ask the same question where the match closes.

That position is the RE_SAVE of slot 1, which the compiler emits once,
around the whole pattern, as the end of group 0. add_thread() and
bt_match() take the test there, and literal_exec() takes it on
found + plen. mrb_re_utf8_interior_p() answers it, the same helper the
seeding loops already use, so a byte that no lead byte reaches stays a
boundary of its own and Regexp.new("\x81") keeps finding a standalone
\x81 byte in a subject.

Rejecting the pattern outright, the way CRuby does, was the other way out.
CRuby's rule is an encoding rule: a pattern must be valid in its own encoding,
so Regexp.new("\x81") raises there too, and the way to write a byte pattern
is Regexp.new("\x81".b), a Regexp whose encoding is BINARY. A Regexp
here carries no encoding, and the binary flag is read off the subject at match
time, never off the pattern, so such a pattern has nothing to opt into.
Rejecting every pattern that is not valid UTF-8 would therefore take byte
patterns away outright, including the ones the tests pin, and a build without
Encoding would have no way to ask for one back.

What the pattern keeps instead is the bytes it names, wherever they form no
character:

Regexp.new("\xc4") =~ "\xc4\xc4"   # 0:   C4 C4 makes no character, so both are boundaries
Regexp.new("\xc4") =~ "ĵ"          # nil: C4 B5 is one, and the match would cut it

The test kills a thread, not the whole attempt, so the other branches stay
alive and a longer one can still match:

j.match(Regexp.new("\xc4(?:\xb5)??"))[0].bytes   # was [196], now [196, 181]
j.match(Regexp.new("\xc4*"))[0].bytes            # was [196], now []

Slot 1 belongs to group 0 alone, so a lookaround, which ends at a position
without consuming it and closes on an RE_MATCH of its own, keeps its answer.
A subject read as binary skips the whole test, as before.

What this does not cover

The end of a capture, which can close inside a character while the whole
match does not, is unchanged: Regexp.new("(\xc4)\xb5") against "ĵ" still
hands back half a character in group 1. The start of a capture has the same
hole today and always has, since the existing rule guards where an attempt is
seeded rather than where group 0 opens. Closing one end of a capture and not
the other would trade a symmetric gap for an asymmetric one, so this change
keeps to group 0, where the rule it mirrors already lives.

Tests

mrbgems/mruby-regexp/test/regexp.rb, a new
Regexp - a match does not end inside a character block before
Regexp - multibyte (UTF-8) match extraction: the C4 pattern through each of
the three engines and through gsub, the greedy and non-greedy branches that
do end on a boundary, a lookahead over the same byte, a stray byte on its own
and after an ASCII one, and the binary reading of the same subject.

Verified on x86_64-linux:

• A sweep of 32508 cases, 126 patterns over 258 subjects built from a,
  "ĵ", "あ", an astral character, \x81 and \xff. Every match was
  checked for a begin and an end that is a character boundary: 3471 ended
  inside a character before this change and none do after, while no match
  begins inside one either way. The same sweep compares a subject that holds
  no multi byte character against its binary reading, where every position is
  a boundary and the two have to agree: no disagreement before or after.
• rake test: 2009 total, 1991 OK, 0 KO, 0 crash, and bintest 105 OK.
• An MRB_INT32 build with clang and -Wall -Wextra: 2078 total, 2068 OK,
  0 KO, 0 crash, and no new warning from any mruby-regexp file (regexp.c
  already emits four -Wunused-parameter).

Summary by CodeRabbit

• Bug Fixes

  • Improved regular expression matching with UTF-8 text.
  • Matches that start or end inside a multibyte character are now rejected.
  • Valid alternative and longer matches continue to work correctly.
  • Preserved expected behavior for ASCII-8BIT data.

• Tests

  • Added coverage for quantifiers, groups, lookarounds, substitutions, and invalid byte sequences.