mruby-regexp: let a byte that starts no character begin a match - #7059

A match may not start inside a character, so pike_vm() (re_exec.c:333) and
backtrack_exec() (re_exec.c:677) refuse to seed an attempt at a byte in
0x80-0xBF. The test looks at the byte alone, and that is not the same
question: a byte no lead byte reaches is inside nothing. literal_exec(),
the fast path for a pure literal pattern, carries no such test at all, so the
three engines answer differently for the same pattern.

b = "\x81"
Regexp.new(b + b)     =~ (b + b)  # 0,   literal fast path
Regexp.new(b + "{2}") =~ (b + b)  # 0,   literal fast path
Regexp.new(b + "+")   =~ (b + b)  # nil, pike VM
Regexp.new(b + "*")   =~ (b + b)  # 2,   pike VM, an empty match past both bytes

Which engine runs is an implementation detail of the pattern: {2} copies the
atom and keeps the pattern literal, + emits an RE_SPLIT and does not. The
first two lines are what the same subject read as binary gives, so the pike VM
is the one out of step.

Nothing raises in any of these, and no pattern whose first byte is ASCII or a
lead byte is affected, since neither can fall inside a character.

Fix

Ask whether the byte is the interior of a character that starts earlier: walk
back to the nearest byte that is not a continuation byte and see whether the
length mrb_re_utf8_charlen() reports for it reaches this far. At most three
bytes are looked at, and only for a byte in 0x80-0xBF that the search has
stopped on.

literal_exec() takes the same test, so all three engines answer alike. It
needs the binary flag for that, which mrb_re_exec() already holds and
passes to the other two.

The fast path therefore changes in both directions: it declines a byte it used
to match, since Regexp.new("\x81") no longer finds the second byte of "あ",
which is the rule the other two engines already followed.

A subject read as binary is unaffected: the whole test is skipped there, as
before.

Tests

mrbgems/mruby-regexp/test/regexp.rb, a new
Regexp - a byte that belongs to no character is a match position block
before Regexp - multibyte (UTF-8) match extraction: the literal, +, *
and ? forms over a stray byte, a stray byte after an ASCII one, the three
interior positions of a two and a four byte character, and a stray byte on
either side of a character. The last assertion goes through pre_match
bytesize, since MatchData#begin counts characters in a build that has them
and bytes in one that does not.

Verified on x86_64-linux:

• A consistency sweep of 6048 cases over bytes that never form a character
  together (a, \x81, \xFF), 168 patterns against 36 subjects, each run
  once as UTF-8 and once as binary. Every position in such a subject is a
  boundary, so the two readings have to agree: 539 disagreed before this
  change and none do after.
• rake test: 1976 total, 1958 OK, 0 KO, 0 crash, and bintest 105 OK.
• An MRB_INT32 build with clang and -Wall -Wextra: 2045 total, 2035 OK,
  0 KO, 0 crash, and no new warning from any mruby-regexp file (regexp.c
  already emits four -Wunused-parameter).

A start position inside a character, while an attempt runs

The test in pike_vm() carried one more term, curr.count == 0, which the
first commit kept: it skipped the whole loop iteration, so it could only run
while nothing was in flight, or a thread waiting at this position would have
been dropped with it. Any branch that keeps a thread alive past the character
therefore reopens the position.

"ĵ" is C4 B5 and "µ" is C2 B5, so the two share their trailing byte. The
branch of .? that consumes "ĵ" parks a thread past it, and the attempt seeded
at the shared byte then matches that byte on its own, cutting the character in
half.

# "ĵ" is C4 B5, "µ" is C2 B5
"ĵ".match(/.?[µ]/)            # a match of the lone B5, CRuby: nil
"ĵ".gsub(/.?[µ]/, "!").bytes  # [196, 33], CRuby: [196, 181]

This one is older than the rest of the branch and reproduces on master.
backtrack_exec() never carried the term, and /.?[µ](?=)/, which the
backtracking engine runs, answers nil there already.

The second half of the guard is the fix: test the seeding alone instead of the
iteration. The curr.count term goes away, threads seeded earlier still step
at this position, and a start position inside a character stays closed however
many attempts are running.

Tests: a new Regexp - an attempt in flight opens no match position inside a character block after the one above. The match and the gsub over the shared
byte, the same subject behind a three byte character, the two cases where the class
does hold the character that follows, and the shared byte where no lead byte
reaches it. Every case was read off CRuby 4.0.6 first, and the five that are
valid UTF-8 on both sides agree with it. rake test on x86_64-linux: 1989
total, 1970 OK, 0 KO, 0 crash, and bintest 105 OK. Reverting the C change
fails the block on its first three assertions.

Summary by CodeRabbit

• Bug Fixes

  • Improved UTF-8 regular expression matching to avoid starting matches inside valid multibyte characters.
  • Corrected handling of standalone UTF-8 continuation bytes as independently matchable data.
  • Fixed literal and quantified matching positions and byte-based preceding-match sizing.

• Tests

  • Added regression coverage for standalone continuation bytes and multibyte-character boundaries.