mruby-regexp: make a multibyte literal one atom - #7056

compile_atom() emits a literal one byte at a time
(re_compile.c:931). For a character above 127 that means the lead byte and
each continuation byte become atoms of their own, and compile_quantified()
binds a quantifier to the last atom emitted. /Ā+/ therefore compiled as
\xC4(\x80)+: the lead byte once, then the continuation byte repeated.

/Ā+/.match("ĀĀ")[0]      # CRuby: "ĀĀ", mruby: "Ā"
/Ā*/.match("ĀĀ")[0]      # CRuby: "ĀĀ", mruby: "Ā"
/Ā{2}/.match?("ĀĀ")      # CRuby: true,  mruby: false
/Ā{2,3}/.match("ĀĀĀ")   # CRuby: a match, mruby: nil

Three and four byte characters go the same way, and a quantified literal
after another atom is affected too.

/日+/.match("日日")[0]     # CRuby: "日日",   mruby: "日"
/𝕏+/.match("𝕏𝕏")[0]     # CRuby: "𝕏𝕏",   mruby: "𝕏"
/aĀ+/.match("aĀĀ")[0]   # CRuby: "aĀĀ", mruby: "aĀ"

Scanning turns one run into one match per character, which is the form most
likely to be noticed as wrong output rather than as a missing match.

"ĀĀxĀ".scan(/Ā+/)  # CRuby: ["ĀĀ", "Ā"], mruby: ["Ā", "Ā", "Ā"]

Nothing raises in any of these.

Fix

Emit every byte of the character in one go, so the atom spans the whole
character.

That is all it takes, because the quantifier machinery already works on a
span rather than on a single instruction: emit_atom_copy() copies the whole
[start, code_len) range for {n,m}, and the RE_SPLIT that
insert_inst() puts in front of the atom for * and ? covers the range as
well. A group, which is many instructions, has always relied on this; a
multibyte literal was the one atom that did not produce a span.

Where the character ends comes from mrb_re_utf8_charlen(), which derived it
from the lead byte alone and never looked at the bytes that follow. That is too
loose to build an atom from. For the pattern bytes C4 78 2B it reports 2, so
x joins the atom and + repeats \xC4x; for C4 2B it reports 2 as well,
and the + itself is swallowed and stops being a quantifier at all.

b = "\xC4"  # lead byte of a two byte character, standing on its own
Regexp.new(b + "x+").match(b + "xxx")[0].bytesize  # 2 without the check, 4 with it
Regexp.new(b + "+").match(b + b)                   # nil without the check, a match with it

So this checks the continuation bytes there as well. A sequence that never
completes keeps reporting a length of 1, its bytes stay atoms of their own, and
such a pattern reads as the bytes it is made of. mrb_re_utf8_decode() takes
its length from the same function, so the string being matched is read the same
way: . takes an incomplete lead byte alone instead of pairing it with
whatever follows.

Nothing downstream changes. compute_fixed_len() counts each RE_CHAR as one
byte and reaches the same total, first_set_walk() already returns early on a
lead byte above 127, and the literal prefix scan reads whatever run of
RE_CHAR it finds.

Tests

mrbgems/mruby-regexp/test/regexp.rb, two new blocks before
Regexp - multibyte (UTF-8) match extraction.

Regexp - quantifier on a multibyte literal: +, *, {n} and {n,m} on
a two byte literal, the three and four byte cases, a quantified literal
preceded by another atom, a non-greedy form, scan, and an absent optional
literal. The assertions compare bytesize, since a match that stops one
character early is still a match and only the length tells the two apart.

Regexp - quantifier on an invalid multibyte literal: a lead byte followed by
an ASCII byte, a lead byte followed by the quantifier itself, a sequence cut
short by the end of the pattern, a valid character right after a stray lead
byte, and . over a stray lead byte on the subject side.

Verified on x86_64-linux:

• A differential sweep of 3223 cases against CRuby 4.0.6: four literals from
  one to four bytes, eleven quantifier forms, each plain, preceded by another
  atom and preceded by a lookbehind, over four subject lengths, plus five stray
  lead bytes with five followers each. 234 cases disagreed before this change
  and none do after. Patterns that are not valid UTF-8 are compared against a
  binary CRuby regexp, since CRuby rejects them outright while mruby has no
  encodings.
• rake test: 1977 total, 1959 OK, 0 KO, 0 crash, and bintest 105 OK.
• An MRB_INT32 build with clang and -Wall -Wextra: 2046 total, 2036 OK,
  0 KO, 0 crash, and no new warning from any mruby-regexp file (regexp.c
  already emits four -Wunused-parameter).

Summary by CodeRabbit

• Bug Fixes

  • Fixed regular expression handling for multibyte UTF-8 characters, ensuring complete characters are treated as single pattern atoms.
  • Improved quantifier, optional-match, non-greedy, and scanning behavior for multibyte literals.
  • Invalid or incomplete UTF-8 sequences continue to be handled safely.

• Tests

  • Added regression coverage for 2-, 3-, and 4-byte UTF-8 characters across supported quantifier patterns.