mruby-regexp: reject a lookbehind over a multibyte character class - #7069

compute_fixed_len() counts a character class as exactly one byte, but a class can hold
non-ASCII codepoints, so a lookbehind over one rewinds by the wrong number of bytes and
lands in the middle of a character. A positive lookbehind then never matches, and a
negative lookbehind matches where it must not.

"Āx" =~ /(?<=[Ā])x/      # CRuby: 1,   mruby: nil
"Āb" =~ /(?<![Ā])b/      # CRuby: nil, mruby: 2
"Ăx" =~ /(?<=[Ā-ă])x/    # CRuby: 1,   mruby: nil
"Āx" =~ /(?<=[aĀ])x/     # CRuby: 1,   mruby: nil
"ĀĀx" =~ /(?<=[Ā]{2})x/  # CRuby: 2,   mruby: nil

The indices differ between the two anyway, because CRuby counts characters where this
build counts bytes, so read the rows as match against no match. The ASCII side is
unaffected: "ax" =~ /(?<=[aĀ])x/ answers 1 in both.

A negated class goes wrong in the opposite direction, since the rewind lands on a
continuation byte that the complement happily accepts:

"あx" =~ /(?<=[^あ])x/    # CRuby: nil, mruby: 3
"あb" =~ /(?<![^あ])b/    # CRuby: 1,   mruby: nil

Shorthand like \W, \S and \D reaches the same broken position, but a bare
(?<=\W) hides it: the rewind lands on a continuation byte, mrb_re_utf8_decode()
hands that raw byte back as its own codepoint, and utf8_any accepts anything at or
above 128, so the wrong position answers right by accident. Put one more element beside
the class and the accident stops:

"aあx" =~ /(?<=a\W)x/     # CRuby: 2,   mruby: nil
"aあb" =~ /(?<!a\W)b/     # CRuby: nil, mruby: 4
"あĀx" =~ /(?<=\W\W)x/    # CRuby: 2,   mruby: nil

Cause

compute_fixed_len() folds RE_CHAR, RE_CLASS and RE_NCLASS into one arm and adds
1 byte for each. That is right for RE_CHAR, which is a single byte by construction,
and wrong for a class: re_charclass carries a non-ASCII codepoint range list beside
its ASCII bitmap, class_match() reads it for any codepoint at or above 128, and the
executor advances by the decoded character width, not by one byte.

The result is stored in re_inst.a and used as a raw byte count by both lookbehind
opcodes, which subtract it from sp and re-enter bt_match() there. With the count
short by one byte per multibyte class member, that position is a continuation byte, the
sub-pattern fails, and the failure is silent.

Fix

A corrected constant is not available. A class can match characters of different widths
in the same position, so [aĀ] is one byte or two and \W is one to four. This patch
measures only a class that is ASCII throughout, and returns -1 for the rest, which is
what the function already does for RE_ANY and every other shape it cannot measure. A
lookbehind that cannot be measured raises RegexpError instead of answering wrongly.

RE_NCLASS always returns -1: the complement of an ASCII bitmap admits non-ASCII
whatever its members are. RE_CHAR keeps its own arm, since a multibyte literal is a
run of one-byte RE_CHAR instructions and so (?<=Ā)x measures correctly today.

class_is_ascii_only() names the test; first_set_walk() already spelled out the same
condition inline and now shares the helper.

Rewinding by characters instead of bytes would make every class work rather than raise,
and would also let (?<=.) compile, which CRuby accepts and this gem rejects today. It
needs a backwards UTF-8 helper and changes what re_inst.a means for the lookbehind
opcodes, so it is left for a separate change.

Tests

Two cases next to the existing lookbehind group in
mrbgems/mruby-regexp/test/regexp.rb: one pins the patterns that now raise, including
the negated class and the uppercase shorthands, the other pins what must keep working,
[a-z], \d, \s and the multibyte literal. rake test is clean.

Summary by CodeRabbit

• Bug Fixes

  • Fixed regular expression lookbehind validation for character classes that may match multibyte characters.
  • Negated character classes and multibyte-capable shorthands now correctly raise RegexpError when used in unsupported fixed-length lookbehinds.
  • Improved matching optimization without changing valid ASCII-only lookbehind behavior.

• Tests

  • Added regression coverage for multibyte character classes, shorthands, ASCII-only classes, and multibyte literals in lookbehinds.