mruby-regexp: rewind a lookbehind by characters - #7087

Follow-up to #7069, which stopped compute_fixed_len() from measuring a character
class that can match a non-ASCII codepoint. A lookbehind over one raises RegexpError
instead of rewinding into the middle of a character and answering wrongly. That traded
a wrong answer for an exception; it did not make the pattern work. These all raise
today and answer in CRuby:

"Āx" =~ /(?<=[Ā])x/      # CRuby: 1,   mruby: RegexpError
"Āb" =~ /(?<![Ā])b/      # CRuby: nil, mruby: RegexpError
"あx" =~ /(?<=[^あ])x/    # CRuby: nil, mruby: RegexpError
"aあx" =~ /(?<=a\W)x/     # CRuby: 2,   mruby: RegexpError
"ax" =~ /(?<=.)x/        # CRuby: 1,   mruby: RegexpError

The last row is not a class at all: RE_ANY was rejected by compute_fixed_len() for
the same reason, and the same change lets it through. With this patch every row
answers; the indices differ from CRuby only where this build counts bytes and CRuby
counts characters, so the first row answers 2 here and 1 there for the same position.

Why one count cannot do it

The count in re_inst.a is a byte count, and both lookbehind opcodes subtract it from
sp directly. A class has no fixed byte width, but it consumes exactly one character
whatever its members are, so a character count makes every class measurable. The
subject decides the unit, though, and the compiler does not know the subject:
bt_match() advances a binary subject one byte at a time and hands class_match()
the raw byte as its codepoint, so a byte count is right there, for a literal and for a
class alike. A character count is right for a class against a UTF-8 subject and wrong
for a multibyte literal, which is 2 bytes and 1 character. Storing only one of them
regresses the other:

$ ./build/host/bin/mruby -e 's = "Āx".b; p(s =~ /(?<=Ā)x/)'
2

(?<=Ā)x measures correctly today because 2 is a byte count. Make it a character
count and the same pattern rewinds 1 byte against that subject, landing on \x80,
and stops matching.

Fix

The lookbehind carries both numbers. a keeps the byte count, and a carrier
instruction, RE_LB_WIDTH, emitted right after the lookbehind holds the character
count; the sub-pattern body starts at pc + 2. The stream already holds logical units
spanning several 4-byte words, since a multibyte literal is a run of one-byte
RE_CHAR instructions forming one atom, so the carrier adds no new kind of shape.

• compute_fixed_len() returns both counts from its one walk. The byte count stays
  one per consuming instruction, because a binary subject advances one byte whatever
  the instruction is. The character count adds one per class or RE_ANY and counts
  only lead bytes across an RE_CHAR run. The class_is_ascii_only() test and the
  RE_NCLASS and RE_ANY rejections all disappear; RE_SPLIT stays rejected, so
  (?<=ab|c) keeps raising as before.
• The executor keeps sp - a for a binary subject and otherwise steps back
  code[pc + 1].a characters. The backward step is built on
  mrb_re_utf8_interior_p(), whose definition (a continuation byte no lead reaches
  is a character of its own) keeps the walk on the same boundaries the forward decode
  uses, broken input included. Running out of text keeps its current meaning: the
  positive form fails, the negative form succeeds.
• The landing stays exact without new checks: the rewind walks that many character
  boundaries, and each atom of a fixed-length sub-pattern consumes exactly one whole
  character going forward, so RE_MATCH arrives back at sp by the same argument as
  the byte version.
• The 255 limit stays a byte limit, and the character count never exceeds the byte
  count, so it fits the carrier's uint8_t.

Tests

The case from #7069 that pinned the raising patterns flips to real match assertions,
covering the class forms, the negated classes, the shorthands and (?<=.); the case
that pinned what must keep working is untouched. A new case pins binary subjects,
where the byte count is the one a character count would silently regress. rake test
is clean.

Summary by CodeRabbit

• New Features

  • Improved regular-expression lookbehind support for character classes, shorthand classes, and multibyte characters.
  • Lookbehind now correctly handles both UTF-8 text and binary data.

• Bug Fixes

  • Fixed lookbehind position tracking to rewind by character width for text and by byte count for binary subjects.
  • Improved behavior when lookbehind patterns exceed the available input.