mruby-regexp: refuse a pattern too large for its jump targets - #7064

Every jump target in the compiled bytecode lives in re_inst.offset, a
uint16_t (re_internal.h:43), while code_len is a uint32_t
(re_internal.h:78), and nothing checked that the program stayed addressable
by that field. Once a pattern compiled to more than 65535 instructions the
targets wrapped on the way in, the engine jumped to an unrelated instruction,
and the pattern stopped matching text it describes. No exception, no warning,
just a wrong answer.

r = Regexp.new("(?:abc){21843}x*y")
r.match?("abc" * 21843 + "y")      # CRuby: true, mruby: true

r = Regexp.new("(?:abc){21844}x*y")
r.match?("abc" * 21844 + "y")      # CRuby: true, mruby: false

r = Regexp.new("(?:abc){30000}(?:y|z)")
r.match?("abc" * 30000 + "z")      # CRuby: true, mruby: false

The boundary is exact. In (?:abc){n}x*y the split for x* is patched with
3n + 4: the SAVE(0), the 3n character instructions, the x, the split
inserted in front of it and the loop JMP. 21844 is the first count whose
target reaches 65536, and that is where the answers start diverging. A pattern
with no jump past the boundary, such as (?:abcd){20000} on its own, still
works, which is what makes this so quiet: whether a long pattern is right or
wrong depends on where its quantifiers sit.

This is reachable without pathological input. RE_MAX_REPEAT is 32768
(re_compile.c:496), so a single accepted quantifier such as (?:abc){32768}
already emits 98304 instructions, and {n,m} expansion multiplies a group body
by its count in compile_quantified(), so a generated pattern gets there on
ordinary-looking input.

Cause

Every site that stores a target narrows silently. emit() and patch() take
the target as uint16_t, so the conversion happens at the call: forward
targets from c->code_len, backward targets from the atom start.
compile_quantified() and compile_alt() add explicit (uint16_t)c->code_len
casts of their own, which is where the repros lose the skip for x* and both
targets of (?:y|z). emit_atom_copy() rewrites a copied atom's own targets
through the same narrow field, so {n,m} expansion carries the wrap into every
copy. insert_inst() then relocates targets with offset++ on the same field.

Walking the first repro at n = 21850: (?:abc){21850} leaves code_len at
65551, the * calls insert_inst() to put a RE_SPLIT at the atom start, and
the patch stores 65554 & 0xFFFF, that is 18. The split's skip branch lands in
the middle of the literal run, that thread dies, and the pattern can no longer
skip the x*. The loop JMP emitted one line earlier is wrapped too, storing
65551 & 0xFFFF, that is 15, so the loop's back edge is lost along with the
skip.

There is no memory-safety consequence: a wrapped target is always smaller than
the real code_len, so dispatch stays inside the array, and both engines guard
the top end anyway (re_exec.c:140, re_exec.c:484). An ASAN and UBSan build
agrees, running the repros clean and answering false just the same. The
damage is confined to the answer.

Fix

Raise at compile time rather than widen. emit() is the one place code_len
grows, so a single comparison there covers every producer, including
insert_inst(), which grows the array through emit() as well.
RE_MAX_CODE_LEN sits next to it, in the manner of RE_MAX_REPEAT and
RE_MAX_CLASSES, the two limits this gem already reports.

Widening offset to uint32_t is the alternative. It doubles re_inst from 4
to 8 bytes for every pattern in the program to serve patterns that essentially
nobody writes, and the struct's own comment says the 4-byte size is for
alignment. The cheap check looks like the right trade for a gem that targets
small systems. CRuby refuses oversized patterns the same way, for example
Regexp.new("a{100001}") raises RegexpError with too big number for repeat range, so raising here is behaviour to copy rather than invent.

The bound is on the whole program rather than on each target, so a pattern that
used to fit only because its targets happened to stay low now raises as well.
(?:abc){21843}x*y is one: it compiles to exactly 65536 instructions and its
highest target is 65533, so it answered correctly before. The window is one
count wide per shape, which the sweep below measures, and that is the price of
keeping the check to one comparison on the single funnel.

The offset >= 0xffff guard in insert_inst() was an ad hoc brake on the same
overflow. It cannot be reached now: insert_inst() calls emit() first, and
emit() refuses to grow the program to where an offset of 0xffff could
exist, so the guard goes with the change it was standing in for.

The executors are not touched. They read whatever the compiler wrote and are
correct for any in-range target.

Tests

Regexp - pattern too large for its jump targets is refused in
mrbgems/mruby-regexp/test/regexp.rb: the largest (?:abc){n} that still
compiles and the first one that does not, with the message asserted; the two
shapes that used to answer wrongly rather than raise; and (?:ab){32768}, a
quantifier the parser still accepts that reaches the bound on its own. The
counts sit either side of the bound, so a later change to the instruction cost
per atom cannot quietly move the boundary back into the working range.

The assertions are on compilation only. Compiling either pattern is instant,
but matching the below-bound one needs a 64 KB subject and about 3.2s on a host
build, which does not belong in the suite.

Verified on x86_64-linux

• A differential sweep against CRuby 4.0.6 over three shapes
  ((?:abc){n}x*y, (?:abc){n}(?:y|z), (?:abc){n}(?:y)?z) and n from
  21835 to 21850, 48 cases, each matched against a subject the pattern
  describes. CRuby answers true on all 48. Before this change mruby answered
  false on 20 of them; after it, none answers false. 23 now raise
  RegexpError: the 20 wrong answers, plus the 3 cases that were correct
  before and that the whole-program bound gives up (x*y at 21843, (?:y|z)
  at 21843, (?:y)?z at 21844).
• rake test: 1988 total, 1969 OK, 0 KO, 0 crash, and bintest 105 OK.
• MRUBY_CONFIG=build_config/asan.rb rake test (clang, ASAN and UBSan,
  MRB_UTF8_STRING): 2165 total, 2162 OK, 0 KO, 0 crash, no sanitizer report,
  and bintest 78 OK.
• clang -Wall -Wextra over re_compile.c: no new warning.

Summary by CodeRabbit

• Bug Fixes
  • Regular expressions that exceed the supported program size are now rejected with a clear RegexpError.
  • Prevented oversized patterns from producing invalid jump targets or unexpected compilation behavior.
  • Added coverage for large repetitions, quantifiers, and alternation combinations.