mruby-regexp: stop a repetition on an empty iteration in the backtracker - #7269

The Pike VM stops a repetition once its body has run without consuming;
bt_match() had no such stop. A repetition whose body can match empty went
round at the same position until MRB_REGEXP_RECURSION_LIMIT refused the
next frame, and what the engine then answered was whatever the alternatives
left inside the limit produced. That was CRuby's answer often, and not always:

/(?:(?:b*)+)+?/ =~ ""                 # CRuby: 0,     mruby: nil
/(?:(?:b*)+)+?/ =~ "b"                # CRuby: 0,     mruby: nil
/(?:(?:b?)+)+?/ =~ ""                 # CRuby: 0,     mruby: nil
/ca(?:b??b??)+a*?/.match("cab")[0]    # CRuby: "ca",  mruby: "cab"
/(?:a?)*b??/.match("aab")[0]          # CRuby: "aa",  mruby: "aab"
/(?>(?:a*)*)b?/.match("aab")[0]       # CRuby: "aab", mruby: "aa"
/(a*?)*b/.match("aab").begin(1)       # CRuby: 2,     mruby: 0
/(?:(?!a))*b?/.match("b")[0]          # CRuby: "b",   mruby: ""
/a(?:(?<=a))*b?/.match("ab")[0]       # CRuby: "ab",  mruby: "a"

In the first three the limit is reached inside the inner +, and the outer
+? then loops in the frame at the limit until the step budget is gone. In
the fourth the empty iterations run to the limit, and the lazy b?? in the
frames being unwound is then allowed its b. In the last two the frame at
the limit cannot open the branch of the ? that follows and answers with the
branch that skips it. The cost was there even when the answer was right: each
such loop climbed 1000 frames per start position, and a nested one spent the
whole step budget per start position, 7 ms for /(?:(?:b*)+)+?/ =~ "" and
80 ms against "abcdefghij".

The stop

The backtracker gets Onigmo's null check. mark_empty_loops() already finds
every back edge whose loop body can match empty and marks it for the Pike VM;
it now marks the head of a jump-closed loop as well, since that is where an
iteration of e* begins. bt_state gains a record per pc: the offset the
running iteration of the loop that pc keys began at. The frame that runs an
edge into an iteration writes the record and undoes it when the iteration's
frame returns (bt_iter()), so the record follows the backtracking, and the
branch that begins an iteration is run by recursion rather than in place so
that there is one place to undo it. The edge closing the body compares: an
iteration that ends where it began matched empty, and the loop takes its exit
instead of going round again, keeping what the iteration captured, as the
Pike VM does. e* is keyed by its head (the head writes, the closing jump
reads) and e+ by its closing fork (which writes and reads). Only loops
whose body can match empty pay the record and, for a lazy one, the extra
frame; every other fork runs as before.

The recursion limit no longer stops any loop, and its comment says what it
bounds now. The rule from #7256 that a limit is not a cut and not a
lookaround's answer stays: a frame giving up at the limit may still be inside
an atomic group's body or a negative lookaround.

bt_match() took ten arguments, seven of which are the same in every frame
of one search. The first commit moves the shared seven into bt_state, which
backtrack_exec() fills once and each frame reads through a pointer, so that
a frame carries only what is its own: the position, the pc and the depth. The
iteration records then have a place to live.

Lookarounds and backreferences

epsilon_path() answered FALSE at a lookaround and at a backreference, so a
repetition whose body was one of them was not marked, and the stop did not
apply: (?=)+, (?:(?!a))* and (a?)\1* still ran to the recursion limit.
A lookaround is zero-width whatever its sub-pattern does, so the walk steps
over the sub-pattern to the end the instruction records; a backreference to a
group that captured empty consumes nothing, so it can lie on a path that need
not consume. Neither reaches the Pike VM, which runs neither construct, so
this part is the backtracker's alone.

Time

The default gembox plus mruby-benchmark, -O3, Benchmark.realtime over
n matches after three warm-up calls, mean of two runs each, master and this
PR alternated:

CASES = [
  ["/(?:(?:b*)+)+?/ =~ ''",            2000,  -> { /(?:(?:b*)+)+?/ =~ "" }],
  ["/(?:(?:b*)+)+?/ =~ 'abcdefghij'",  200,   -> { /(?:(?:b*)+)+?/ =~ "abcdefghij" }],
  ["/(?:a*)*b+?/ =~ 'b'",              20000, -> { /(?:a*)*b+?/ =~ "b" }],
  # ...
]
CASES.each do |label, n, blk|
  3.times { blk.call }
  t = Benchmark.realtime { n.times { blk.call } }
  puts "%-38s %10.3f us/iter" % [label, t / n * 1e6]
end

The first seven rows are repetitions that used to run to the limit; the
first two are the nested case that used to spend the step budget. The rows
after them are loops that consume on every iteration, or the Pike VM, and
stay where they were; 1.4 us is what such a match costs on the Pike VM too.

Size

.text of bin/mruby, build_config/ci/gcc-clang.rb, each side from a clean
build directory. re_compile.o and re_exec.o are the objects that change;
in bintest they account for -1,280 and -1,744 of the delta; -1,328 of the
second is the first commit alone, the ten-argument recursive calls becoming
four-argument ones.

Verification

The tests go in regexp_syntax.rb beside the Pike VM's empty-iteration
tests. They cover the nested loops that used to answer nil, the lazy body
that stays empty when the loop stops, each layout of a repetition stopping on
its own (e*, e+, e*?, e+?, e{n,}, e{n,}?, and nested), the empty
iteration's capture being kept as under the Pike VM, a repetition of each of
the four lookarounds and of a backreference, the record being undone when an
iteration is backtracked out of, and the stop inside and around an atomic
group.

Differential against CRuby 4.0.6. 10,000 random patterns over a, b,
a?, b?, [ab]?, \w?, (?:), (a|), (|b), (?:a|), the four
lookarounds, the two lookaheads also with a capture inside, backreferences,
plain, non-capturing and atomic groups and alternation, groups nested up to
two deep, every atom quantified with probability 0.7 by one of *, +, *?, +?,
?, ??, {0,}, {1,}, {2,}, {0,}?, {1,}?, half of them with a lazy
x?? appended to keep the pattern on the backtracking engine, each run with
match against one of 18 subjects over a and b and compared as
MatchData#to_a. 5,404 of the patterns mruby refuses to compile (a quantifier
on a quantifier, which CRuby accepts with a warning) and 85 CRuby cannot
finish under a memory or time cap; the rest compare:

Of the 111 lines where both differ, 108 have a capture group inside a
lookaround, and so do all 10 where only this PR differs. That is one thing
this engine already answers differently from Onigmo: a lookaround's body runs
as a sub-pattern call, so a capture written inside it stays written when the
text after the lookaround fails and the engine backtracks past it, where
Onigmo undoes it. On master those patterns ran to the recursion limit and
failed outright; here the loop stops, the match goes on, and the leaked
capture is what gets read. The remaining 3 lines differ on master as well.

rake test, build_config/ci/gcc-clang.rb, no compiler warning:

The default configuration: 2,128 total, 0 KO, 0 crash, plus its 112 bintests.

Environment

# ci/gcc-clang bintest
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK -I"include" -I"mrbgems/mruby-regexp/include" -I"build/bintest/include" -o "build/bintest/mrbgems/mruby-regexp/src/re_exec.o" "mrbgems/mruby-regexp/src/re_exec.c"

# ci/gcc-clang full-debug
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"mrbgems/mruby-regexp/include" -I"build/full-debug/include" -o "build/full-debug/mrbgems/mruby-regexp/src/re_exec.o" "mrbgems/mruby-regexp/src/re_exec.c"

# ci/gcc-clang cxx_abi
gcc -MMD -c -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"mrbgems/mruby-regexp/include" -I"build/cxx_abi/include" -o "build/cxx_abi/mrbgems/mruby-regexp/src/re_exec.o" "mrbgems/mruby-regexp/src/re_exec.c"

# ci/gcc-clang byte-string
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"mrbgems/mruby-regexp/include" -I"build/byte-string/include" -o "build/byte-string/mrbgems/mruby-regexp/src/re_exec.o" "mrbgems/mruby-regexp/src/re_exec.c"

# ci/gcc-clang ascii-case
gcc -MMD -c -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -I"include" -I"mrbgems/mruby-regexp/include" -I"build/ascii-case/include" -o "build/ascii-case/mrbgems/mruby-regexp/src/re_exec.o" "mrbgems/mruby-regexp/src/re_exec.c"

Summary by CodeRabbit

• Bug Fixes

  • Improved regular expression handling for repetitions that consume no input.
  • Prevented potential hangs or excessive backtracking with empty patterns, lookarounds, backreferences, nested repetitions, and atomic groups.
  • Improved matching consistency for greedy and lazy quantifiers.

• Tests

  • Added regression coverage for empty-iteration behavior and capture preservation across complex regular expression patterns.