Walk the backward regexp search by byte - #7148

String#rindex, String#byterindex and String#rpartition reach a Regexp answer through __regexp_rsearch, which walks the subject from the start and keeps the last match that starts at or before the limit, because the engine searches forward only. The walk spoke character offsets, and a character offset is not a place a subject can be read from:

("ab" * 10000).rindex(/b/)   # 0.012s
("あb" * 10000).rindex(/b/)  # 0.584s   same match count, same walk

Every turn of that loop counted the whole subject, up to three times:

• Regexp.__search(pattern, self, pos) reaches the byte offset the engine takes by counting pos characters out from the start of the subject (re_char_to_byte(), then mrb_str_char_to_byte(mrb, str, 0, char_off)).
• md.begin(0) counts a byte offset back to a character offset the same way (re_byte_to_char()), and the loop asked for it twice: once for the limit test, once for the position to resume at.

Neither counts for anything on a single-byte subject, where MRB_STR_SINGLE_BYTE makes both conversions the identity. On a multibyte one the walk read the whole subject once per match and took quadratic time.

Walk in bytes

Nothing is given up by leaving character space. A byte inside a character is not a position a match can start at, and the engine steps over one rather than seed an attempt there, in nfa_exec(), bt_exec() and literal_exec() alike (mrb_re_utf8_interior_p()), so resuming one byte past the match start reaches the next character by itself:

Regexp.__byte_search(/./, "あb", 1).__byte_begin(0)  #=> 3, not 1

Resuming one position past the match start rather than at the match end is what keeps an overlapping match in view ("aaa".rindex(/aa/) is 1, where resuming at the end would answer 0). That is a statement about overlap and not about character boundaries, and one byte past keeps it exactly the way one character past did.

So limit is a byte offset for every caller, which is what byterindex already handed it, and the loop reads a match start with __byte_begin(0). rindex reads the character position it was given into a byte one where it is given, once: the whole subject is its own byte length, which is the form rindex is in whenever it is called with one argument, and only a position the caller named is measured. Regexp.__byte_search does not range check its position, so the walk stops itself at the end of the subject.

Numbers

Full-core build with MRB_UTF8_STRING, best of three:

                                     before    after
("あb" *  5000).rindex(/b/)          0.174s    0.007s
("あb" * 10000).rindex(/b/)          0.584s    0.019s
("あb" * 20000).rindex(/b/)          2.315s    0.060s
("あb" * 10000).byterindex(/b/)      0.393s    0.017s
("あb" * 10000).rpartition(/b/)      0.574s    0.016s
("ab"  * 10000).rindex(/b/)          0.012s    0.012s

Doubling the subject quadrupled the time before and roughly triples it now, which is the part this does not fix: every successful search publishes the pre match and post match globals, so the walk still copies the subject once per match. That is common to every search loop in the gem, gsub, split and scan included, and is its own change.

Tests

The first commit pins where the answers stand before any of it moves. rindex bounds the match start in characters and byterindex bounds it in bytes, which is the whole of the difference between them, and no test said so: the ones that pass a position pass it to "hello", where the two readings name the same place, and the one multibyte pair reads the answer back rather than the argument. The new test asks both with a position on "あいうあいう", and asks whether an overlapping match still stands when the match ahead of it is multibyte. Every answer is CRuby's. It skips on a build that reads every string as bytes, where the two readings agree again.

Dropping the character position conversion the second commit adds turns 4 of its assertions red, including the one argument form.

Beyond the suite, a differential ran the character-space walk and the new byte-space one against each other over 27 subjects (ASCII, multibyte, 4 byte characters, byte-read, and bytes that read as no character) by 24 patterns (empty, literal, quantified, anchored, lookahead, lookbehind, backreference, alternation), at every position from past one end to past the other, comparing the answer, the exception where one is raised, and $~, $&, $`, $' and $1 after the call. 21,672 checks on the UTF-8 build and 24,552 on the byte-indexing one, 0 mismatches.

Verification

Full suite green at every commit on build_config/ci/gcc-clang.rb: full-core with MRB_GC_STRESS (2281), bintest (2282), the C++ ABI (2282), and the default gembox, where the new test skips (2067). 0 failures, 0 crashes, no new compiler warnings.

Summary by CodeRabbit

• Bug Fixes
  • Improved reverse regular-expression searches for UTF-8 and other multibyte strings.
  • Corrected character-based limits for rindex and byte-based limits for byterindex.
  • Fixed rpartition behavior when processing multibyte text, including negative and overlapping-match scenarios.
• Tests
  • Added regression coverage for UTF-8 indexing, byte offsets, bounds, and partitioning.