mruby-regexp: skip free-spacing whitespace in the parser and leave the pre-pass with comments and escape widths - #7272

preprocess_pattern() in mruby-regexp does three things before the parser reads the pattern: it removes (?#...) groups, it removes # comments under /x, and it removes whitespace under /x. CRuby does the first two in a pre-pass (re.c) and the third in the tokenizer, which skips whitespace only where it fetches a token and reads every token, an escape, a group name, an interval {n,m}, a (? opener, with the whitespace inside it in place. Doing all three by deleting bytes from the source means that whatever the deletion brings together is what the parser reads, and each kind of token boundary has needed a rule of its own: #7252 (inline x scoping) and #7268 (the (?:) written between a digit escape and a digit) each added one, and master still reads differently at every boundary without one:

Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (target of repeat operator is not specified), mruby: 0
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
# CRuby: nil (`{1, 2}` is not an interval), mruby: 0
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
# CRuby: ["a b"], mruby: ["ab"]
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
# CRuby: 0 (`\k` is the letter and `<ab>` a literal), mruby: nil (it reads `\k<ab>`)
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
# CRuby: nil (a vertical tab is not free-spacing whitespace), mruby: 0

The comment rule from #7268, that a removed comment does join the bytes on either side of it, is right for \1-\9 and wrong for the escapes CRuby's pre-pass has already read whole, and rewritten at a fixed width, by the time it reaches the comment:

Regexp.new("\\1(?#c)0") =~ "\x08"      # CRuby: 0 (`\10`), mruby: 0
Regexp.new("\\u12(?#c)34") =~ "ሴ"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it reads `\u1234`)
Regexp.new("\\u#c\n{61}", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it reads `\u{61}`)
Regexp.new("\\x(?#c)61") =~ "a"
# CRuby: RegexpError (invalid hex escape), mruby: 0 (it reads `\x61`)
Regexp.new("\\x6(?#c)1") =~ "a"
# CRuby: nil (`\x06` then `1`), mruby: 0 (it reads `\x61`)
Regexp.new("\\0(?#c)61") =~ "1"
# CRuby: nil (`\0` then `61`), mruby: 0 (it reads `\061`)

The two layers

Each of mruby's two layers gets one of CRuby's two, so that no rule has to be added per boundary. Two commits, and a third with one test line.

Whitespace moves into the parser. RE_FLAG_EXTENDED was already carried in c->flags and scoped by the save and restore every group does for (?i) and (?m); skip_extended_space() reads it at the two places a token can end: at the top of compile_seq(), before each atom and before the | or ) that ends the sequence, and in compile_quantified() between the atom and its quantifier. Nowhere else, so {1, 2} is not an interval, ( ? is a group and then a ?, a group name keeps its blanks, and a numeric escape reads what stands after it: \x6 1 is \x06 and 1 because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under /x in CRuby and now here. The (?:) insertion goes with its esc_end and blank_out bookkeeping (the #7268 tests stay as they are and hold through the parser), and has_rewritten_group(), which decided whether the pass runs at all, is replaced by a memchr() for #: both things the pass still removes are spelled with one, and neither the /x flag nor a (?x) needs the pass any more.

The pre-pass keeps what re.c does, (?#...) under any flags and #... under /x, with the scope stack from #7252 saying where /x is on. Both stay in the pre-pass rather than moving to the parser because CRuby removes both before it reads escapes: \1#c, a newline and 0 is \10 there just as \1(?#c)0 is. skip_uninterpreted() steps over an escape at the width CRuby's pre-pass reads it: \u{...} to its brace, \uXXXX and the next four bytes whatever they are, \x and its hex digits (with none, the one byte after, for the parser to reject), \0 and its octal digits, written at full width (\x6 as \x06, \0 as \000) so that a removed comment cannot lengthen them; the rewrite buffer is twice the source for that. \1-\9 are the backslash and one digit, and the digits after them are plain bytes, so a removed comment joins them into \10 as it does in CRuby.

One shape of pattern that compiled on master is refused now. a* ? under /x was read as the non-greedy a*? once the blank was gone, and a{2} ? the same way; CRuby reads a ? a blank away from the quantifier as a repeat of the repeat, (?:a*)?, and this engine refuses a repeat of a repeat wherever it is written, a** included, so it refuses these too rather than give them a meaning CRuby does not.

Time

Wall clock, minimum of 9 alternating runs, -O3, default gembox plus mruby-benchmark: the path this PR changes, a Regexp.new under /x with and without a # and a /x literal, which is compiled on every turn.

x = Regexp::EXTENDED
xpat = "(?<n>\\d+) # the number\n\\s* (?<u>\\w+) # the unit\n"
100000.times { Regexp.new("a b c", x) }                 # new_x_nocomment
100000.times { Regexp.new("a # c\nb # d\nc", x) }       # new_x_comment
100000.times { Regexp.new(xpat, x) }                    # new_x_named
100000.times { Regexp.new("abc") }                      # new_plain
100000.times { Regexp.new("a(?#c)bc") }                 # new_comment_group
100000.times { "abc" =~ /a b c/x }                      # match_x_literal
100000.times { "42 px" =~ /(?<n>\d+) # the number
                           \s* (?<u>\w+) # the unit
                          /x }                          # match_x_comment

A /x pattern without a # no longer runs the pass or takes its allocation, which is the -16%. With a # the pass still runs and now copies the whitespace through for the parser to skip, so the pattern's whitespace is walked twice instead of once; match_x_comment, whose literal carries about a hundred bytes of indentation, pays that second walk, +3%.

Size

.text of bin/mruby, build_config/ci/gcc-clang.rb, each side from a clean build directory at the same path. re_compile.o is the only object that changes and accounts for the whole delta.

Verification

Two new blocks in mrbgems/mruby-regexp/test/regexp_syntax.rb, one per commit. The first covers whitespace between tokens (literals, groups, alternatives, anchors, quantifiers, lookarounds, an atomic group, a named group and its \k, a numbered backreference), the five bytes skipped and the vertical tab that is not, escaped blanks, and whitespace inside a token: ( ?i), (?i ), {1, 2}, { 2}, {2 }, a* ? against a*?, the \u, \x and \0 spellings CRuby rejects or reads short, and group names with a blank inside, declared and referenced, with \k <ab> as the letter and \k#c\n<ab> and \k(?#c)<ab> as references. The second covers a comment removed from inside \uXXXX, \u{...}, \x and \0, with and without /x, the short escapes written at full width (\x6(?#c)1, \0(?#c)61), and a class keeping the bytes of (?#c) as members. The block that checks that the pass and the named-group scan step over the same constructs now names the three readers (the pass, the scan, the parser) and says which rows each reads, with a # added to two of its class rows so that the pass still runs there. Every expected value was checked against CRuby 4.0.6 first. The third commit adds one line to the Regexp#to_s block in regexp.rb: a /x pattern with a # comment printed as (?x-mi:...) and read back through Regexp.new, the one spelling that goes through the comment pass and the inline x scope on a single compile.

Differential against CRuby 4.0.6, bintest build (MRB_UTF8_STRING) of master and of this PR. 100,000 random patterns of one to six tokens drawn from a, b, digits, the five whitespace bytes and a vertical tab, #c with and without a newline, (?#c), \x, \x6, \x61, \0, \06, \1, \10, \u, \u00, \u0061, \u{61}, \u{61 62}, \u{, \u{61, \k, \k<a>, \k<a b>, \k'a', (?<a>, (?<a b>, (?'a', (?'a b', (a), (, ), (?:, (?x), (?-x), (?x:, (?i), (?xi:, (?=, (?<=, (?>, [, ], [a b], [#], [\x6 1], {, }, {2}, {1, 2}, {1,2}, *, +, ?, |, \ , \\, ^, $, ., \b, \d, <, >, -, ', e, f; each compiled with or without Regexp::EXTENDED and matched against one of 35 subjects, compared as MatchData#to_a and Regexp#names. A pattern both sides refuse compares as equal whatever the message.

Of the 782, 481 are patterns master refused as a quantifier with no target, having made an interval of a {1, 2} with nothing before it or removed a vertical tab before a quantifier, 293 are answers that change (a blank inside a name, \k <, a vertical tab, ( ?), and 8 are \u#c and \x#c under /x. Of the 1,889, 1,533 are a \1 that names no group, which this engine accepts and CRuby rejects, 160 are a repeat of a repeat, which this engine refuses and CRuby accepts, 151 are a [ inside a class, and 42 are answers that differ on master as well (an inline toggle before a |, a(?i)|b, which Onigmo scopes over the alternation and this engine does not; \b at a non-ASCII word character; {n}?, which Onigmo reads as an optional repeat); none is new. All 12 where only this PR differs are {1, 2} before a \1 with no group: master stopped at the interval, this PR reads it as CRuby does and reaches the \1.

rake test, build_config/ci/gcc-clang.rb, no compiler warning:

The default configuration: 2,138 total, 0 KO, 0 crash, plus its 112 bintests.

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/re_compile.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK mrbgems/mruby-regexp/src/re_compile.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/re_compile.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/re_compile.c
# ascii-ctype
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CTYPE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/re_compile.c

Summary by CodeRabbit

• Bug Fixes

  • Improved extended-mode regular expressions so whitespace is handled correctly between tokens without altering spaces inside intervals, group names, or Unicode codepoint lists.
  • Improved handling of comments, escapes, quantifiers, character classes, named groups, and scoped options.
  • Corrected jump and lookaround positioning after pattern processing.
  • Improved error messages to reference the original regular expression accurately.
  • Preserved source comments and newlines when recompiling extended regular expressions.

• Tests

  • Added regression coverage for extended-mode parsing, escapes, Unicode patterns, POSIX brackets, character classes, and error reporting.