mruby-regexp: scope the inline x option the way i and m are scoped - #7252
Conversation
`Regexp#to_s` spells an extended pattern as `(?x-mi:...)`, and the parser
refused that form, so an extended Regexp could not recompile from its own
printed form and could not be interpolated into another pattern:
```ruby
Regexp.new(/a b/x.to_s) # CRuby: /(?x-mi:a b)/
# mruby: RegexpError, inline extended mode (?x) is not supported
```
Free-spacing is applied by `preprocess_pattern()` before the parser runs,
which is why the letter could not be honoured where the parser reads it: by
then the whitespace it governs is either gone or kept. So the pass now
tracks the option itself. It keeps one bit per open group, pushed at every
`(` it interprets and popped at the matching `)`, and reads the letters of
each `(?imx-imx)` and `(?imx-imx:` it passes: the toggle form switches x
for the rest of the enclosing group, the scoped form for its own body. That
is the scope the parser already gives i and m, and it is what Onigmo does
with the whitespace under `(?x)`. `parse_inline_flags()` then carries x
like the other two letters, and nothing in the parser reads the bit.
Two consequences follow. `(?-x:...)` inside a pattern that is itself
extended brings the whitespace back for its scope, where before the letter
was accepted and dropped. And a `#` comment or a run of whitespace ends
where the group that turned x on ends; CRuby's own pre-pass in `re.c`
strips `#` comments after a `(?x)` toggle to the end of the pattern while
Onigmo restores the whitespace at the group's `)`, and this pass follows
the group for both.
The pre-check that lets a plain pattern skip the pass and its allocation
now also fires on an option group that turns x on. The scope stack lives
behind the rewritten pattern in the same allocation, one bit per byte of
source, since a group opener is one byte.
The README's limitation entry goes, and the option groups join the list of
supported syntax.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Included review availability: Your plan includes up to 8 reviews per rolling hour; 6 remain after this review. 📝 WalkthroughWalkthroughThe regexp compiler now supports inline and scoped ChangesInline extended-mode regexp support
Estimated code review effort: 3 (Moderate) | ~25 minutes Merge Risk: ⚪ Minimal · up to The PR adds scoped extended-mode handling for regular expressions and documents broad validation, with no actionable merge-blocking risk remaining beyond normal checks and review. Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Pattern
participant preprocess_pattern
participant RegexpParser
Pattern->>preprocess_pattern: provide inline x groups
preprocess_pattern->>preprocess_pattern: apply nested scope and strip whitespace/comments
preprocess_pattern->>RegexpParser: pass rewritten pattern with option syntax
RegexpParser-->>Pattern: produce compiled regexp
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:
```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (target of repeat operator is not specified), mruby: 0
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
# CRuby: nil (`{1, 2}` is not an interval), mruby: 0
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
# CRuby: ["a b"], mruby: ["ab"]
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
# CRuby: 0 (`\k` is the letter and `<ab>` a literal), mruby: nil (it read `\k<ab>`)
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
# CRuby: nil (a vertical tab is not free-spacing whitespace), mruby: 0
```
The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.
The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.
One pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone; CRuby reads a `?` a blank away from the `*` as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses this one too rather than give it a meaning CRuby does not.
`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:
```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
```
The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.
The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.
One shape of pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone, and `a{2} ?` the same way; CRuby reads a `?` a blank away from the quantifier as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses these too rather than give them a meaning CRuby does not.
`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:
```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
```
The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.
The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.
One shape of pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone, and `a{2} ?` the same way; CRuby reads a `?` a blank away from the quantifier as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses these too rather than give them a meaning CRuby does not.
Regexp#to_sspells an extended pattern as(?x-mi:...), and the parserrefused that form. So an extended Regexp could not recompile from its own
printed form and could not be interpolated into another pattern:
Where x has to be resolved
Free-spacing is applied by
preprocess_pattern()before the parser runs: itcopies the pattern into a buffer with the whitespace and
#comments dropped.That is why
parse_inline_flags()could not honour the letter where it readsiandm: by the time the parser reaches a(?x, the whitespace it governsis either already gone or already kept.
So the pass now tracks the option itself. It keeps one bit per open group,
pushed at every
(it interprets and popped at the matching), and reads theletters of each
(?imx-imx)and(?imx-imx:it passes over: the toggle formswitches x for the rest of the enclosing group, the scoped form for its own
body. That is the scope the parser already gives
iandm, and it is whatOnigmo does with the whitespace under
(?x).parse_inline_flags()thencarries x as a bit like the other two letters, and nothing in the parser reads
it.
Two consequences follow.
(?-x:...)inside a pattern that is itself extended brings the whitespace backfor its scope. Before, the letter was accepted so that
Regexp#to_soutputwould recompile, and then dropped:
A
#comment or a run of whitespace ends where the group that turned x onends. CRuby is itself split on this: Onigmo restores the whitespace at the
group's
), so/((?x)a) b/matches"a b"there as here, but the pre-pass inre.c(unescape_nonascii0()) that strips#comments sets its extended flagfor the rest of the pattern on a toggle, so
/((?x)a)#c\nb/matches"ab"inCRuby and does not here. This pass follows the group for both.
The pre-check that lets a plain pattern skip the pass and its allocation now
also fires on an option group that turns x on, so a pattern without one costs
what it cost. The scope stack lives behind the rewritten pattern in the same
allocation, one bit per byte of source, since a group opener is one byte.
The README's limitation entry goes, and the option groups join the list of
supported syntax.
Size
.textofbin/mruby,build_config/ci/gcc-clang.rb, each side from a cleanbuild directory.
re_compile.ois the only object that changes.bintestascii-casebyte-stringcxx_abifull-debug(-O0)Verification
The tests are in the two files that already cover
Regexp#to_sand the inlineoptions. The
to_sblock gains the round trip that motivated this: the printed(?x-mi:a b)reads back as free-spacing, matches"ab"and not"a b", andinterpolates. The inline-options block replaces the two
assert_raisethatpinned the refusal with the toggle and scoped forms in plain, named,
non-capturing and lookahead groups,
xbesidei,(?-x)inside(?x),free-spacing following the scope through comments,
(?#groups, escapes andclasses, a comment swallowing a
)on its line as it does in CRuby, and-xbringing whitespace back inside an extended pattern.
Differential against CRuby 4.0.6. 57 hand-written patterns covering the
forms above and the malformed spellings, each under four flag sets and against
26 subjects: identical, except
(?-)and(?-x-x), which Onigmo accepts andthis gem refuses on master as after. Then 6,000 random patterns over two seeds,
well nested from plain, named, lookaround and option groups, toggles, escapes,
classes and runs of whitespace, each under
0andEXTENDEDagainst 30subjects: 12,000 lines, output byte-identical. Alternation is left out of the
generator, since the scope a toggle takes across
|differs from Onigmo foriandmalready. A third run of 3,000 patterns that adds#comments to thealphabet differs in 2 lines of 6,000, both a comment after the
)of the groupthat turned x on, the
re.cbehaviour described above.rake test,build_config/ci/gcc-clang.rb, no compiler warning:full-debugbintestbintest(bintest suite)cxx_abibyte-stringascii-caseThe default configuration: 2,124 total, 0 KO, 0 crash, plus its 112 bintests.
Environment
Details
Compile lines for
mrbgems/mruby-regexp/src/re_compile.cin the builds quotedabove, paths shortened:
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests