Skip to content

mruby-regexp: keep a character class open past [:name:] and a leading ] - #7041

Merged
matz merged 1 commit into
mruby:masterfrom
takumin:regexp-extended-charclass-terminator
Aug 9, 2026
Merged

mruby-regexp: keep a character class open past [:name:] and a leading ]#7041
matz merged 1 commit into
mruby:masterfrom
takumin:regexp-extended-charclass-terminator

Conversation

@takumin

@takumin takumin commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

The /x preprocessing pass ends a character class at the first ] it sees, so
class content written after a POSIX bracket or after a leading literal ] is
treated as free-spacing text and stripped.

strip_extended() clears its in_class flag on any ]. compile_charclass()
does not: the ] that closes [:alpha:] is consumed inside the class, and a
] written first in a class is a literal member. The two disagree, so the pass
hands the parser a pattern the user did not write, and the parser compiles it
without complaint.

Regexp.new("[[:alpha:] ]", Regexp::EXTENDED) =~ " "
# CRuby: 0
# mruby: nil     the space is stripped, leaving /[[:alpha:]]/

A # after the bracket starts a line comment, so the rest of the class is lost
rather than a single character.

Regexp.new("[[:alpha:]#x]", Regexp::EXTENDED)
# CRuby: compiles, matches "#"
# mruby: RegexpError (unterminated character class: /[[:alpha:]#x]/)

The literal ] diverges in both directions: one drops a character the class was
meant to contain, the other keeps one it was meant to exclude.

Regexp.new("[] ]", Regexp::EXTENDED) =~ " "
# CRuby: 0
# mruby: nil

Regexp.new("[^] ]", Regexp::EXTENDED) =~ " "
# CRuby: nil
# mruby: 0

Only /x patterns are affected. mrb_re_compile() runs the pass only when
RE_FLAG_EXTENDED is set, so an ordinary pattern never reaches it.

Fix

Both additions are written to agree with compile_charclass() rather than to
re-derive character class rules.

  • Inside a class, a POSIX bracket is copied as a unit. On [ followed by :,
    scan to the first : or ]; if the terminator is :], copy through it
    without touching in_class. Otherwise fall through and copy the [ as an
    ordinary member, which is the same outcome the parser reaches when it resets
    to its saved position.
  • On entering a class, an optional ^ and then an optional ] are copied
    before returning to the loop, mirroring the first flag in
    compile_charclass().

Both sit after the backslash pass-through, which stays first, so \] inside a
class keeps its existing handling and never reaches either branch. The pass
keeps its single in_class flag and its output buffer: neither addition emits
more bytes than it consumes, so the buffer bound is unchanged.

Behaviour

Measured against CRuby 4.0.6.

Pattern (Regexp::EXTENDED) CRuby before after
[[:alpha:] ] =~ " " 0 nil 0
[[:digit:] ]+ over " 1 " " 1 " "1" " 1 "
[[:alpha:]#x] =~ "#" 0 RegexpError 0
[] ] =~ " " 0 nil 0
[^] ] =~ " " nil 0 nil

CRuby warns character class has ']' without escape for the last two, and its
parser rejects the literal form /[]]/ outright, but Regexp.new accepts the
string form and reads the ] as a literal, which is what mruby's parser does
too.

Tests

The four cases above are added to assert("Regexp extended mode (x flag)") in
mrbgems/mruby-regexp/test/regexp.rb, next to the existing case for whitespace
inside a character class. The block's other assertions cover the regression
side: free-spacing outside a class must keep working.

rake test passes.

Summary by CodeRabbit

  • Bug Fixes

    • Improved extended-mode regular expressions to correctly handle POSIX character classes such as [:alpha:].
    • Preserved spaces, #, and literal closing brackets inside character classes.
    • Ensured quantifiers and negated character classes behave correctly in these cases.
  • Tests

    • Added coverage for POSIX classes and extended character-class parsing scenarios.

…ng `]`

`strip_extended()` ends a character class at the first `]` it sees, but
`compile_charclass()` does not. The `]` that closes a POSIX bracket is
consumed inside the class, and a `]` written first in a class is a literal
member. Under `/x` the pass therefore leaves the class early and applies
free-spacing to bytes the parser then reads as class members.

```ruby
Regexp.new("[[:alpha:] ]", Regexp::EXTENDED) =~ " "
# CRuby: 0
# mruby: nil
```

A `#` after the bracket costs more than a single character, because it
starts a line comment and the rest of the class goes with it.

```ruby
Regexp.new("[[:alpha:]#x]", Regexp::EXTENDED)
# CRuby: compiles, matches "#"
# mruby: RegexpError (unterminated character class)
```

The literal `]` diverges in both directions.

```ruby
Regexp.new("[] ]", Regexp::EXTENDED) =~ " "
# CRuby: 0
# mruby: nil

Regexp.new("[^] ]", Regexp::EXTENDED) =~ " "
# CRuby: nil
# mruby: 0
```

Teach the pass the two rules `compile_charclass()` already follows: copy a
POSIX bracket as a unit without touching `in_class`, and copy an optional
`^` and an optional `]` on entering a class. Both branches sit after the
backslash pass-through, so `\]` inside a class keeps its existing handling.
Only `/x` patterns reach the pass, so no other pattern changes.
@takumin
takumin requested a review from matz as a code owner August 9, 2026 13:04
@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f3e296e7-94d0-4816-8792-48f5f59035c6

📥 Commits

Reviewing files that changed from the base of the PR and between 9a3d566 and daa988b.

📒 Files selected for processing (2)
  • mrbgems/mruby-regexp/src/re_compile.c
  • mrbgems/mruby-regexp/test/regexp.rb

📝 Walkthrough

Walkthrough

Extended-mode regexp preprocessing now keeps POSIX bracket expressions and initially literal ] characters inside character classes. Tests cover positive and negated classes, spaces, #, and quantifiers.

Changes

Regexp character-class parsing

Layer / File(s) Summary
Preserve character-class members
mrbgems/mruby-regexp/src/re_compile.c, mrbgems/mruby-regexp/test/regexp.rb
strip_extended preserves POSIX bracket expressions and an initially literal ], including after ^. Extended-mode tests cover class members, quantifiers, and negated classes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • mruby/mruby#7024: Both changes modify POSIX bracket-class handling and related tests in re_compile.c.
  • mruby/mruby#7031: Both changes modify extended-mode regexp preprocessing and add related tests.

Suggested labels: mrbgems

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main fix for POSIX bracket expressions and leading literal ] characters in regexp character classes.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants