mruby-regexp: support (?#...) comment groups - #7055

(?#...) is a comment group: Ruby drops it from the pattern and compiles what is
left. The (? dispatch in compile_atom() recognises (?:, (?=, (?!,
(?<=, (?<!, (?<name> and the inline options, but has no # branch, so a
pattern holding a comment group falls through to undefined (?...) sequence and
does not compile at all.

/a(?#note)b/.match?("ab")
# CRuby: true
# mruby: RegexpError (undefined (?...) sequence: /a(?#note)b/)

Regexp.new("a(?#x\\)y)b").match?("ab")
# CRuby: true
# mruby: RegexpError (undefined (?...) sequence: /a(?#x\)y)b/)

Regexp.new("(a(?#c)b)").match("ab").to_a
# CRuby: ["ab", "ab"]
# mruby: RegexpError (undefined (?...) sequence: /(a(?#c)b)/)

The group is not an atom, so a quantifier written after it repeats what came
before it:

Regexp.new("a(?#x)*") =~ "aaa"
# CRuby: 0
# mruby: RegexpError (undefined (?...) sequence: /a(?#x)*/)

Under /x the same pattern fails for a second reason. Extended mode is
preprocessed over the whole pattern before the parser runs, and its # branch
skips to the end of the line, so a (?#note) b loses #note) b and leaves a
dangling (? at the end of the stripped buffer. A fix confined to
compile_atom() would leave this half broken, and comment groups written
alongside /x are where they are most common.

Regexp.new("a (?#note) b", Regexp::EXTENDED).match?("ab")
# CRuby: true
# mruby: RegexpError (target of repeat operator is not specified: /a (?#note) b/)

Fix

Remove the group in the preprocessing pass rather than in the parser.

That is what lets a(?#x)* compile as a*: the group has to be gone before the
quantifier is parsed. An atom is too late. compile_quantified() returns early
when compile_atom() emitted no instruction, so a comment group handled there
would leave the * with no target, and an atom that did emit one would change
what the pattern matches.

strip_extended() becomes preprocess_pattern() and takes the /x behaviour
as a flag. Comment group removal runs unconditionally, whitespace and #
line-comment stripping only when RE_FLAG_EXTENDED is set. mrb_re_compile()
now enters the pass when either the flag is set or has_comment_group() finds
(?# in the pattern, so an ordinary pattern without one still skips the pass
and its mrb_malloc(). That gate is a memchr() scan for (.

The new branch sits after the backslash pass-through and after the character
class branch, so the two ways of writing those bytes without meaning a comment
group keep the behaviour they have now:

Regexp.new("a[(?#c)]b").match?("a#b")  # both: true, class member
Regexp.new("a\\(?#note)b")             # both: RegexpError, escaped '('

The group ends at the first ) not preceded by a backslash, and it does not
nest, so x(?#a(?#b))y closes at the first ) and reports the second as
unmatched, as CRuby does.

An unterminated group is copied through the pass instead of being dropped,
which is what the new # branch in compile_atom() is for: reaching it means
the group was never closed, and it raises rather than letting the rest of the
pattern be swallowed silently.

Regexp.new("a(?#note")
# CRuby: RegexpError (end pattern in group)
# this PR: RegexpError (unterminated comment group)

Error messages still quote the pattern as written, since compile_error()
already quotes c->orig rather than the preprocessed buffer.

Scope

Nothing outside the pattern compiler changes. The literal form /a(?#note)b/
already passed through the parser without complaint, which is why the
reproduction raises RegexpError rather than a parse-time error, so
mruby-compiler needs no change. regexp.c is not involved either, since the
failure happens during compilation before any Regexp method runs.

(?~...) and (?(...) remain unimplemented and still raise from the same
catch-all.

Tests

mrbgems/mruby-regexp/test/regexp.rb:

• Regexp - comment groups (?#...), a new block next to
  Regexp - inline options (?i) / (?i:...), the nearest coverage of the same
  dispatch chain: the literal and constructor forms, leading, trailing and
  empty comments, a newline inside one, a comment inside a capture group, the
  quantifier target case, \) inside a comment and the escaped backslash that
  ends one early, the non-nesting case, the unterminated case with its message,
  the character class member, and the escaped (.
• Regexp extended mode (x flag): a comment group under /x, one followed by
  an ordinary # line comment on the same line, and the unterminated case.

Verified on x86_64-linux:

• Every reproduction above now agrees with CRuby 4.0.6, error message wording
  aside.
• rake test: 1976 total, 1958 OK, 0 KO, 0 crash, and bintest 105 OK.
• An MRB_INT32 build with clang and -Wall -Wextra: 1878 total, 1860 OK,
  0 KO, 0 crash, and no new warning from any mruby-regexp file.
• build_config/clang-asan.rb: 2153 total, 2143 OK, 0 KO, 0 crash, with no
  leak or invalid access reported.

Summary by CodeRabbit

• New Features

  • Added support for (?#...) comment groups in regular expressions.
  • Documented regular-expression comment-group syntax.
  • Comment groups now work with extended-mode patterns and character classes.

• Bug Fixes

  • Unterminated comment groups now produce a specific parsing error.
  • Improved handling of escaped syntax, nesting, and quantifier interactions.