mruby-regexp: read a digit escape as octal when it names no group - #7268

`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:

```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (target of repeat operator is not specified), mruby: 0
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
# CRuby: nil (`{1, 2}` is not an interval), mruby: 0
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
# CRuby: ["a b"], mruby: ["ab"]
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
# CRuby: 0 (`\k` is the letter and `<ab>` a literal), mruby: nil (it read `\k<ab>`)
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
# CRuby: nil (a vertical tab is not free-spacing whitespace), mruby: 0
```

The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.

The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.

One pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone; CRuby reads a `?` a blank away from the `*` as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses this one too rather than give it a meaning CRuby does not.

`preprocess_pattern()` removes `(?#...)` groups and, under `/x`, `#` comments, and copies the rest through, stepping over an escape as the backslash and one letter (`\u{...}` apart) so that a comment is not read out of one. Bytes it removed from inside a longer escape left the parser a different, valid escape. CRuby's pre-pass (`re.c`) has read the escape whole, and rejected it or written it at full width, before it reaches the comment:

```ruby
Regexp.new("\\u12(?#c)34") =~ "ሴ"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u1234`)
Regexp.new("\\u#c\n{61}", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u{61}`)
Regexp.new("\\x(?#c)61") =~ "a"
# CRuby: RegexpError (invalid hex escape), mruby: 0 (it read `\x61`)
Regexp.new("\\x6(?#c)1") =~ "a"
# CRuby: nil (`\x06` then `1`), mruby: 0 (it read `\x61`)
Regexp.new("\\0(?#c)61") =~ "1"
# CRuby: nil (`\0` then `61`), mruby: 0 (it read `\061`)
```

`skip_uninterpreted()` now steps over the escape at the width `re.c` reads it: `\u{...}` through its brace as before, `\uXXXX` and the next four bytes whatever they are, `\x` and its hex digits (with none, the one byte after, for the parser to reject), `\0` and its octal digits. What the pass copies is then what the parser reads, and `\u12(?` reaches `unicode_escape_first()` for it to reject. An escape that stops short of its full width (`\x6`, `\0`) is written at full width (`\x06`, `\000`) so that a digit the pass brings next cannot lengthen it; the rewrite buffer is twice the source again for that, the most an escape grows being `\0` to `\000`. `\1`-`\9` stay the backslash and the digit, and the digits after them plain bytes, as they are to `re.c`, so a removed comment joins them as it does there and the tests from mruby#7268 for `\1(?#c)0` still hold. `\p{...}` and `\g<...>` are not implemented by this engine, so there is nothing to protect.

Whitespace inside `\u{...}` (`\u{ 61 }`, `\u{61  62}`) is accepted by CRuby with or without `/x`, and it was already accepted here; the list is still copied through as one escape. Inside a character class the pass removes nothing, so `[\x6(?#c)1]` keeps the bytes of `(?#c)` as members and its escape as written.

`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:

```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
```

The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.

The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.

One shape of pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone, and `a{2} ?` the same way; CRuby reads a `?` a blank away from the quantifier as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses these too rather than give them a meaning CRuby does not.

`preprocess_pattern()` removes `(?#...)` groups and, under `/x`, `#` comments, and copies the rest through, stepping over an escape as the backslash and one letter (`\u{...}` apart) so that a comment is not read out of one. Bytes it removed from inside a longer escape left the parser a different, valid escape. CRuby's pre-pass (`re.c`) has read the escape whole, and rejected it or written it at full width, before it reaches the comment:

```ruby
Regexp.new("\\u12(?#c)34") =~ "ሴ"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u1234`)
Regexp.new("\\u#c\n{61}", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u{61}`)
Regexp.new("\\x(?#c)61") =~ "a"
# CRuby: RegexpError (invalid hex escape), mruby: 0 (it read `\x61`)
Regexp.new("\\x6(?#c)1") =~ "a"
# CRuby: nil (`\x06` then `1`), mruby: 0 (it read `\x61`)
Regexp.new("\\0(?#c)61") =~ "1"
# CRuby: nil (`\0` then `61`), mruby: 0 (it read `\061`)
```

`skip_uninterpreted()` now steps over the escape at the width `re.c` reads it: `\u{...}` through its brace as before, `\uXXXX` and the next four bytes whatever they are, `\x` and its hex digits (with none, the one byte after, for the parser to reject), `\0` and its octal digits. What the pass copies is then what the parser reads, and `\u12(?` reaches `unicode_escape_first()` for it to reject. An escape that stops short of its full width (`\x6`, `\0`) is written at full width (`\x06`, `\000`) so that a digit the pass brings next cannot lengthen it; the rewrite buffer is twice the source again for that, the most an escape grows being `\0` to `\000`. `\1`-`\9` stay the backslash and the digit, and the digits after them plain bytes, as they are to `re.c`, so a removed comment joins them as it does there and the tests from mruby#7268 for `\1(?#c)0` still hold. `\p{...}` and `\g<...>` are not implemented by this engine, so there is nothing to protect.

Whitespace inside `\u{...}` (`\u{ 61 }`, `\u{61  62}`) is accepted by CRuby with or without `/x`, and it was already accepted here; the list is still copied through as one escape. Inside a character class the pass removes nothing, so `[\x6(?#c)1]` keeps the bytes of `(?#c)` as members and its escape as written.

`preprocess_pattern()` removed the whitespace of `/x` from the source before the parser read it, so whatever the removal brought together was what the parser read. CRuby does this in the other layer: its tokenizer skips whitespace only where it fetches a token, and reads each token, an escape, a group name, an interval `{n,m}`, a `(?` opener, with the whitespace inside it in place. Removing the bytes instead means every kind of token needs a rule of its own to keep the two apart, the `(?:)` written between a digit escape and a digit (mruby#7268) being one, and the tokens without one still read differently:

```ruby
Regexp.new("( ?i)A", Regexp::EXTENDED) =~ "a"
Regexp.new("a{1, 2}", Regexp::EXTENDED) =~ "aa"
Regexp.new("(?<a b>x)", Regexp::EXTENDED).names
Regexp.new("(?<ab>x)\\k <ab>", Regexp::EXTENDED) =~ "xk<ab>"
Regexp.new("a\vb", Regexp::EXTENDED) =~ "ab"
```

The parser now skips the whitespace where the tokenizer does. `RE_FLAG_EXTENDED` was already carried in `c->flags` and scoped by the save and restore every group does for `(?i)` and `(?m)`; `skip_extended_space()` reads it at the two places a token can end: at the top of `compile_seq()`, before each atom and before the `|` or `)` that ends the sequence, and in `compile_quantified()` between the atom and its quantifier. Nowhere else, so `{1, 2}` is not an interval, `( ?` is a group and then a `?`, a group name keeps its blanks, and a numeric escape reads what stands after it: `\x6 1` is `\x06` and `1` because the hex digits stop at the space, with nothing written between them. The bytes skipped are the five Onigmo skips; the vertical tab the pass also removed is a literal under `/x` in CRuby and now here.

The pass keeps what CRuby's own pre-pass (`re.c`) removes before its tokenizer runs, `(?#...)` groups under any flags and `#` comments under `/x`, with the scope stack from mruby#7252 saying where `/x` is on. Both stay in the pass rather than moving to the parser because a removed comment does join the bytes on either side of it in CRuby, `\1(?#c)0` and `\1#c`, a newline and `0` both being `\10`; the tests from mruby#7268 that pin this are unchanged. The `(?:)` insertion goes with its `esc_end` and `blank_out` bookkeeping, and with it the reason the rewrite buffer was twice the source. `has_rewritten_group()`, which decided whether the pass runs at all, is replaced by a `memchr()` for `#`: both things the pass removes are spelled with one, and neither the `/x` flag nor a `(?x)` needs the pass any more.

One shape of pattern that compiled on master is refused now. `a* ?` under `/x` was read as the non-greedy `a*?` once the blank was gone, and `a{2} ?` the same way; CRuby reads a `?` a blank away from the quantifier as a repeat of the repeat, `(?:a*)?`, and this engine refuses a repeat of a repeat wherever it is written, `a**` included, so it refuses these too rather than give them a meaning CRuby does not.

`preprocess_pattern()` removes `(?#...)` groups and, under `/x`, `#` comments, and copies the rest through, stepping over an escape as the backslash and one letter (`\u{...}` apart) so that a comment is not read out of one. Bytes it removed from inside a longer escape left the parser a different, valid escape. CRuby's pre-pass (`re.c`) has read the escape whole, and rejected it or written it at full width, before it reaches the comment:

```ruby
Regexp.new("\\u12(?#c)34") =~ "ሴ"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u1234`)
Regexp.new("\\u#c\n{61}", Regexp::EXTENDED) =~ "a"
# CRuby: RegexpError (invalid Unicode escape), mruby: 0 (it read `\u{61}`)
Regexp.new("\\x(?#c)61") =~ "a"
# CRuby: RegexpError (invalid hex escape), mruby: 0 (it read `\x61`)
Regexp.new("\\x6(?#c)1") =~ "a"
# CRuby: nil (`\x06` then `1`), mruby: 0 (it read `\x61`)
Regexp.new("\\0(?#c)61") =~ "1"
# CRuby: nil (`\0` then `61`), mruby: 0 (it read `\061`)
```

`skip_uninterpreted()` now steps over the escape at the width `re.c` reads it: `\u{...}` through its brace as before, `\uXXXX` and the next four bytes whatever they are, `\x` and its hex digits (with none, the one byte after, for the parser to reject), `\0` and its octal digits. What the pass copies is then what the parser reads, and `\u12(?` reaches `unicode_escape_first()` for it to reject. An escape that stops short of its full width (`\x6`, `\0`) is written at full width (`\x06`, `\000`) so that a digit the pass brings next cannot lengthen it; the rewrite buffer is twice the source again for that, the most an escape grows being `\0` to `\000`. `\1`-`\9` stay the backslash and the digit, and the digits after them plain bytes, as they are to `re.c`, so a removed comment joins them as it does there and the tests from mruby#7268 for `\1(?#c)0` still hold. `\p{...}` and `\g<...>` are not implemented by this engine, so there is nothing to protect.

Whitespace inside `\u{...}` (`\u{ 61 }`, `\u{61  62}`) is accepted by CRuby with or without `/x`, and it was already accepted here; the list is still copied through as one escape. Inside a character class the pass removes nothing, so `[\x6(?#c)1]` keeps the bytes of `(?#c)` as members and its escape as written.