mruby-regexp: a byte that starts no character is a byte in a class - #7078

A character class read a pattern byte that starts no whole character as the
codepoint its number spells, while the literal path read the same byte as
itself. The two halves of one pattern disagreed about what the pattern holds.

mu = "\xC2\xB5"                     # U+00B5 MICRO SIGN, two bytes

mu =~ Regexp.new("[\xB5]")          # mruby: 0,          CRuby: RegexpError
mu =~ Regexp.new("\xB5")            # mruby: nil,        CRuby: RegexpError
mu.gsub(Regexp.new("[\xB5]"), "!")  # mruby: "!",        CRuby: RegexpError
mu.gsub(Regexp.new("\xB5"), "!")    # mruby: "\xC2\xB5", CRuby: RegexpError

read_class_atom() returned anything below 0xC0 without consulting the
decoder, so a lone continuation byte was a codepoint. The literal path asks the
decoder: emit_char_folded() stands aside when the decode consumed one byte,
and emit_char_bytes() emits the byte.

The same split reached case folding. A class refuses a codepoint the build
cannot fold, and 0xB5 was a codepoint to the class path, so the refusal fired
on a pattern holding no character at all. The first range in re_cased.h is
U+00B5 to U+017E, so of the 64 continuation bytes, 0xB5 to 0xBF were
refused inside a class and none was refused as a literal. A byte range is how a
continuation byte gets spelled, so /i turned a working scan into a compile
error:

# build without MRB_REGEXP_UNICODE_CASE
data = "\xC2\xB5A\xCE\xBC".b
data.scan(Regexp.new("[\x80-\xBF]")).size                      # 2
data.scan(Regexp.new("[\x80-\xBF]", Regexp::IGNORECASE)).size  # RegexpError

With MRB_REGEXP_UNICODE_CASE the class folded instead, so the byte 0xB5
reached the two Greek letters U+00B5 pairs with, and a negated class turned
away a character the pattern says nothing about:

# build with MRB_REGEXP_UNICODE_CASE
"μ" =~ Regexp.new("[\xB5]", Regexp::IGNORECASE)   # 0,   CRuby: RegexpError
"μ" =~ Regexp.new("[^\xB5]", Regexp::IGNORECASE)  # nil, CRuby: RegexpError

What this does

read_class_atom() now runs the decode the literal path runs and reports
whether the atom is a byte or a codepoint. Byte members live in the range list
the codepoints already use, tagged with RE_CLASS_BYTE, and class_match()
reads the tagged half when the subject position holds a byte that starts no
whole character, which is every position of a byte-indexed subject. The tag
carries what the number cannot: over U+0080 to U+00FF the byte 0xB5 and the
character U+00B5 both arrive as 0xB5.

Folding steps over the tagged ranges, because a byte that starts no character
has no case. That is what retires the /i refusal and the folding above.

CRuby settles the whole question with the pattern's encoding and raises
RegexpError: invalid multibyte character for a UTF-8 pattern holding such a
byte, then compiles and matches bytes for the ASCII-8BIT spelling of it. This
gem has no encoding to consult, so it reads the byte as the byte on both sides,
which gives the two halves of a pattern one rule.

Behaviour changes

• [\xB5] no longer matches "µ" (C2 B5), matching what \xB5 has always
  done outside a class.
• [\x80-\xBF] is a class of bytes rather than of U+0080 to U+00BF. It matches
  a byte-indexed subject where it used to match those characters.
• A range whose ends are a byte and a character ([\x80-µ], [\u{B5}-\xBF])
  names neither and raises RegexpError.
• /i no longer refuses a class of continuation bytes on a build without
  MRB_REGEXP_UNICODE_CASE, and no longer folds one on a build with it.

Two existing tests cover this ground. "overlong UTF-8 is not the character it
spells" asserts the same direction and is unchanged: [\xC0\xBC] does not
match "<". "an attempt in flight opens no match position inside a character"
asserted the subject side of the same collision, where [µ] matched a lone
0xB5; it now asks a class that holds the byte, and asserts that [µ] does
not match it.

Rebased onto #7074

This started out beside #7074, which touched the same lines, and is now rebased
onto it. read_class_atom() reads (c, cc, &is_byte) and class_add_member()
carries the byte tag; \u passes FALSE, since it names a codepoint outright.

That gives the two escapes one member each and a way to ask for either:

"µ" =~ Regexp.new("[\\u{B5}]")  # 0,   the character
"µ" =~ Regexp.new("[\\xB5]")    # nil, the byte

A range still has to pick one, so [\u{B5}-\xBF] is refused with the rest.

Testing

rake test is green with and without MRB_REGEXP_UNICODE_CASE, on full-core
with gcc on x86_64-linux.

A sweep over all 128 non-ASCII byte values, with and without /i, against
twelve subjects (decoded and byte-indexed, valid and invalid) now finds the
class and the literal spelling of each byte in agreement everywhere. Before
this change the same sweep found seven disagreements and, on a build without
MRB_REGEXP_UNICODE_CASE, refused 75 of the 128 bytes inside a class while
refusing none as a literal.

Summary by CodeRabbit

• New Features

  • Improved regular expression support for byte-oriented character classes.
  • Added handling for raw and invalid UTF-8 bytes, including byte literals and ranges.
  • Preserved correct case-insensitive behavior without treating raw bytes as Unicode characters.

• Bug Fixes

  • Invalid and overlong UTF-8 sequences are handled consistently.
  • Mixed byte and non-ASCII character ranges now raise appropriate errors.

• Documentation

  • Expanded guidance on byte-oriented patterns and UTF-8 boundaries.