mruby-regexp: reject overlong and out-of-range UTF-8 sequences - #7068

mrb_re_utf8_charlen() checks that a sequence has the right number of continuation bytes and that they are shaped 10xxxxxx, but not that the value they encode needed that many bytes. mrb_re_utf8_decode() then assembles the codepoint from whatever it was handed. An overlong encoding therefore decodes to the character it only pretends to be, which leaves the two paths through the engine disagreeing about the same subject: a character class matches it, the identical literal does not.

"\xC0\xBC" =~ /[<]/            # before: 0,        after: nil
"\xC0\xBC" =~ /</              # before: nil,      after: nil
"\xC0\xBC".gsub(/[<]/, "&lt;") # before: "&lt;",   after: "\xC0\xBC"
"\xE0\x80\xBC" =~ /[<]/        # before: 0,        after: nil

Regexp.new("[Ā]").match?("\xE0\x84\x80")  # before: true,  after: false
/Ā/.match?("\xE0\x84\x80")                # before: false, after: false

C0 BC is the two-byte overlong spelling of <, and E0 84 80 the three-byte spelling of Ā. CRuby raises ArgumentError (invalid byte sequence in UTF-8) on every one of these, since it refuses to match against a string that is not valid in its encoding at all.

Escaping code is where this shows. s.gsub(/[<>&]/, ...) and s.gsub(/</, ...) are interchangeable in every reading of the source, and here they are not: the class form rewrites the overlong bytes, the literal form leaves them. [^<] sides with the class, reading the same bytes as the < it excludes and passing them through, so it parts from the literal too. Whichever direction a filter needs, it cannot get it consistently from the same subject.

Cause

mrb_re_utf8_charlen() derives a length from the lead byte and validates the continuation bytes. It has no minimum-value rule, so C0 and C1, which can only ever start an overlong two-byte sequence, are accepted as leads, and E0 8x, F0 8x and the surrogate range ED Ax are accepted too. mrb_re_utf8_decode() then shifts the payload bits together and returns 0x3C for C0 BC.

The two paths part company after that:

• A class compares the decoded codepoint, class_match(), so the overlong form hits the same bitmap bit as the real character.
• A literal compares bytes, RE_CHAR, so it never matches a spelling other than the one in the pattern.

Neither is wrong on its own terms. What is wrong is that the decoder promises a codepoint and delivers one for input that encodes none.

read_class_atom() decodes the pattern side through the same helper, so an overlong sequence written into a pattern currently joins the class as the character it imitates: Regexp.new("[\xC0\xBC]").match?("<") is true.

Fix

Reject the overlong, surrogate and out-of-range forms in mrb_re_utf8_charlen(), returning 1 the way it already does for a truncated sequence and a bad continuation byte. Three rules cover it:

• a lead of C0 or C1 is never valid;
• a three-byte sequence needs the codepoint at or above 0x800, which rules out E0 80 through E0 9F, and outside the surrogate range D800 to DFFF, which rules out ED A0 through ED BF;
• a four-byte sequence needs the codepoint at or above 0x10000 and at most 0x10FFFF, which rules out F0 80 through F0 8F and anything above F4 8F.

All three are decidable from the lead byte together with the first continuation byte, so the check costs one range comparison ahead of the existing loop and needs no second pass. The function's own comment already described the shape of the contract; this extends it from well-formedness to validity, and the invalid bytes fall back to the byte-at-a-time treatment every other malformed input gets. Everything else decodes through this one function, mrb_re_utf8_decode() for the length and both engines through mrb_re_charlen() and mrb_re_decode_char(), so the pattern side is covered by the same change, with the pattern byte then landing in the class as its raw value.

Deliberately not proposed: raising on an invalid subject the way CRuby does. That would mean validating the whole string on every match, which is not what a byte-oriented mruby String can afford, and the fallback already gives a defined answer.

Tests

mrbgems/mruby-regexp/test/regexp.rb gains Regexp - overlong UTF-8 is not the character it spells, next to the two blocks that already pin what malformed UTF-8 does here. The class and the literal are asserted in the same block, since the point is that they now agree. The block also pins the surrogate and above-U+10FFFF forms as byte-at-a-time, and the shortest valid spelling on each side of every new bound (C2 80, E0 A0 80, ED 9F BF, EE 80 80, F0 90 80 80, F4 8F BF BF) as one character, so the tightened ranges cannot drift.

rake test passes: 2009 assertions, 0 failures.

Summary by CodeRabbit

• Bug Fixes

  • Improved UTF-8 validation in regular expressions.
  • Invalid sequences, including overlong encodings, surrogate values, and out-of-range code points, are now handled correctly.
  • Valid boundary code points continue to match as single characters.

• Tests

  • Added regression coverage for invalid and valid UTF-8 sequences across literals, character classes, and wildcard matching.