Share the UTF-8 decoder between core, mruby-regexp and mruby-string-ext - #7128

The tree assembles a codepoint from UTF-8 bytes in two places, and the two are the same function: mrb_re_utf8_decode() in mruby-regexp and utf8code() in mruby-string-ext each measure the character with mrb_utf8len() and then fold the same masks over the same bytes (the 0xff >> (len + 1) one spells is the 0x1f, 0x0f, 0x07 the other names). They differ only in what an invalid sequence becomes: the engine matches the lead byte as a byte, String#ord and String#codepoints raise ArgumentError.

This is the decoder counterpart of #7094 (the encoder) and #7109 (the scanner and character heads): core gains the one missing piece and both gems call it. mruby-pack's unpack("U") decoder is left alone deliberately, since it implements CRuby's older, laxer dialect (surrogates pass; malformed and redundant are distinct errors), not the RFC 3629 reading mrb_utf8len() enforces.

What core gains

uint32_t mrb_utf8_decode(const char *p, const char *e, mrb_int *lenp);

The engine's body, moved beside the mrb_utf8len() it reads through, behind the same MRB_UTF8_STRING || MRB_UTF8_SCAN guard and declared next to it in internal.h. A sequence the scanner rejects comes back as its lead byte over one byte, so a value of 0x80 or above beside a length of 1 marks an invalid sequence, and whether that is an error stays the caller's question.

What the gems drop

• mruby-regexp deletes mrb_re_utf8_decode(). The compiler calls core directly where it reads a class atom or folds a character the pattern spells out; the executor keeps mrb_re_decode_char(), which now narrows the length back to the engine's int at the same seam where mrb_re_charlen() already does. re_utf8.c keeps the case folding and the \w test.
• mruby-string-ext's utf8code() keeps only what is String's to decide, the ArgumentError. Its callers, String#ord and String#codepoints, are untouched.

Equivalence

Checked by brute force rather than argued: both current implementations against the new function over 21,233,664 cases (both lead bytes exhaustively, nine representative tail bytes in each of the last two positions, every available length from 1 to 4), with no difference in the codepoint, the byte length, or the error condition.

Size

text grows slightly instead of shrinking: 1,721,815 → 1,722,415 on the default gembox, 1,812,452 → 1,812,996 on a full-core host build. The decoder now lives in the TU whose scanner it calls, so the compiler inlines mrb_utf8len() into it (nm puts the new function at 494 bytes where the engine's was 163); the bytes buy back the call and drop the duplicate source.

Verified

• rake test on the default gembox, where the engine reads UTF-8 without MRB_UTF8_STRING: 2037 tests, all green on each commit
• MRUBY_CONFIG=ci/gcc-clang rake test (full-core: MRB_UTF8_STRING everywhere, MRB_REGEXP_UNICODE_CASE on full-debug, and the C++ cxx_abi build): all green
• MRUBY_CONFIG=minimal rake builds, where neither define exists and the decoder compiles out with the rest of the scan
• prek run --all-files passes, except that markdownlint could not install locally (npm engine mismatch); no Markdown is touched here

Summary by CodeRabbit

• New Features
  • Improved UTF-8 decoding across strings and regular expressions.
  • Added consistent handling for valid, invalid, and truncated UTF-8 sequences.
• Bug Fixes
  • Improved UTF-8 character length reporting and error handling in string operations.
  • Standardized Unicode character decoding during regular-expression processing.