mruby-pack: check the pack("U") range before the cast to uint32_t - #7114

[(1 << 32) + 0x41].pack("U") packs "A" on a build with a 64 bit
mrb_int.

pack_utf8() hands its argument to mrb_utf8_to_buf() as a uint32_t
and treats a return of 0 as out of range:

uint32_t c = (uint32_t)mrb_integer(o);

len = (int)mrb_utf8_to_buf(utf8, c);
if (len == 0) {
  mrb_raise(mrb, E_RANGE_ERROR, "pack(U): value out of range");
}

The cast wraps, so the encoder never sees the value that was passed in and
answers with a perfectly good byte count:

# mruby
[(1 << 32) + 0x41].pack("U").bytes      #=> [65]
[(1 << 32) + 0x110000].pack("U")        # RangeError, but only because it wraps to 0x110000

# CRuby
[(1 << 32) + 0x41].pack("U")            # RangeError: pack(U): value out of range

Fix

Check c < 0 || 0x10FFFF < c on the mrb_int before the cast and raise
the same RangeError with the same message. int_chr_utf8() in
mruby-string-ext guards the shared encoder the same way. With the range
checked, mrb_utf8_to_buf() cannot answer 0, so the test on its return
value is dropped.

A negative argument raised before this too, but by wrapping to 0xFFFFFFFF
and landing above U+10FFFF, not because its sign was ever examined. The
result is unchanged and now follows from the check.

Scope

The accepted range stays at U+10FFFF. CRuby reaches 0x7FFFFFFF using the 5
and 6 byte sequences of the pre RFC 3629 encoding:

# CRuby
[0x200000].pack("U").bytes    #=> [248, 136, 128, 128, 128]
[0x7FFFFFFF].pack("U").bytes  #=> [253, 191, 191, 191, 191, 191]

mruby's unpack("U") rejects those sequences as malformed, so widening
pack("U") alone would leave the pair inconsistent, and widening both
would put the core UTF-8 helpers back outside RFC 3629. That is a separate
decision from this bug, so the range difference is left as it is.

Related: #7113 fixes the same wrapping cast in sprintf("%c"), where the
symptom was sharper because the encoder writes nothing to its buffer for a
value it rejects.

Tests

A new case in mrbgems/mruby-pack/test/pack.rb covers U+10FFFF, the first
value above it, and the wrapping argument. The shift is computed at run time
rather than written as a literal, because on a build with a 32 bit mrb_int
and no bigint the constant folder rejects 1 << 32 while the file is being
compiled and the whole test file is dropped without a word. On such a build
the value cannot be constructed at all, so the assertion is skipped.

Suites pass:

• full-core with MRB_UTF8_STRING: 2250 tests, 0 failures
• default build: 2059 tests plus 105 bintests, 0 failures
• MRB_INT32 without bigint: 1504 tests, 0 failures. With the literal form
  it is 1456, the difference being every mruby-pack test.

Summary by CodeRabbit

• Bug Fixes

  • Fixed Unicode packing to reject negative values and code points above the Unicode maximum.
  • Prevented invalid values from being converted into seemingly valid Unicode characters.
  • Added clear range errors for out-of-range input.

• Tests

  • Added regression coverage for the maximum valid code point and invalid boundary and overflow values.
  • Confirmed valid Unicode boundaries continue to pack correctly.