vm.c: restore the GC arena where the Integer boxing macro allocates - #7042

Follow-up to 65bc860, which restored the GC arena in the five inline opcode
branches that allocate by calling an allocating C function. This covers the
sites of the other shape, where the allocation is inside the boxing macro and
there is no call to bracket.

The cause

An inline opcode answers from C without going through a method call, so nothing
runs the cfunc epilogue that shrinks the arena (src/vm.c:2263 and :2971).
Whatever such an opcode allocates stays pinned until the enclosing method
returns or some other call shrinks the arena back to the baseline
mrb_vm_exec() saved on entry. A send anywhere in the loop body clears it, and
so does any inline opcode that restores on its own account, OP_STRING,
OP_ARRAY and OP_HASH among them, which is why this goes unnoticed: the body
has to be built entirely from opcodes that do not restore for it to show.

Under word boxing an mrb_int outside the fixnum range is a heap object, so
SET_INT_VALUE() allocates one. That is not a configuration to worry about at
the margin: include/mrbconf.h selects word boxing whenever none of the three
boxing modes is defined, so it is what an ordinary build gets. The fixnum range
ends at 2**62-1 there, well below the mrb_int overflow that reaches
OP_MATH_OVERFLOW_INT, so the ordinary integer path allocates long before the
BigInt path is considered and does so on builds without MRB_USE_BIGINT too.

SET_INT_VALUE() resolves to mrb_boxing_int_value(), which is compiled under
MRB_WORD_BOXING and under MRB_NAN_BOXING with MRB_INT64. The condition in
the fix is that disjunction rather than #ifdef MRB_WORD_BOXING because what it
keys on is whether the macro can allocate, not which boxing mode is in use.

The sites

• the non-overflow integer arm of the OP_MATH family, five occurrences of
  SET_INT_VALUE(mrb,regs[a], z). Three are live, reached from OP_ADD,
  OP_SUB, OP_MUL, OP_ADDI, OP_SUBI and the ILV forms; the two in
  OP_MATH_CASE_INTEGER and OP_MATHI_CASE_INTEGER sit in macros that are
  defined and never expanded. Converting all five keeps the family consistent
• vm_op_div(), whose mrb_div_int_value() call is a separate route with no
  ai in scope at all
• OP_LOADL, which boxes the pool entry afresh on every execution rather than
  handing back a stored object. The pool holds an integer as a raw i32 or
  i64 and a big integer as its digits; there is no mrb_value in it
• OP_LOADI32, which boxes the immediate the same way. It can only allocate on
  a 32-bit host under word boxing, where the fixnum range ends at 2**30

Measurements

Growth in GC.stat[:live] over a loop of n = 20000, on a default
build_config/default.rb build, with the change applied and then reverted:

The same on an i686 build, where mrb_int is 32 bits wide and the fixnum range
ends at 2**30, which is the only configuration in which OP_LOADI32 can
allocate:

The last three rows are the same binary and are what make the first five a
finding rather than a retest: adding a send to the loop body takes the growth
down to the same residue, which is the signature of arena retention rather than
of ordinary garbage.

The curve is quadratic, time roughly quadrupling per doubling:

The BigInt column stays heavier because the object is genuinely allocated on
every iteration there. What the change removes is the quadratic mark cost, not
the allocation. The timings are one run on one machine; the shape is the point.

Under MRB_GC_FIXED_ARENA they are a hard error

build_config/ci/gcc-clang.rb and build_config/ci/msvc.rb define
MRB_GC_FIXED_ARENA, which caps the arena instead of growing it. There the same
loops raise after 96 or 97 iterations, MRB_GC_ARENA_SIZE less what the setup
already holds:

$ ./build/fixedarena/bin/mruby -e 'x=4611686018427387903; i=0; while i < 100000; x+1; i+=1; end; puts "ok"'
trace (most recent call last):
-e:1: arena overflow error (NoMemoryError)
$ ./build/fixedarena/bin/mruby -e 'a=[1,2,3]; i=0; while i < 100000; a[1]; i+=1; end; puts "ok"'
ok

The last line is the same binary. That makes this a functional bug in a
supported configuration and not only a performance one. All four such loops
print ok with the change applied.

Why the restore is conditional on having allocated

VM_SET_INT_VALUE() skips the restore unless the store produced a heap object.
The obvious reason is that the integer arm of OP_MATH is the hottest path in
the VM and the fixnum case allocates nothing.

The other reason is that an unconditional restore quietly breaks the assertions
65bc860 added. i += 1 compiles to OP_ADDILV, so restoring there empties
the arena on every iteration of every while loop written that way. Measured on
a tree carrying this change with 65bc860bf's src/vm.c hunks reverted, so the
code those assertions guard is broken again:

Their threshold is 5000, so with an unconditional restore all five pass on a
tree where the code they cover is broken, and with the conditional one all five
fail, which is what they are supposed to do. The same trap applies to the new
assertions here: with an unconditional restore the OP_DIV loop reads 528 on a
tree whose vm_op_div() restore has been removed, against 20001 with the
conditional one.

The alternative worth weighing, if a conditional on that path is unwelcome, is
to put the restore inside SET_INT_VALUE() under word boxing instead. That
takes it off the opcode and onto every caller of the macro, including C code
that legitimately holds unrooted objects across the call, so it is offered as
the question rather than as the proposal.

Tests

Four assertions, all reading GC.stat[:live], which is a cfunc and so reports
the count before its own epilogue restores. The two arithmetic ones use
1 << shift with a variable shift for shift in 30, 31 and 62, straddling the
fixnum boundary of every mode that can allocate; a constant shift is folded at
compile time and a folded result out of mrb_int range makes the build fail
rather than raise. The OP_LOADI32 one uses 2**30, which fits in the operand
of that opcode rather than going to the pool and is representable on every
build, so it needs no guard; it is the assertion that only bites on a 32-bit
host.

The arithmetic loops do not include x += 1, which would be the third
expansion of the family, because on a build where x is a big integer that
compiles to an OP_ADDILV whose send fallback corrupts the frame. That is an
unrelated bug in the opcode and is being reported separately.

The OP_LOADL assertion is with mruby-bigint's tests rather than in
test/t/gc.rb, because the literal that reaches the opcode is wider than 32
bits and there is no portable way to write one: without the gem it is out of
mrb_int range on an MRB_INT32 build, and an unrepresentable literal is a
compile-time error rather than something a test can rescue, so test/t/gc.rb
would fail to load whole and every assertion in it would be silently lost. With
the gem the literal reaches IREP_TT_INT64 where mrb_int is 64 bits wide and
IREP_TT_BIGINT where it is not.

Without the change, a default mrbtest reports 3 KO and a MRB_GC_FIXED_ARENA
one reports 3 Crash on NoMemoryError; an i686 mrbtest reports 3 KO, the
OP_LOADI32 assertion among them. With the change all three report 0 KO and 0
Crash. Also run green: MRB_GC_STRESS with MRB_USE_DEBUG_HOOK (full-debug
from build_config/ci/gcc-clang.rb, where the new assertions add about 0.9 s),
and an MRB_INT32 build without mruby-bigint.

Summary by CodeRabbit

• Bug Fixes

  • Reduced unnecessary memory retention when processing large integer values.
  • Improved garbage collection behavior for integer arithmetic, division, and constant loading.
  • Preserved correct results while preventing excessive growth of retained objects.

• Tests

  • Added coverage for repeated large-integer operations and loads.
  • Added validation across multiple integer sizes and arithmetic operations.