vm.c: restore the GC arena in the allocating inline opcodes - #7022

The bug

OP_GETIDX, OP_GETIDX0, OP_SETIDX and OP_MATH answer from C without going through a method call, so the arena shrink that every cfunc return performs never runs for them. Five of their branches allocate and leave the result in the GC arena, where it stays until the enclosing method returns.

The Array branches return an element that already exists or write through an existing buffer, so they allocate nothing and are left alone. A Hash with a default value rather than a default proc allocates nothing either, and an Integer key needs no dup, which is why the plain shapes look clean.

What it costs

A loop whose body is built only from opcodes that do not restore pins one object per iteration, and the mark phase then walks all of them, so the loop is quadratic. GC.stat[:live] shows the retention directly, at n = 20000:

The signal is O(n) and the residue is O(1). In time, with while i < n; s[1]; i += 1; end:

and with x = 4611686018427387904; while i < n; x + x; i += 1; end:

Under MRB_GC_FIXED_ARENA they are a hard error

build_config/ci/gcc-clang.rb and build_config/ci/msvc.rb define MRB_GC_FIXED_ARENA, which caps the arena instead of growing it. In such a build none of the five loops is merely slow:

$ ./build/fixedarena/bin/mruby -e 's="hello"; i=0; while i < 100000; s[1]; i+=1; end'
-e:1: arena overflow error (NoMemoryError)
$ ./build/fixedarena/bin/mruby -e 'x=4611686018427387904; i=0; while i < 100000; x+x; i+=1; end'
-e:1: arena overflow error (NoMemoryError)
$ ./build/fixedarena/bin/mruby -e 'h=Hash.new { Object.new }; i=0; while i < 100000; h[1]; i+=1; end'
-e:1:in allocate: NoMemoryError
$ ./build/fixedarena/bin/mruby -e 'h=Hash.new { Object.new }; i=0; while i < 100000; h[0]; i+=1; end'
-e:1:in allocate: NoMemoryError
$ ./build/fixedarena/bin/mruby -e 'h={}; k="a"; i=0; while i < 100000; h[k]=1; i+=1; end'
-e:1: arena overflow error (NoMemoryError)

All five raise after roughly 95 iterations, which is MRB_GC_ARENA_SIZE less what the setup already holds. The two default-proc loops report from allocate rather than with the arena overflow error message, since the arena is full when the proc allocates rather than when the opcode stores. All five print ok with the restores in place.

Why this has gone unnoticed

The arena is restored to the baseline mrb_vm_exec() saves on entry, and several inline opcodes already restore to it after allocating: OP_STRING, OP_ARRAY, OP_ARRAY2, OP_ARYCAT, OP_ARYSPLAT, OP_HASH, and the String branch of OP_MATH a few lines above the macro this changes. So anything in the loop body that either sends or is one of those opcodes empties what the leaking opcode accumulated, and the loop is linear again, at n = 20000:

A loop that indexes a string almost always does something with the result, and doing almost anything with it is a send. What is left is narrow but real, and the memory consequence needs no pathological loop: until the enclosing method returns, every substring is pinned, so a long loop holds the whole run of them live at once even though the program keeps none.

The fix

Restore the arena around each allocating branch, exactly as the epilogues and the neighbouring opcodes do. The results need no mrb_gc_protect(): they are stored into regs[], and the VM stack is a GC root, which is why the cfunc epilogues can shrink unconditionally too.

The tests

Five assertions at the end of test/t/gc.rb, which already reads GC.stat[:live]. GC.stat is a cfunc, so it runs before the epilogue that would shrink the arena and still sees what the loop pinned. The threshold is n / 4, which leaves a factor of 4 below the signal and about 5 above the largest residue measured; a tighter one is a trap, since GC.interval_ratio = 1000 alone takes the residue to 996.

The OP_ADD assertion is guarded rather than tested for. Without mruby-bigint the overflow raises RangeError instead of promoting, so the guard skips; the shift count is held in a variable because a constant shift is folded at compile time, and a folded result out of mrb_int range makes the build fail rather than raise. It then runs over both 1 << 30 and 1 << 62, so that whichever width the build has, one of the two takes the overflow branch.

Checked configurations

All at mruby/mruby@fd6a18284, with the change applied and reverted.

The MRB_INT32 rows are what the 1 << 30 case is for: the OP_ADD assertion fails there without the change, so it covers the overflow branch on that width too rather than passing vacuously.

Summary by CodeRabbit

• Bug Fixes

  • Improved memory management during optimized indexing, hash operations, string access, and large-integer arithmetic.
  • Reduced unnecessary retention of temporary objects during repeated operations.

• Tests

  • Added regression coverage for memory growth across indexing, hash defaults, key assignment, and overflowing integer calculations.
  • Added compatibility handling for environments without large-integer support.