string.c: append into a shared buffer instead of copying it - #7039

Summary

Taking a substring longer than RSTRING_EMBED_LEN_MAX (27 bytes on a 64-bit build)
shares the parent's buffer, and str_share() trims the parent's spare capacity away when
it does. Every later append to the parent therefore has to unshare, which copies the
entire buffer. A loop that appends and looks at a slice does O(N) work per iteration and
O(N^2) in total, even when the slice is dropped immediately and nothing is retained.

An append only writes [len, len + addlen), a region no other holder of the buffer can
see, so that copy is the conservative "never write into a shared buffer" rule rather than
something correctness needs. This adds a reserved watermark to mrb_shared_string, the
offset past the last byte any sharer can see, stops trimming spare capacity on share, and
routes mrb_str_cat() through a new str_modify_cat() that appends in place when the
string ends at or above the watermark and the write stays inside the allocation. Growing
past the allocation still detaches, but capacity grows geometrically, so those copies are
amortized instead of one per append.

The first of the two commits here is the one sent separately as #7038, and it is a
prerequisite rather than a cleanup. mrb_str_cat() takes len as a size_t and hands it
to mrb_int_add_overflow(), whose parameters are mrb_int, so a length above
MRB_INT_MAX reaches the check already converted and negative. That is a latent
out-of-bounds memcpy() on master on its own, and with the watermark in place it would
additionally lower the watermark, which is the one way a co-sharer can be handed bytes
another string still reads. If #7038 lands first, this branch rebases onto it and the
commit disappears from here.

#7043 is a third change to the same handful of lines, independent of both this and #7038.
Whichever of the three lands first, I will rebase the others onto it.

Measurements

build_config/default.rb (-g -O3), gcc 13.3.0, Linux x86_64, 64-bit build, no boxing
overrides. Time and peak RSS from /usr/bin/time, each run under a 6 GB RLIMIT_AS.
CRuby 4.0.6 is there for scale, not as a target.

Dropping the slice at once, so nothing is retained and the live data is one string:

s = ""
n.times { s << "abcdefghijabcdefghij"; s[0, 30] }

dup alone, with no slicing at all:

s = ""
n.times { s << "0123456789012345678901234567890123456789"; t = s.dup }

Consuming a line buffer, the shape an incremental parser or a log scanner has. The slice
is most of the buffer here, so no threshold on slice size would cover it:

buf = ""
n.times do |i|
  buf << "hello world this is a line of text\n"
  if (i % 3) == 0
    idx = buf.index("\n")
    buf = buf[(idx + 1)..-1]
  end
end

What is left grows linearly. The first loop:

Retaining every slice is quadratic in memory on master and in CRuby alike, because each
retained slice pins a full-length snapshot. It stops being quadratic here, since every
slice and the parent go on sharing one buffer:

subs = []
s = ""
n.times { s << "0123456789012345678901234567890123456789"; subs << s[-30, 30] }

Shapes with no shared buffer in them are unchanged, best of seven interleaved runs at
n = 2000000:

The invariant

The watermark must only ever grow. str_init_shared() raises it to the end of every
string that joins the buffer, and an in-place append raises it past the bytes it claims.
Otherwise two strings sharing one buffer would both write at the same offset and silently
corrupt each other:

s1 = "a" * 100
s1 << "z" * 100
s1 = s1[0, 100]
s2 = s1.dup
s1 << "x"
s2 << "y"
s1              # "a" * 100 + "x"
s2              # "a" * 100 + "y"

Whichever of the two appends first claims the bytes; the other no longer ends at the
watermark and copies the buffer as it did before. The same holds for a tail slice and its
parent, whose ends coincide: the first append wins and the second one detaches. Both
orders, the interior-slice case, appending a slice of a buffer to the string it came from,
and a frozen sharer are covered by the test added to mruby-string-ext.

Consequences

• A const char* obtained from mrb_string_value_cstr() can lose its NUL terminator when
  another string sharing the buffer is appended to, because the in-place write can land on
  that byte. Ruby level code is unaffected: mrb_string_value_cstr() already detects a
  missing terminator and unshares. The repair is not free, though, since it then returns a
  pointer into a fresh allocation, so an extension caching the pointer across mruby
  operations is left holding a stale pointer rather than merely an unterminated one.
• Keeping spare capacity on share means a retained dup or slice of a string built by
  appending can pin up to twice the bytes it does today. I could not get this to show up
  in peak RSS, because the untouched tail of the allocation is never faulted in, but the
  virtual size is real.

MRB_STR_FSHARED strings and growth through mrb_str_resize() still detach in full.
Neither shape measured here needs otherwise, and both are room for a follow-up rather than
part of this change.

Testing

All green, KO: 0 and Crash: 0 everywhere:

• rake test with build_config/default.rb
• MRUBY_CONFIG=asan rake test (address,undefined)
• MRUBY_CONFIG=ci/gcc-clang rake test, which covers MRB_GC_STRESS with
  MRB_USE_DEBUG_HOOK, MRB_GC_FIXED_ARENA, and the C++ ABI build
• MRB_UTF8_STRING
• MRB_INT32 with MRB_WORD_BOXING and MRB_UTF8_STRING
• MRB_NAN_BOXING

The MRB_NAN_BOXING build fails three mruby-string-bitops assertions, and fails the
same three on master with this branch reverted, so they are not from this change.