Read a string built out of two strings off both of them - #7137

Stacked on #7136, which carried the byte reading onto a string built out of nothing but one byte-read string's bytes: the pieces cut out of it, its repetitions and the pads around it. Its four commits are the first four here and are not part of this change; the two after them answer the question a copy never raises.

When a string is built out of two strings, which operand's reading does the result get? Every seam that does this answered "UTF-8, whatever went in", so each of these resurrected the state #7132 removed:

171.chr + 171.chr          #=> UTF-8, invalid   (CRuby: ASCII-8BIT)
buf = ""; buf << 171.chr   #=> UTF-8, invalid   (CRuby: ASCII-8BIT)
[171.chr, 171.chr].join    #=> UTF-8, invalid   (CRuby: ASCII-8BIT)
"ab".gsub("a", 171.chr)    #=> UTF-8, invalid   (CRuby: ASCII-8BIT)
"ab".center(4, 255.chr)    #=> UTF-8, invalid   (CRuby: ASCII-8BIT)

The rule

Bytes that were read as bytes and go above ASCII spell no character where they land, so they hand the result the byte reading along with themselves. ASCII bytes read the same under any reading and move nothing.

• Sums — + now reads both operands: two byte-read operands stay byte-read; a byte-read operand with a byte above ASCII wins; an all-ASCII byte-read operand yields to the other side. That reproduces CRuby's answer for every pair it accepts; the pairs CRuby refuses with Encoding::CompatibilityError come out byte-read here, saying nothing rather than something false.
• Appends and splices — the rule sits in mrb_str_cat_str(), so <<, concat, interpolation, join and everything built on them take it; the splices that memmove instead carry it by hand (insert, prepend, []=, __sub_replace, and mruby-regexp's __sub_str/__gsub_str). A pad argument reaches the result through the same append, which is what the two new center cases ask.

Whether a byte above ASCII is there is answered by str_ascii_p(), which reads MRB_STR_SINGLE_BYTE where a walk already settled it and leaves the flag behind where it walks itself: a walk that finds every byte ASCII made exactly the statement that flag makes.

sub_replace() took its replacement and match as char * through mrb_get_args(mrb, "ssi", …), which is not enough to see what they were read as, so it takes them as S and reads the pointers off the values.

Two receivers deliberately do not move

• append_as_bytes takes only the bytes of its argument. CRuby specifies exactly this, and there is a test pinning it.
• A byte-read receiver stays byte-read even where CRuby would lift an all-ASCII one to a UTF-8 argument's encoding. That lift makes a claim; staying byte-read makes none.

Left out, on purpose

• format/% (mruby-sprintf) builds through a raw buffer and never sees the argument's flags, so "%s" % 171.chr still reports UTF-8. A separate seam.
• Array#pack marks nothing either ([171].pack("C") reports UTF-8, invalid). That is producer-side, the same kind of fix Stop Integer#chr from calling a stray byte a character #7132 made for chr, and its own change.
• In builds without mruby-regexp, the pure-Ruby gsub rebuilds its result from pieces, so an all-ASCII result cut from a byte-read receiver comes back UTF-8 ("ab".b.gsub("a", "-")). The C path used with mruby-regexp gets this right; the Ruby path cannot see the receiver's flag from Ruby.

Tests

The pin commit in #7136 recorded where each of these answers stood before anything moved; the two commits here flip exactly the pins they change. New matrix tests cover sums, appends, splices and append_as_bytes in mruby-encoding, and sub/gsub results in mruby-regexp.

The large-character-class regexp test built its pattern as sums of chr pieces, spelling raw bytes it means to have read as UTF-8. That spelling now honestly says byte-read, so it builds them with append_as_bytes instead, which lays bytes down without moving how the string is read.

Full suite green on host-debug (full-core, MRB_UTF8_STRING) and the default config (byte strings, MRB_UTF8_SCAN regexp), at every commit: 2278 + 116 and 2069 + 105 tests, 0 failures, 0 crashes.

Summary by CodeRabbit

• Bug Fixes

  • Improved UTF-8 and binary string encoding handling across slicing, copying, repetition, concatenation, replacement, insertion, padding, and interpolation.
  • Preserved binary status when extracting substrings, lines, characters, and regular-expression matches.
  • Improved sub and gsub results when binary data or non-ASCII bytes are involved.
  • Preserved raw byte content during byte-oriented concatenation and append operations.

• Tests

  • Added comprehensive coverage for ASCII-only, UTF-8, and non-ASCII binary string behavior.