string.c: a binary string is measured in bytes - #7081

chars2bytes() and bytes2chars() both ask RSTR_BINARY_P() before deciding
whether an index counts bytes or characters, so a string marked by String#b
is indexed by byte. utf8_strlen() asks only about the single-byte flag, so
the same string is measured as UTF-8. One string answers two ways, and the
length is the one every method bounded by it believes.

s = "\u{1F600}".b   # F0 9F 98 80: four bytes, one character

s.size          # 1     <- CRuby: 4
s.bytesize      # 4
s[0]            # "\xF0"   indexed as a byte all along
s[1]            # nil   <- CRuby: "\x9F"
s[3]            # nil   <- CRuby: "\x80"
s[0, 2]         # "\xF0"   <- CRuby: "\xF0\x9F"
s.slice(1, 2)   # ""    <- CRuby: "\x9F\x98"
s.chars.size    # 4, walking bytes and disagreeing with s.size
s.reverse!      # the string unchanged: one character reversed is itself

The change

utf8_strlen() returns the byte length for a binary string, which is what its
two neighbours already answer for the index. One condition.

The single-byte flag is deliberately not set on that path. That flag says the
bytes hold nothing multi-byte, while the early return is about how the string
is read, and force_encoding can hand the same bytes back to the UTF-8
reading:

s = "\u{1F600}".b
s.size                             # 4
s.force_encoding(Encoding::UTF_8)
s.size                             # 1, not a cached 4

What this does not reach

String#reverse copies the string first, and str_replace() does not carry
MRB_STR_BINARY to the copy, so it still reads the copy as UTF-8. #reverse!
is fixed here. #7080 makes the flag survive a copy, and with the two together
the copying form follows. Neither change depends on the other landing first,
and they merge cleanly in either order: they touch different functions in
src/string.c and add their tests at different points in the same file.

String#inspect and String#chop! each walk mrb_utf8len() themselves rather
than going through the length, so a binary string is read as UTF-8 there for a
reason of their own. Those are separate fixes and are not mixed in here.

mrb_str_index_m() and mrb_str_rindex_m() take a byte path on the same flag
the others read, and are left alone. The general path they fall into converts
through bytes2chars(), which returns the byte offset for a binary string
already, so the answer is the same either way and this would be a speed-up
rather than a fix.

Testing

rake test is green on full-core with gcc on x86_64-linux: 2216 asserts,
no failures. Both new tests fail without the change:

Fail: String#length of a binary string counts bytes
Fail: String#reverse! on a binary string reverses bytes
  KO: 2   ->   KO: 0

Merged with #7080 locally, rake test is green as well (2218 asserts), and
String#reverse joins #reverse! in answering in bytes.

Checked against CRuby 4.0.6 over size, length, bytesize, [] by index
and by pair, slice, chars, each_char, reverse!, center, ljust,
rjust and insert, on a four-byte one-character subject and on a mixed one:
every case agrees afterwards except the ones listed above as out of reach.

Summary by CodeRabbit

• Bug Fixes

  • Corrected string length handling for binary strings so lengths are reported in bytes.
  • Improved behavior for indexing, slicing, encoding changes, and reversing binary and multibyte UTF-8 strings.

• Tests

  • Added coverage for UTF-8 string operations, including byte-based lengths and String#reverse!.