string.c: a copy of a binary string is binary - #7080

str_replace() copies MRB_STR_SINGLE_BYTE from the string it copies and not
MRB_STR_BINARY, so every copy of a byte-indexed string comes back without the
flag that says to read it as bytes.

s = "\u{1F600}".b   # F0 9F 98 80: four bytes, one character

s.encoding          # ASCII-8BIT
s.dup.encoding      # UTF-8   <- CRuby: ASCII-8BIT
s.clone.encoding    # UTF-8   <- CRuby: ASCII-8BIT
"x".replace(s)      # UTF-8   <- CRuby: ASCII-8BIT

Why it is more than a label

chars2bytes() and bytes2chars() both read MRB_STR_BINARY to decide
whether an index is a byte or a character, so a copy that lost the flag has
every offset computed off it switch units.

mruby-regexp reaches this without the caller making a copy at all.
create_matchdata() snapshots the subject with mrb_str_dup_frozen(), so that
changing the subject afterwards is not visible through the MatchData. The
snapshot loses the flag, and re_byte_to_char() counts UTF-8 lead bytes over a
string that holds no characters:

s = "\u{1F600}".b

s =~ Regexp.new("\x80")                        # 1  <- CRuby: 3
s.byteindex(Regexp.new("\x80"))                # 3, correct
md = s.match(Regexp.new("\x80"))
md.begin(0)                                    # 1
md.pre_match.bytesize                          # 3  <- disagrees with begin(0)

One MatchData reported the same span in two units.

The change

Add RSTR_COPY_BINARY_FLAG beside RSTR_COPY_SINGLE_BYTE_FLAG and call it in
str_replace(). A copy holds the same bytes as what it copies, so it is
byte-indexed exactly when that is. Three lines.

The flag is copied rather than set, so a copy of a UTF-8 string stays UTF-8 and
String#b, which dups and then sets the flag, is unaffected.

Testing

rake test is green on full-core with gcc on x86_64-linux: 2216 asserts,
no failures.

Both new tests fail without the change and pass with it:

Fail: String#encoding survives a copy (mrbgems: mruby-encoding)
Fail: Regexp - a byte-indexed subject is reported in bytes (mrbgems: mruby-regexp)
  KO: 2   ->   KO: 0

A differential sweep against CRuby 4.0.6, over a byte-indexed subject with a
byte-indexed pattern (0x80 to 0xFF in four shapes, with and without /i, ten
subjects, 10240 cells), reading the position with =~:

The 84 this change closes are all offset reporting; the rest belong to #7078
and are unrelated to the flag.

Not in this change

String#* and the substring family have the same gap at their own sites, and
String#+ takes two strings, so which side decides is a rule to choose rather
than a flag to copy. String#size reads a binary string as UTF-8 for a
different reason: utf8_strlen() does not ask about the flag, while its two
neighbours do.

Summary by CodeRabbit

• Bug Fixes

  • String copies and replacements now preserve binary or UTF-8 encoding correctly.
  • Regular expression matching on binary strings now reports consistent byte-based positions.
  • String#byteindex and MatchData#begin/end now return correct offsets for binary data.
  • Existing UTF-8 strings continue to report character-based positions as expected.

• Tests

  • Added coverage for encoding preservation through duplication, cloning, freezing, and replacement.
  • Added regression tests for multibyte and binary string indexing behavior.