string.c: keep one answer about a string's bytes, not three flags - #7158

Stacked on #7157, which is the first commit here. The second commit is this PR's own change, and the diff to read is git diff <first commit>..HEAD. Merging #7157 first leaves this one a single commit.

Reading a string's bytes as the encoding it carries comes back with one of four answers: not asked yet, nothing but ASCII, read whole and sound, read and found broken. include/mruby/string.h keeps them as three flags, MRB_STR_SINGLE_BYTE, MRB_STR_VALID_ENC and MRB_STR_BROKEN_ENC, each with a predicate, a set, an unset and a copy of its own.

No writer sets two of them, so the answers are exclusive already. What the three flags leave to their readers is the map from combinations back to answers:

/* src/string.c, mrb_str_valid_encoding_p(): SINGLE_BYTE implies VALID_ENC,
   so the reader answers off one flag and then writes the other. */
if (RSTR_SINGLE_BYTE_P(s)) {
  RSTR_SET_VALID_ENC_FLAG(s);
  return TRUE;
}

/* src/string.c, str_replace(): a copy stands where its source stands,
   spelled once per flag. */
RSTR_COPY_SINGLE_BYTE_FLAG(s1, s2);
RSTR_COPY_VALID_ENC_FLAG(s1, s2);
RSTR_COPY_BROKEN_ENC_FLAG(s1, s2);

This PR gives the answer a name of its own. The three bits stay exactly where they are and the accessors are written over them, so what changes is the vocabulary and not the layout.

What goes into include/mruby/string.h

#define MRB_STR_CODERANGE_UNKNOWN 0
#define MRB_STR_CODERANGE_7BIT    1
#define MRB_STR_CODERANGE_VALID   2
#define MRB_STR_CODERANGE_BROKEN  3

#ifdef MRB_UTF8_STRING
# define RSTR_CODERANGE(s) \
  (((s)->flags & MRB_STR_BROKEN_ENC) ? MRB_STR_CODERANGE_BROKEN : \
   ((s)->flags & MRB_STR_SINGLE_BYTE) ? MRB_STR_CODERANGE_7BIT : \
   ((s)->flags & MRB_STR_VALID_ENC) ? MRB_STR_CODERANGE_VALID : \
   MRB_STR_CODERANGE_UNKNOWN)
# define RSTR_CODERANGE_SET(s, cr) /* writes all three bits */
#else
# define RSTR_CODERANGE(s) MRB_STR_CODERANGE_7BIT
# define RSTR_CODERANGE_SET(s, cr) ((void)0)
#endif

#define RSTR_ENC_CR_COPY(dst, src)
#define RSTR_ENC_CR_COPY_FOR_SUBSTR(dst, src)

The four answers are the set CRuby closes on in ENC_CODERANGE_UNKNOWN, _7BIT, _VALID and _BROKEN, and the names follow. UNKNOWN is 0 for the same reason it is 0 there: a fresh string is filled with zeroes and has been asked nothing.

The values are small integers rather than the flag positions CRuby uses, because here they are not positions of anything: they sit over three bits that are not adjacent, and MRB_STR_SINGLE_BYTE|MRB_STR_VALID_ENC is not the bit pattern of any answer. A build without MRB_UTF8_STRING reads every byte as a character and asks the bytes nothing, which is where 7BIT stands, so RSTR_CODERANGE() is that constant there and RSTR_CODERANGE_SET() records nothing. That is what RSTR_SINGLE_BYTE_P() and RSTR_VALID_ENC_P() being TRUE said before.

Two copies, and why they carry the encoding too

The encoding travels with the answer in one macro rather than beside it in another, since a copy that keeps one and drops the other is how flags went missing on copy paths before (f71bd22 and the whole of #7136 are that shape: a path that copied the bytes and left the flag behind on the original). The two copies stay outside the #ifdef, so a byte indexed build still carries the encoding across.

There are two because a subrange is not an exact copy. A cut can leave a character in pieces, and it can cut away the piece that spelled none, so a subrange inherits neither soundness nor brokenness. Nothing but ASCII survives being cut anywhere, so 7BIT is the one answer mrb_str_byte_subseq() carries over. CRuby splits the same two ways, in rb_enc_cr_str_exact_copy() and rb_enc_cr_str_copy_for_substr(); the second of those also walks the subrange to promote VALID to 7BIT, which is not what this code does today and is not added here.

The sites

RSTR_SET_ASCII_FLAG() was an alias for RSTR_SET_SINGLE_BYTE_FLAG(), and it goes with the flag it aliased. Its callers say RSTR_CODERANGE_SET(s, MRB_STR_CODERANGE_7BIT), so there is one word for the answer rather than two.

Writes where one flag became three bits

RSTR_CODERANGE_SET() writes all three bits where the old sets wrote one, so every converted write is a place where clearing the other two has to lose nothing. It does, because no writer sets two of them and 7BIT is the one answer that implies another. Three are worth naming:

• mrb_str_valid_encoding_p() no longer writes "sound" onto a string that is already nothing but ASCII. It is the one write that goes rather than moves.
• mrb_str_modify_keep_ascii() and mrb_str_modify() are where every in-place write passes and where the answer stops holding. As a pair they took soundness and brokenness away and then took ASCII away as well; now the first keeps 7BIT and writes UNKNOWN over anything else, and the second writes UNKNOWN. str_modify_cat(), which reaches neither when it appends into a shared buffer, writes UNKNOWN in one go instead of clearing three bits.
• String#* still refuses to call a nought times repetition broken. The receiver's answer is copied and then taken back off the empty result, which is where it stood before.

Generated code

Two full-core builds, with mruby-encoding and with the gem removed from the box, gcc -O3, measured against the tree with #7157 alone:

Where the UTF-8 build moves:

Six objects change at all, and src/numeric.o changes without growing. The four that grow are the files whose only change is RSTR_SET_ASCII_FLAG() becoming a write of one answer: a bit set turns into a read, a mask and an or. Inside src/string.o the sites this PR touches move by tens of bytes in both directions (mrb_str_times() +68, mrb_str_char_len() +64, mrb_str_modify_keep_ascii() +64, mrb_str_valid_encoding_p() -41), and the largest single movers are functions this PR does not touch at all: mrb_str_resize() 1057 -> 208 and mrb_str_to_s() 665 -> 110, where the inliner rearranges around a str_replace() that is now four macro expansions shorter.

Testing

rake -m test on both builds above, plus the C++ ABI build, all green:

• full-core with MRB_UTF8_STRING: 2291 tests, 2281 OK, 10 skip, 0 KO, 0 crash, 0 warnings.
• full-core without mruby-encoding, where strings index by byte: 2229 tests, 2200 OK, 29 skip, 0 KO, 0 crash, 0 warnings.
• the same box with enable_cxx_abi: 2291 tests, 2281 OK, 10 skip.

No test accompanies the change. No string answers anything different, so there is nothing new to pin.

Not in this PR

The three bits are not folded into a two bit field, and none of them moves. MRB_STR_SINGLE_BYTE, MRB_STR_VALID_ENC and MRB_STR_BROKEN_ENC keep their values and are now named only inside the two accessors.

Summary by CodeRabbit

• Improvements
  • Improved internal handling of string encoding and character validity.
  • String operations now better preserve encoding information when copying, slicing, combining, and transforming text.
  • Enhanced handling for ASCII, UTF-8, binary, valid, and invalid string content across common string operations.