string.h: fold the coderange's three bits into a two-bit field - #7170

RString keeps what reading its bytes as the encoding they carry came back with, and it keeps that answer in three separate bits, one per answer:

#define RSTR_CODERANGE(s) \
  (((s)->flags & MRB_STR_BROKEN_ENC) ? MRB_STR_CODERANGE_BROKEN : \
   ((s)->flags & MRB_STR_SINGLE_BYTE) ? MRB_STR_CODERANGE_7BIT : \
   ((s)->flags & MRB_STR_VALID_ENC) ? MRB_STR_CODERANGE_VALID : \
   MRB_STR_CODERANGE_UNKNOWN)

The four answers are exclusive, so three bits can spell states no writer makes, and the order above is there only to decide what such a state would read as. "Not asked yet" has no value of its own either: it is the three bits all being clear, which the reader has to know to spell as MRB_STR_CODERANGE_UNKNOWN.

This PR gives the answer bits of its own. The set is closed at four, the same set CRuby names with ENC_CODERANGE_*, so two bits hold every one of them and can spell nothing besides.

What goes into include/mruby/string.h

#define MRB_STR_CODERANGE_SHIFT 4
#define MRB_STR_CODERANGE_BITS 2
#define MRB_STR_CODERANGE_MASK (((1 << MRB_STR_CODERANGE_BITS) - 1) << MRB_STR_CODERANGE_SHIFT)

# define RSTR_CODERANGE(s) \
  (((s)->flags & MRB_STR_CODERANGE_MASK) >> MRB_STR_CODERANGE_SHIFT)
# define RSTR_CODERANGE_SET(s, cr) \
  ((s)->flags = ((s)->flags & ~MRB_STR_CODERANGE_MASK) | \
                (((cr) & ((1 << MRB_STR_CODERANGE_BITS) - 1)) << MRB_STR_CODERANGE_SHIFT))

The three constants take the shape MRB_STR_EMBED_LEN_SHIFT / _BITS / _MASK and MRB_STR_ENCODING_SHIFT / _BITS / _MASK already use for the other two fields in the word.

MRB_STR_CODERANGE_UNKNOWN stays 0, so the zeroed flags of a freshly allocated string already say it and the path every string is made on still stores nothing.

The three bits go with the answers they held. Since f51a52e put the last readers and writers on the accessors, every mention of them on master is inside this header, and nothing outside it changes:

$ git grep -nE 'MRB_STR_SINGLE_BYTE|MRB_STR_VALID_ENC|MRB_STR_BROKEN_ENC'   # on master
include/mruby/string.h:54:#define MRB_STR_SINGLE_BYTE 32
include/mruby/string.h:57:#define MRB_STR_VALID_ENC 2048
include/mruby/string.h:58:#define MRB_STR_BROKEN_ENC 4096
include/mruby/string.h:64:   row and MRB_STR_SINGLE_BYTE holds the one beside it. Moving them down is
include/mruby/string.h:115:  (((s)->flags & MRB_STR_BROKEN_ENC) ? MRB_STR_CODERANGE_BROKEN : \
include/mruby/string.h:116:   ((s)->flags & MRB_STR_SINGLE_BYTE) ? MRB_STR_CODERANGE_7BIT : \
include/mruby/string.h:117:   ((s)->flags & MRB_STR_VALID_ENC) ? MRB_STR_CODERANGE_VALID : \
include/mruby/string.h:121:                 ~(MRB_STR_SINGLE_BYTE|MRB_STR_VALID_ENC|MRB_STR_BROKEN_ENC)) | \
include/mruby/string.h:122:                (((cr) == MRB_STR_CODERANGE_7BIT) ? MRB_STR_SINGLE_BYTE : \
include/mruby/string.h:123:                 ((cr) == MRB_STR_CODERANGE_VALID) ? MRB_STR_VALID_ENC : \
include/mruby/string.h:124:                 ((cr) == MRB_STR_CODERANGE_BROKEN) ? MRB_STR_BROKEN_ENC : 0))

$ git grep -nE 'MRB_STR_SINGLE_BYTE|MRB_STR_VALID_ENC|MRB_STR_BROKEN_ENC'   # on this branch
$

Where the field sits

Bits 4-5 are the lowest pair the flags word has free. Bit 4 came free with 839b164, and bit 5 is the first of the three this PR folds.

Three bits go in and two come out, so bits 11-12 come free as a pair. Where the field finally sits is for whenever the word is laid out afresh, as is where the encoding index finally sits: neither move is in this PR.

The write is masked to the field

RSTR_ENCODING_SET() masks its index to the width of its field as of f90657d. This write does the same, and here the reason is sharper: the bits above this field are the embedded length rather than free ones.

RSTR_CODERANGE_SET(s, 4)   /* 4 << 4 == 64 == bit 6: one more embedded byte */

A fifth answer is a mistake either way, and widening MRB_STR_CODERANGE_BITS is what carrying one would need. What the mask decides is whether the mistake stays a wrong answer instead of becoming a longer string.

The three bits leave a public header

include/mruby/string.h is a public header, so an out of tree gem that names one of the three stops compiling. That is the point, and it matters more here than it did for MRB_STR_BINARY in 839b164: those values are not merely unread now, they name parts of a field that reads back as a different answer.

An alias would keep every one of those compiling while it stopped meaning what it meant, which is the failure this shape is meant to prevent. A compile error names the lines to change, and what to change them to is the accessors that have been there since f51a52e:

Names in this family have left the header before: MRB_STR_BINARY in 839b164, RSTR_SET_BINARY_FLAG() and its companions in 4848467, the unused unset macros in c18cd60, and the ASCII flag renamed to MRB_STR_SINGLE_BYTE in 57fd0ed.

Builds that index by byte

A build without MRB_UTF8_STRING keeps RSTR_CODERANGE() at MRB_STR_CODERANGE_7BIT and RSTR_CODERANGE_SET() a no-op, so it carries no field at all, exactly as before. RSTR_ENC_CR_COPY() and RSTR_ENC_CR_COPY_FOR_SUBSTR() still copy the encoding there, which str_replace() needs in every build.

Generated code

gcc -O3, against the same objects built from master. Each of the four builds ci/gcc-clang makes:

The objects that differ are the ones that name the accessors: src/string.o, src/object.o, src/symbol.o, src/numeric.o, mruby-string-ext, and mruby-time. Comparison is objdump -d over every .o in the build; every other object is identical instruction for instruction, and in byte-string, where the field does not exist, so is every object there is.

The shrinking is the immediate. On x86-64 a mask over bits 4 and 5 fits one byte and one that reaches bit 11 or 12 does not. Where the compiler has the flags word shifted down to work on, as in mrb_str_byte_subseq() asking whether the source is 7BIT and writing the answer, the instruction goes from six bytes to three:

   4552:  81 e1 20 10 00 00     and    $0x1020,%ecx        <- 7BIT set and BROKEN clear
   4558:  83 f9 20              cmp    $0x20,%ecx
   455b:  0f 94 c1              sete   %cl
   455e:  81 e2 df e7 ff ff     and    $0xffffe7df,%edx    <- clear the three
   456c:  c1 e1 05              shl    $0x5,%ecx

   4622:  83 e1 30              and    $0x30,%ecx          <- read the field
   4625:  83 f9 10              cmp    $0x10,%ecx
   4628:  0f 94 c1              sete   %cl
   462b:  83 e2 cf              and    $0xffffffcf,%edx    <- clear the field
   4636:  c1 e1 04              shl    $0x4,%ecx

Where it works on the word in place instead, the mask is shifted up past a byte either way and the instruction keeps its width, which is why some of the six differ in what they hold without changing size.

This gives back what 839b164 took on when it put the encoding index in bits 13 and 14, for the field that is read and written far more often.

Testing

rake -m test over ci/gcc-clang, all four builds green, 0 KO, 0 crash, 0 warnings:

byte-string is the build without mruby-encoding, where strings index by byte and no field is carried. The other three are what pin the fold: every path that records an answer or copies one runs there, String#valid_encoding? and String#scrub for the recording, String#* and String#replace for the whole copy, byte slicing for the copy that carries 7BIT alone.

No test accompanies the change. No string answers anything different, so there is nothing new to pin.

Not in this PR

The encoding index does not move, the field does not go to its final position, and nothing new writes a coderange.

Summary by CodeRabbit

• Refactor
  • Streamlined internal string encoding metadata into a compact two-bit coderange field.
  • Updated coderange reading and writing to use the packed representation with bounds protection.
  • Replaced obsolete encoding flag constants with the new coderange field definitions.