string.c: remember that a string was read as broken - #7144

mrb_str_valid_encoding_p() marks a string whose bytes it has read as UTF-8,
and answers a later question off that mark rather than by another walk. Only
the answer of true is kept that way. A string holding a byte that spells no
character leaves nothing behind, so every question after the first walks it
again, however many times it is asked: once per needle in str_index_str(),
once per call at the entry to split, once per subject handed to the regexp
engine. This keeps the other answer the same walk comes back with.

The mark

MRB_STR_BROKEN_ENC takes the bit next to MRB_STR_VALID_ENC, with the four
accessors beside the ones it mirrors. The two marks stand or fall together,
since a write to the bytes unmakes either answer:

• mrb_str_valid_encoding_p() leaves one where the walk reaches the end and
  the other where it fails.
• mrb_str_modify_keep_ascii() unmakes both. Every in-place write reaches
  there.
• The shared appending path in str_modify_cat() unmakes both.
• str_replace() writes over the bytes without passing either of those, so it
  takes both answers from the string it copies. This one is not an
  optimization: without it a broken string that replace makes valid keeps
  saying it is broken.
• mrb_str_times() carries both, guarded as described below.
• mrb_str_byte_subseq() carries neither, as it already carried neither of
  the answer it had. A cut leaves a character in pieces, and it also cuts away
  the piece that spelled none, so a subrange inherits validity in neither
  direction. The comment there said this about one mark and now says it about
  both.

"a\x80".valid_encoding?                  #=> false
"a\x80".byteslice(0, 1).valid_encoding?  #=> true
"あ".valid_encoding?                     #=> true
"あ".byteslice(0, 1).valid_encoding?     #=> false

Where the two marks part

* is the one place. A repetition of broken bytes reaches the same broken
place the first copy does, so the result is broken too. Nought copies keep
none of the bytes, though, and an empty string is not broken whatever it was
made from, so the mark travels only where something was copied. The mark
beside it needs no such guard, since an empty string is valid either way.

force_encoding

A byte-indexed string is answered before either mark is read, so a broken
string that force_encoding makes binary still says it is valid, and one made
UTF-8 again meets the mark it left. The bytes did not change, only what they
are taken to be, so that is the answer to give.

Builds without MRB_UTF8_STRING

There is no encoding for a string to break, and the mark is nothing there, as
the one beside it already is. mrb_str_valid_encoding_p() keeps answering
true for every string.

Tests

Nothing a program can see changes, so the tests here are what stops a mark
from outliving the bytes it was left for. String#valid_encoding? survives what the string goes through already asks the same question of the same bytes
twice, once with the answer remembered and once without, but every base it
started from was valid and every change it made broke them, so it only ever
asked about a mark left by an answer of true. It now starts from broken bases
as well, and makes changes that repair them.

The test commit comes first and passes on its own, since the answers it asks
for are what they were. What it pins is the mark. Two mutations show that it
reaches the code: dropping the line that unmakes the mark in
mrb_str_modify_keep_ascii(), and dropping the guard on nought copies in
mrb_str_times(). Each one fails it. A third came up while writing this: with
str_replace() left out of the second commit, the test failed there too,
which is why that line sits with the rest rather than with *.

Testing

Each of the three commits was built and run on its own against the last two of
those, and all three are green.

Summary by CodeRabbit

• Bug Fixes
  • Improved UTF-8 encoding validation for strings by caching invalid results.
  • Ensured encoding status is correctly updated after string mutations, slicing, replacement, and repetition.
  • Prevented zero-length repetitions and substrings from incorrectly inheriting invalid encoding status.
• Tests
  • Expanded coverage for valid, malformed UTF-8, binary, and mutation scenarios.