Stop Integer#chr from calling a stray byte a character - #7132
Merged
Conversation
`String#length` counts such a byte as a character of its own, and the tests that ask it do so of strings holding nothing else. What the same byte counts as beside whole characters, and what index it takes, is asked nowhere, and both are answers the character count decides. So ask them: a broken sequence next to ASCII and next to a character of several bytes, and `String#[]` reading a surrogate's bytes back one at a time. `String#chars` already pins the split itself, and these are the count and the index the split is read through.
…aracter `inspect` walks the string it prints, and what it meets on the way settles MRB_STR_SINGLE_BYTE on the receiver: the flag stays set unless a character of several bytes is copied across whole. A byte that spells no character is copied one at a time, like an ASCII one, so a string carrying one came out of `inspect` flagged as holding one character per byte. That flag is read as leave for stepping through the bytes without decoding them. `String#codepoints` and `String#ord` then hand the bytes back as the characters the string does not hold, and `String#scrub` finds nothing to replace, so what a broken string answers turns on whether `inspect` was called on it first: ```ruby s = "\xED\xA0\x80" s.codepoints # ArgumentError s.inspect s.codepoints #=> [237, 160, 128] s.scrub.bytes #=> [237, 160, 128], where it was [239, 191, 189] * 3 ``` CRuby answers those the same way whatever ran before. Ask the walk about the byte instead of about the character it failed to begin: a non-ASCII byte leaves the string more bytes than characters whether or not it spells one. What `inspect` returns is unchanged, and the flag it puts on that return is too. `\xNN` is what a byte spelling no character is escaped to, so the string that comes back is still one character per byte, which only a whole character copied across takes from it. The two answers needed separating to say so.
…cter Counting characters sets MRB_STR_SINGLE_BYTE when the count comes out equal to the byte count, and the flag is read as leave for stepping through the bytes without decoding them. A byte that spells no character is counted as one, so a string carrying nothing else counts one per byte and set the flag too. Its bytes then came back as the characters it does not hold: ```ruby s = "\xED\xA0\x80" s.codepoints # ArgumentError s.length #=> 3 s.codepoints #=> [237, 160, 128] s.ord #=> 237, where it raised s.scrub.bytes #=> [237, 160, 128], where it was [239, 191, 189] * 3 ``` CRuby answers those the same way whatever ran before. Every character a non-ASCII byte begins spells two bytes or more, and a non-ASCII byte that begins none spells no character at all, so a string holds one character per byte exactly when every byte of it is ASCII. Ask that instead: the count already looks for the first non-ASCII byte before it counts anything, so where there is none the answer is the byte count and the flag holds, and where there is one the count carries on from it. `String#length` still counts a byte spelling no character as a character, and `String#chars` and `String#[]` still give it a position of its own. Only what is remembered about the string changes, and the walk those two take without the flag reads such a byte as one character anyway. The test before this one asks for that. `RSTR_SET_ASCII_FLAG` is already a second name for setting this flag, and `String#ascii_only?` sets it exactly where every byte is ASCII, so this is the meaning the flag's readers were written against. The comment in `mrb_str_valid_encoding_p` argued from the wider reading and goes.
`Integer#chr` takes any byte from 0 to 0xff. A byte of 0x80 and above spells no UTF-8 character on its own, so pin what the one-byte string it returns answers before anything about it moves: its bytes, its length, the code points and the ordinal read off it, what `scrub` and `inspect` make of it, the encoding it reports and whether it calls itself valid. `<<` and `append_as_bytes` reach the same code, so pin what they append too. Neither takes anything from that string but its bytes.
`int_chr_binary()` set `MRB_STR_SINGLE_BYTE` on whatever one-byte string it built. That flag says the string holds one character per byte, and its readers take it as leave to step through the bytes without decoding them, which is why `RSTR_SET_ASCII_FLAG` is a second name for setting it. A byte of 0x80 and above spells no UTF-8 character at all, so the flag was a false statement about every such string, and the string went on reporting UTF-8 while refusing to read as it. ```ruby 171.chr.encoding #=> ASCII-8BIT in CRuby, UTF-8 in mruby 171.chr.valid_encoding? #=> true in CRuby, false in mruby ``` Set `MRB_STR_BINARY` there instead and keep the ASCII flag for a byte below 0x80. CRuby draws the same line, handing back US-ASCII below 0x80 and ASCII-8BIT above it; mruby has no US-ASCII and reads such a string as UTF-8, which holds it exactly. The code points and the ordinal read off the string do not change: they already came out right, through the branch that byte-indexed strings take anyway. `inspect` does not change either, since a byte spelling no character was escaped as `\xNN` before and is escaped as `\xNN` now. `<<`, `concat` and `append_as_bytes` reach the same helper but take only the bytes off what it returns, through `mrb_str_cat_str()`, so nothing they append changes.
`MRB_STR_SINGLE_BYTE` now says that every byte of the string is ASCII, and ASCII reads as UTF-8 as it stands, so a string carrying that flag is valid without a walk. Return early on it and remember the answer in `MRB_STR_VALID_ENC`, which is where the walk would have left it. A string counted before it is asked about comes in carrying the flag, which is where this saves the walk. Asking 2000 counted 5 KB ASCII strings once each: 0.5 ms before, 0.2 ms after. A string that was not counted first, and one that is not ASCII, are unaffected. This is sound only because the flag is set nowhere else than on a string of ASCII bytes. `int_chr_binary()` was the last place that set it on a byte spelling no character. Two comments in the tests said that measuring a string of stray bytes marks it one byte per character. That stopped being so when the flag was narrowed, and it is what this change now rests on, so drop them. The assertions under them are what pins it and they stay.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe change corrects UTF-8 character-length, validity, binary character construction, and inspection flag handling. Tests cover malformed bytes across indexing, sizing, inspection, codepoint conversion, ordinal access, scrubbing, and encoding checks. ChangesUTF-8 string handling
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 13, 2026
This was referenced Aug 13, 2026
This was referenced Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7131, which narrowed
MRB_STR_SINGLE_BYTEto mean that every byteof the string is ASCII. Its three commits are the first three here and are not
part of this change; the three after them close the one place left that set the
flag on a byte spelling no character, and then put the narrower meaning to use.
Integer#chrint_chr_binary()takes any byte from 0 to 0xff, and setMRB_STR_SINGLE_BYTEon whatever one-byte string it built.
RSTR_SET_ASCII_FLAGis a second name forsetting that flag, and its readers take it as leave to step through the bytes
without decoding them. A byte of 0x80 and above spells no UTF-8
character at all, so the flag was a false statement about every such string,
and the string went on reporting UTF-8 while refusing to read as it:
MRB_STR_BINARYis what says a string is read by bytes, so set that instead,and keep the ASCII flag for a byte below 0x80. CRuby draws the same line,
handing back US-ASCII below 0x80 and ASCII-8BIT above it; mruby has no
US-ASCII and reads such a string as UTF-8, which holds it exactly.
Nothing else about the string moves. The code points and the ordinal read off
it already came out right, through the branch a byte-indexed string takes
anyway, and
inspectescaped the byte as\xNNbefore and escapes it as\xNNnow.<<,concatandappend_as_bytesreach the same helper buttake only the bytes off what it returns, through
mrb_str_cat_str().valid_encoding?With that closed,
MRB_STR_SINGLE_BYTEis set nowhere but on a string ofASCII bytes, and ASCII reads as UTF-8 as it stands.
mrb_str_valid_encoding_p()can answer
truefrom the flag and leave the answer inMRB_STR_VALID_ENC,which is where the walk would have put it.
That is what a string counted before it is asked about comes in carrying.
Asking 2000 counted 5 KB ASCII strings once each: 0.5 ms before, 0.2 ms after.
A string that was not counted first, and one that is not ASCII, are unaffected.
The order matters: the fast path on its own, without the first change, makes
0xE3.chrreport UTF-8 and call itself valid, and the whole test suite stillpasses. That is why the two are separate commits and why the test below asks
for the round trip rather than for either answer.
What is left
String#+builds its result withstr_new()and carries no flag over, so171.chr + "b"still comes out as a UTF-8 string of a byte that spellsnothing, where CRuby gives
[171, 98]. That ismrb_str_plus(), not this,and is left alone here.
String#dupdoes carry the flag, throughstr_replace().Tests
The fourth commit pins what
Integer#chrhands back for a byte above ASCIIbefore anything moves, including the two answers the fifth changes.
The sixth adds two invariants. One asks
valid_encoding?of the same bytestwice, once after the string has been read by
length,chars,[],each_char,inspect,ordorcodepoints, and once not: the two have toagree, which is what makes reading the answer off a flag sound. The other asks
that a string reporting UTF-8 and calling itself valid spells its own bytes
back from its code points, which is the check the flag cannot satisfy by
itself. Taking the
Integer#chrchange back out fails both and nothing else.Two comments in the existing tests said that measuring a string of stray bytes
marks it one byte per character. That stopped being so in #7131 and is what
the last commit rests on, so they are dropped; the assertions under them stay.
rake testis green on three builds:full-corewithMRB_UTF8_STRING, thedefaultgembox without it, andfull-corewithMRB_UTF8_STRINGandMRB_INT32. Each commit was checked on its own.Summary by CodeRabbit
Bug Fixes
Tests