Skip to content

Stop Integer#chr from calling a stray byte a character - #7132

Merged
matz merged 6 commits into
mruby:masterfrom
takumin:int-chr-binary-encoding
Aug 12, 2026
Merged

Stop Integer#chr from calling a stray byte a character#7132
matz merged 6 commits into
mruby:masterfrom
takumin:int-chr-binary-encoding

Conversation

@takumin

@takumin takumin commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Stacked on #7131, which narrowed MRB_STR_SINGLE_BYTE to mean that every byte
of the string is ASCII. Its three commits are the first three here and are not
part of this change; the three after them close the one place left that set the
flag on a byte spelling no character, and then put the narrower meaning to use.

Integer#chr

int_chr_binary() takes any byte from 0 to 0xff, and set MRB_STR_SINGLE_BYTE
on whatever one-byte string it built. RSTR_SET_ASCII_FLAG is a second name for
setting that flag, and its readers take it as leave to step through the bytes
without decoding them. A byte of 0x80 and above spells no UTF-8
character at all, so the flag was a false statement about every such string,
and the string went on reporting UTF-8 while refusing to read as it:

171.chr.encoding          #=> ASCII-8BIT in CRuby, UTF-8 in mruby
171.chr.valid_encoding?   #=> true in CRuby, false in mruby

MRB_STR_BINARY is what says a string is read by bytes, so set that instead,
and keep the ASCII flag for a byte below 0x80. CRuby draws the same line,
handing back US-ASCII below 0x80 and ASCII-8BIT above it; mruby has no
US-ASCII and reads such a string as UTF-8, which holds it exactly.

Nothing else about the string moves. The code points and the ordinal read off
it already came out right, through the branch a byte-indexed string takes
anyway, and inspect escaped the byte as \xNN before and escapes it as
\xNN now. <<, concat and append_as_bytes reach the same helper but
take only the bytes off what it returns, through mrb_str_cat_str().

valid_encoding?

With that closed, MRB_STR_SINGLE_BYTE is set nowhere but on a string of
ASCII bytes, and ASCII reads as UTF-8 as it stands. mrb_str_valid_encoding_p()
can answer true from the flag and leave the answer in MRB_STR_VALID_ENC,
which is where the walk would have put it.

That is what a string counted before it is asked about comes in carrying.
Asking 2000 counted 5 KB ASCII strings once each: 0.5 ms before, 0.2 ms after.
A string that was not counted first, and one that is not ASCII, are unaffected.

The order matters: the fast path on its own, without the first change, makes
0xE3.chr report UTF-8 and call itself valid, and the whole test suite still
passes. That is why the two are separate commits and why the test below asks
for the round trip rather than for either answer.

What is left

String#+ builds its result with str_new() and carries no flag over, so
171.chr + "b" still comes out as a UTF-8 string of a byte that spells
nothing, where CRuby gives [171, 98]. That is mrb_str_plus(), not this,
and is left alone here. String#dup does carry the flag, through
str_replace().

Tests

The fourth commit pins what Integer#chr hands back for a byte above ASCII
before anything moves, including the two answers the fifth changes.

The sixth adds two invariants. One asks valid_encoding? of the same bytes
twice, once after the string has been read by length, chars, [],
each_char, inspect, ord or codepoints, and once not: the two have to
agree, which is what makes reading the answer off a flag sound. The other asks
that a string reporting UTF-8 and calling itself valid spells its own bytes
back from its code points, which is the check the flag cannot satisfy by
itself. Taking the Integer#chr change back out fails both and nothing else.

Two comments in the existing tests said that measuring a string of stray bytes
marks it one byte per character. That stopped being so in #7131 and is what
the last commit rests on, so they are dropped; the assertions under them stay.

rake test is green on three builds: full-core with MRB_UTF8_STRING, the
default gembox without it, and full-core with MRB_UTF8_STRING and
MRB_INT32. Each commit was checked on its own.

Summary by CodeRabbit

  • Bug Fixes

    • Improved handling of malformed and incomplete UTF-8 strings.
    • Corrected character indexing, length, encoding validity, escaping, inspection, and scrubbing behavior for invalid bytes.
    • Ensured ASCII and binary strings are classified correctly.
    • Preserved consistent results after character-counting and read operations.
  • Tests

    • Added comprehensive coverage for invalid UTF-8 sequences, standalone bytes, valid characters, codepoints, and custom scrub replacements.

`String#length` counts such a byte as a character of its own, and the tests
that ask it do so of strings holding nothing else. What the same byte counts
as beside whole characters, and what index it takes, is asked nowhere, and
both are answers the character count decides.

So ask them: a broken sequence next to ASCII and next to a character of
several bytes, and `String#[]` reading a surrogate's bytes back one at a
time. `String#chars` already pins the split itself, and these are the count
and the index the split is read through.
…aracter

`inspect` walks the string it prints, and what it meets on the way settles
MRB_STR_SINGLE_BYTE on the receiver: the flag stays set unless a character of
several bytes is copied across whole. A byte that spells no character is
copied one at a time, like an ASCII one, so a string carrying one came out of
`inspect` flagged as holding one character per byte.

That flag is read as leave for stepping through the bytes without decoding
them. `String#codepoints` and `String#ord` then hand the bytes back as the
characters the string does not hold, and `String#scrub` finds nothing to
replace, so what a broken string answers turns on whether `inspect` was
called on it first:

```ruby
s = "\xED\xA0\x80"
s.codepoints         # ArgumentError
s.inspect
s.codepoints         #=> [237, 160, 128]
s.scrub.bytes        #=> [237, 160, 128], where it was [239, 191, 189] * 3
```

CRuby answers those the same way whatever ran before. Ask the walk about the
byte instead of about the character it failed to begin: a non-ASCII byte
leaves the string more bytes than characters whether or not it spells one.

What `inspect` returns is unchanged, and the flag it puts on that return is
too. `\xNN` is what a byte spelling no character is escaped to, so the string
that comes back is still one character per byte, which only a whole character
copied across takes from it. The two answers needed separating to say so.
…cter

Counting characters sets MRB_STR_SINGLE_BYTE when the count comes out equal to
the byte count, and the flag is read as leave for stepping through the bytes
without decoding them. A byte that spells no character is counted as one, so a
string carrying nothing else counts one per byte and set the flag too. Its
bytes then came back as the characters it does not hold:

```ruby
s = "\xED\xA0\x80"
s.codepoints         # ArgumentError
s.length             #=> 3
s.codepoints         #=> [237, 160, 128]
s.ord                #=> 237, where it raised
s.scrub.bytes        #=> [237, 160, 128], where it was [239, 191, 189] * 3
```

CRuby answers those the same way whatever ran before. Every character a
non-ASCII byte begins spells two bytes or more, and a non-ASCII byte that
begins none spells no character at all, so a string holds one character per
byte exactly when every byte of it is ASCII. Ask that instead: the count
already looks for the first non-ASCII byte before it counts anything, so where
there is none the answer is the byte count and the flag holds, and where there
is one the count carries on from it.

`String#length` still counts a byte spelling no character as a character, and
`String#chars` and `String#[]` still give it a position of its own. Only what
is remembered about the string changes, and the walk those two take without
the flag reads such a byte as one character anyway. The test before this one
asks for that.

`RSTR_SET_ASCII_FLAG` is already a second name for setting this flag, and
`String#ascii_only?` sets it exactly where every byte is ASCII, so this is the
meaning the flag's readers were written against. The comment in
`mrb_str_valid_encoding_p` argued from the wider reading and goes.
`Integer#chr` takes any byte from 0 to 0xff. A byte of 0x80 and above
spells no UTF-8 character on its own, so pin what the one-byte string it
returns answers before anything about it moves: its bytes, its length,
the code points and the ordinal read off it, what `scrub` and `inspect`
make of it, the encoding it reports and whether it calls itself valid.

`<<` and `append_as_bytes` reach the same code, so pin what they append
too. Neither takes anything from that string but its bytes.
`int_chr_binary()` set `MRB_STR_SINGLE_BYTE` on whatever one-byte string
it built. That flag says the string holds one character per byte, and
its readers take it as leave to step through the bytes without decoding
them, which is why `RSTR_SET_ASCII_FLAG` is a second name for setting it.
A byte of 0x80 and above spells no UTF-8 character at all, so the flag
was a false statement about every such string, and the string went on
reporting UTF-8 while refusing to read as it.

```ruby
171.chr.encoding          #=> ASCII-8BIT in CRuby, UTF-8 in mruby
171.chr.valid_encoding?   #=> true in CRuby, false in mruby
```

Set `MRB_STR_BINARY` there instead and keep the ASCII flag for a byte
below 0x80. CRuby draws the same line, handing back US-ASCII below 0x80
and ASCII-8BIT above it; mruby has no US-ASCII and reads such a string as
UTF-8, which holds it exactly.

The code points and the ordinal read off the string do not change: they
already came out right, through the branch that byte-indexed strings take
anyway. `inspect` does not change either, since a byte spelling no
character was escaped as `\xNN` before and is escaped as `\xNN` now.

`<<`, `concat` and `append_as_bytes` reach the same helper but take only
the bytes off what it returns, through `mrb_str_cat_str()`, so nothing
they append changes.
`MRB_STR_SINGLE_BYTE` now says that every byte of the string is ASCII,
and ASCII reads as UTF-8 as it stands, so a string carrying that flag is
valid without a walk. Return early on it and remember the answer in
`MRB_STR_VALID_ENC`, which is where the walk would have left it.

A string counted before it is asked about comes in carrying the flag,
which is where this saves the walk. Asking 2000 counted 5 KB ASCII
strings once each: 0.5 ms before, 0.2 ms after. A string that was not
counted first, and one that is not ASCII, are unaffected.

This is sound only because the flag is set nowhere else than on a string
of ASCII bytes. `int_chr_binary()` was the last place that set it on a
byte spelling no character.

Two comments in the tests said that measuring a string of stray bytes
marks it one byte per character. That stopped being so when the flag was
narrowed, and it is what this change now rests on, so drop them. The
assertions under them are what pins it and they stay.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a519d494-65f3-4f99-ba6d-242372f5eda1

📥 Commits

Reviewing files that changed from the base of the PR and between 57b0e66 and 95afe8b.

📒 Files selected for processing (6)
  • mrbgems/mruby-encoding/test/numeric.rb
  • mrbgems/mruby-encoding/test/string.rb
  • mrbgems/mruby-string-ext/src/string.c
  • mrbgems/mruby-string-ext/test/string.rb
  • src/string.c
  • test/t/string.rb

📝 Walkthrough

Walkthrough

The change corrects UTF-8 character-length, validity, binary character construction, and inspection flag handling. Tests cover malformed bytes across indexing, sizing, inspection, codepoint conversion, ordinal access, scrubbing, and encoding checks.

Changes

UTF-8 string handling

Layer / File(s) Summary
Byte classification and character measurement
src/string.c, mrbgems/mruby-string-ext/src/string.c, test/t/string.rb, mrbgems/mruby-encoding/test/numeric.rb, mrbgems/mruby-encoding/test/string.rb
Character-length detection no longer marks malformed non-ASCII data as single-byte. Validity checks recognize single-byte strings. Integer#chr marks bytes below 0x80 as ASCII and higher bytes as binary. Tests cover indexing, sizing, byte construction, and repeated validity checks.
Inspection state preservation
src/string.c, mrbgems/mruby-string-ext/test/string.rb
Inspection tracks source and result single-byte state separately. Tests verify that malformed UTF-8 remains invalid after inspection and character operations, while scrubbing and ASCII handling retain expected results.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

  • mruby/mruby#7131: Directly overlaps the UTF-8 string fixes and malformed-sequence regression tests.
  • mruby/mruby#7105: Modifies related malformed UTF-8 handling in codepoints, ord, and scrub.
  • mruby/mruby#7093: Covers related UTF-8 validation and invalid-sequence behavior.

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the primary change to prevent Integer#chr from treating invalid non-ASCII bytes as characters.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants