Skip to content

mruby-string-ext: String#slice! cuts a multibyte string by characters - #7103

Merged
matz merged 2 commits into
mruby:masterfrom
takumin:string-ext-slice-bang-char-offset
Aug 12, 2026
Merged

mruby-string-ext: String#slice! cuts a multibyte string by characters#7103
matz merged 2 commits into
mruby:masterfrom
takumin:string-ext-slice-bang-char-offset

Conversation

@takumin

@takumin takumin commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

String#slice! in mruby-string-ext converts between character indexes and byte
offsets on its own, and both directions are wrong on a multibyte string. The
result is a broken return value and a broken receiver:

s = "あいうえお"
s.slice!(3, 2)  #=> "\xE3\x81"     (CRuby: "えお")
s               #=> "あいう\x88お"  (CRuby: "あいう")

s = "あいう"
s.slice!("い")   #=> ""            (CRuby: "い")
s               #=> "あいう"        (CRuby: "あう")

Two independent defects, one commit each.

Character index to byte offset

str_char_to_byte_offset() and str_chars_to_byte_len() hand mrb_utf8len()
an end of p + byte_len - byte_offset. That is the end of the string only
while byte_offset is zero: every character consumed moves it one character
closer to the start. A character reaching past the moved end is measured as
truncated, which mrb_utf8len() reports as a single byte, so the conversion
answers an offset that lands inside a character.

mrb_str_char_to_byte() does this in core and is what String#[] goes
through, which is why "あいうえお"[3, 2] answers "えお" where slice! does
not. Both static helpers are deleted.

Byte offset to character index

mrb_str_index() answers a byte offset, which was passed to
mrb_str_substr() as a character count. mrb_str_byte_to_char() is the
conversion this wants. It also settles the case the byte search can produce
and the old code could not express: an offset inside a character, which it
reports as -1. CRuby finds no match in that position, so slice! answers nil
for it.

Both core conversions are declared outside the MRB_UTF8_STRING guard and are
the identity on a build without UTF-8 support, so the guarded branches go away
with the helpers.

Tests

Two assertions in mrbgems/mruby-string-ext/test/string.rb, guarded by the
UTF8STRING flag the file already uses. Each fails before its own commit and
passes after it. The existing String#slice! assertions are ASCII only, where
a character is a byte and neither defect shows.

Verified

  • rake test on a full-core build (MRB_UTF8_STRING through mruby-encoding):
    2237 tests, all green
  • rake test on the default gembox (no MRB_UTF8_STRING): 2053 tests, all
    green
  • prek run --all-files passes, except that markdownlint could not install
    locally (npm engine mismatch); no Markdown is touched here

Summary by CodeRabbit

  • Bug Fixes
    • Improved String#slice! handling for UTF-8 and multibyte strings.
    • Character-based indexes, ranges, negative indexes, and byte lengths now produce consistent results.
    • Prevented slicing from starting within a UTF-8 character.
    • Improved slicing by multibyte string matches, including unmatched and invalid byte-position cases.

`String#slice!` walked the string itself to turn a character index into a
byte offset, in `str_char_to_byte_offset()` and `str_chars_to_byte_len()`.
Both handed `mrb_utf8len()` an end of `p + byte_len - byte_offset`, which
is the end of the string only while `byte_offset` is zero: every character
consumed moves that end one character closer to the start. A character
reaching past the moved end is measured as truncated, which `mrb_utf8len()`
reports as a single byte, so the conversion answers an offset that lands
inside a character:

```ruby
"あいうえお".slice!(3, 2)  #=> "\xE3\x81"  (expected "えお")
"あいう".slice!(1..2)      #=> "い\xE3"    (expected "いう")
"あいう".slice!(-1)        #=> "\xE3"      (expected "う")
```

The bytes the receiver keeps are the ones the result did not take, so the
receiver is broken as well: the first line above leaves it holding
`"あいう\x88お"`.

`mrb_str_char_to_byte()` does this conversion in core and is what
`String#[]` goes through, which is why `"あいうえお"[3, 2]` answers `"えお"`
where `slice!` does not. Call it for both offsets and drop the two static
helpers. It is declared outside the `MRB_UTF8_STRING` guard and is the
identity on a build without UTF-8 support, so the guarded branch goes with
them.
`String#slice!(str)` searches with `mrb_str_index()`, which answers a byte
offset, and turned that into a character index by counting the characters
of `mrb_str_substr(mrb, self, 0, pos)`. `mrb_str_substr()` reads its
arguments as character positions, so the byte offset arrived there as a
character count and the substring it cut is longer than the part before the
match. The index that comes out is too large:

```ruby
"あいう".slice!("い")  #=> ""  (expected "い")
```

`"い"` starts at byte 3, `mrb_str_substr()` takes the 3 as three characters
and hands back the whole receiver, and the match is reported at character
3 rather than 1. The length is then clamped against the receiver, which
leaves nothing to cut.

`mrb_str_byte_to_char()` is the conversion this wants, and it also settles
what to do when the offset is not the start of a character: the search runs
over bytes and can land inside one, which it reports as -1. CRuby finds no
match in that position, so answer nil for it.
@takumin
takumin requested a review from matz as a code owner August 12, 2026 07:09
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 22b4a79d-1830-41c2-8020-2b82bd9afece

📥 Commits

Reviewing files that changed from the base of the PR and between e24e810 and c0501d1.

📒 Files selected for processing (2)
  • mrbgems/mruby-string-ext/src/string.c
  • mrbgems/mruby-string-ext/test/string.rb

📝 Walkthrough

Walkthrough

String#slice! now uses shared UTF-8 byte and character conversion APIs. New tests cover indexed, ranged, negative, length-based, and substring slicing on multibyte strings.

Changes

UTF-8 String slicing

Layer / File(s) Summary
Slice conversion and validation
mrbgems/mruby-string-ext/src/string.c, mrbgems/mruby-string-ext/test/string.rb
String#slice! converts substring match offsets and deletion ranges through shared byte/character APIs. UTF-8 tests cover indexes, ranges, negative indexes, lengths, substring matches, absent matches, and matches inside multibyte characters.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • mruby/mruby#7063: Both PRs modify String#slice! behavior for different argument types.
  • mruby/mruby#7097: This PR uses the shared conversion APIs introduced there.
  • mruby/mruby#7098: This PR relies on the strengthened byte-to-character offset handling.

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change to make String#slice! handle multibyte strings by characters.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@matz
matz merged commit 5b5d148 into mruby:master Aug 12, 2026
21 checks passed
@takumin
takumin deleted the string-ext-slice-bang-char-offset branch August 12, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants