mruby-string-ext: String#slice! cuts a multibyte string by characters - #7103
Merged
Conversation
`String#slice!` walked the string itself to turn a character index into a byte offset, in `str_char_to_byte_offset()` and `str_chars_to_byte_len()`. Both handed `mrb_utf8len()` an end of `p + byte_len - byte_offset`, which is the end of the string only while `byte_offset` is zero: every character consumed moves that end one character closer to the start. A character reaching past the moved end is measured as truncated, which `mrb_utf8len()` reports as a single byte, so the conversion answers an offset that lands inside a character: ```ruby "あいうえお".slice!(3, 2) #=> "\xE3\x81" (expected "えお") "あいう".slice!(1..2) #=> "い\xE3" (expected "いう") "あいう".slice!(-1) #=> "\xE3" (expected "う") ``` The bytes the receiver keeps are the ones the result did not take, so the receiver is broken as well: the first line above leaves it holding `"あいう\x88お"`. `mrb_str_char_to_byte()` does this conversion in core and is what `String#[]` goes through, which is why `"あいうえお"[3, 2]` answers `"えお"` where `slice!` does not. Call it for both offsets and drop the two static helpers. It is declared outside the `MRB_UTF8_STRING` guard and is the identity on a build without UTF-8 support, so the guarded branch goes with them.
`String#slice!(str)` searches with `mrb_str_index()`, which answers a byte
offset, and turned that into a character index by counting the characters
of `mrb_str_substr(mrb, self, 0, pos)`. `mrb_str_substr()` reads its
arguments as character positions, so the byte offset arrived there as a
character count and the substring it cut is longer than the part before the
match. The index that comes out is too large:
```ruby
"あいう".slice!("い") #=> "" (expected "い")
```
`"い"` starts at byte 3, `mrb_str_substr()` takes the 3 as three characters
and hands back the whole receiver, and the match is reported at character
3 rather than 1. The length is then clamped against the receiver, which
leaves nothing to cut.
`mrb_str_byte_to_char()` is the conversion this wants, and it also settles
what to do when the offset is not the start of a character: the search runs
over bytes and can land inside one, which it reports as -1. CRuby finds no
match in that position, so answer nil for it.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthrough
ChangesUTF-8 String slicing
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
String#slice!in mruby-string-ext converts between character indexes and byteoffsets on its own, and both directions are wrong on a multibyte string. The
result is a broken return value and a broken receiver:
Two independent defects, one commit each.
Character index to byte offset
str_char_to_byte_offset()andstr_chars_to_byte_len()handmrb_utf8len()an end of
p + byte_len - byte_offset. That is the end of the string onlywhile
byte_offsetis zero: every character consumed moves it one charactercloser to the start. A character reaching past the moved end is measured as
truncated, which
mrb_utf8len()reports as a single byte, so the conversionanswers an offset that lands inside a character.
mrb_str_char_to_byte()does this in core and is whatString#[]goesthrough, which is why
"あいうえお"[3, 2]answers"えお"whereslice!doesnot. Both static helpers are deleted.
Byte offset to character index
mrb_str_index()answers a byte offset, which was passed tomrb_str_substr()as a character count.mrb_str_byte_to_char()is theconversion this wants. It also settles the case the byte search can produce
and the old code could not express: an offset inside a character, which it
reports as -1. CRuby finds no match in that position, so
slice!answers nilfor it.
Both core conversions are declared outside the
MRB_UTF8_STRINGguard and arethe identity on a build without UTF-8 support, so the guarded branches go away
with the helpers.
Tests
Two assertions in
mrbgems/mruby-string-ext/test/string.rb, guarded by theUTF8STRINGflag the file already uses. Each fails before its own commit andpasses after it. The existing
String#slice!assertions are ASCII only, wherea character is a byte and neither defect shows.
Verified
rake teston a full-core build (MRB_UTF8_STRINGthrough mruby-encoding):2237 tests, all green
rake teston the default gembox (noMRB_UTF8_STRING): 2053 tests, allgreen
prek run --all-filespasses, except thatmarkdownlintcould not installlocally (npm engine mismatch); no Markdown is touched here
Summary by CodeRabbit
String#slice!handling for UTF-8 and multibyte strings.