Skip to content

mruby-regexp: String#split steps by a byte through a binary subject #7085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matz merged 1 commit into from
Aug 10, 2026
Merged

mruby-regexp: String#split steps by a byte through a binary subject #7085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions mrbgems/mruby-regexp/mrblib/string_regexp.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,7 @@ def split(pattern = nil, *args)
search_pos = 0
len = self.bytesize
count = 0
binary = Regexp.__binary_string?(self)
while search_pos <= len
if limit > 0 && count >= limit - 1
result << (self.byteslice(field_start..-1) || "")
Expand All @@ -289,12 +290,20 @@ def split(pattern = nil, *args)
match_end = md.__byte_end(0)

if match_start == match_end
rest = self.byteslice(match_end..-1)
if rest && rest.bytesize > 0
char = rest[0]
search_pos = match_end + char.bytesize
else
if binary
# A byte-indexed subject has one position per byte, and the step
# below reads the rest of it as UTF-8: `byteslice` hands back a
# string without the flag, so its first element is a whole character
# again. `gsub` steps by a byte here for the same reason.
search_pos = match_end + 1
else
rest = self.byteslice(match_end..-1)
if rest && rest.bytesize > 0
char = rest[0]
search_pos = match_end + char.bytesize
else
search_pos = match_end + 1
end
end
next if match_start == field_start
end
Expand Down
15 changes: 15 additions & 0 deletions mrbgems/mruby-regexp/test/regexp.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -918,6 +918,21 @@
assert_equal ["あ", ",", "い", ",", "う"], "あ,い,う".split(/(,)/)
end

assert("String#split - a byte-indexed subject is split by byte") do
skip unless __ENCODING__ == "UTF-8"
# An empty match steps to the next position, and `String#b` makes every byte
# one. The step read the subject as UTF-8 and cleared a whole character,
# so a four-byte string came back in one piece.
s = "\u{1F600}".b # F0 9F 98 80: four bytes, one character
assert_equal ["\xF0".b, "\x9F".b, "\x98".b, "\x80".b], s.split(//)
assert_equal 4, s.split(//).size
assert_equal ["a".b, "\xC3".b, "\xA9".b, "b".b], "a\u{E9}b".b.split(//)
# a subject read as UTF-8 is still split by character
assert_equal ["\u{1F600}"], "\u{1F600}".split(//)
# and the limit still counts fields, not bytes
assert_equal ["\xF0".b, "\x9F\x98\x80".b], s.split(//, 2)
end

assert("Regexp.escape") do
assert_equal "a\\.b\\*c", Regexp.escape("a.b*c")

Expand Down
Loading