Skip to content

mruby-regexp: speed up String#sub and #gsub #7274

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
matz merged 1 commit into from
Aug 19, 2026
Merged

mruby-regexp: speed up String#sub and #gsub #7274

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 71 additions & 75 deletions mrbgems/mruby-regexp/mrblib/string_regexp.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
# With the type established, each override reaches the engine through class
# methods that take the pattern as an argument (`Regexp.__search`,
# `__byte_search`, `__byte_rsearch`, `__search_p`, `__sub_str`, `__gsub_str`,
# `__scan`), so
# `__sub_lit`, `__gsub_lit`, `__gsub_block`, `__scan`), so
# nothing rewritten on the pattern instance is consulted on the way: the C
# side searches, and the loops and blocks stay here. The MatchData those
# side searches, and what loops and blocks remain stay here. The MatchData those
# searches answer is built in C too, so what the overrides read from it
# (`[]`, `pre_match`, `begin` and the rest) cannot have been planted by an
# argument. What remains reachable is a class-wide redefinition of a
Expand Down Expand Up @@ -97,31 +97,40 @@ def =~(re)

def sub(*args, &block)
# CRuby accepts 1..2 arguments with a block, but demands exactly 2
# without one, and reports the expected count accordingly.
# without one, and reports the expected count accordingly. The count is
# read once and compared, rather than asked of a Range built for the
# question: this is the first thing every call does, and a Range costs an
# object and a call to answer what two comparisons answer.
argc = args.length
if block
unless (1..2).include?(args.length)
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 1..2)"
unless argc == 1 || argc == 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 1..2)"
end
elsif args.length != 2
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 2)"
elsif argc != 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 2)"
end
pattern, replacement = *args
pattern = Regexp.__check_pattern(pattern)
pattern = Regexp.__check_pattern(args[0])
# Unlike `match`, a String pattern is quoted rather than compiled: it is a
# literal here, the distinction CRuby draws between get_pat_quoted and
# get_pat. Only the quoting is taken from it: get_pat_quoted also accepts
# anything answering `to_str`, where `__check_pattern` keeps to a real
# String, as `match` already does.
literal = String === pattern
# A replacement argument wins over the block, as in CRuby. A literal goes
# to `__sub_lit`, which searches for its bytes without compiling anything
# to search with. What a compiled pattern would be needed for is the
# Regexp the `$~` it publishes names, and `MatchData#regexp` quotes that
# one where CRuby quotes it: on the first call that asks for it.
if argc == 2
replacement = args[1].to_s
return Regexp.__sub_lit(pattern, self, replacement) if literal
return Regexp.__sub_str(pattern, self, replacement)
end
# CRuby searches for a literal byte by byte and never reads the subject as
# UTF-8 on the way, so quoting one into a Regexp here must not put the
# subject through a check CRuby does not make: `"a\x80b".sub("b", "!")`
# answers there, where the same call with `/b/` is refused.
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# A replacement argument wins over the block, as in CRuby.
if args.length == 2
return Regexp.__sub_str(pattern, self, replacement.to_s, literal)
end
md = Regexp.__search(pattern, self, 0, literal)
return self.dup unless md
md.pre_match + block.call(md[0]).to_s + md.post_match
Expand All @@ -133,22 +142,33 @@ def sub!(*args, &block)
# `"abc".freeze.sub!(:b, "X")` TypeError, while the two-argument form on
# the same receiver raises FrozenError. `gsub!` orders it the other way,
# also as CRuby does.
argc = args.length
if block
unless (1..2).include?(args.length)
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 1..2)"
unless argc == 1 || argc == 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 1..2)"
end
elsif args.length != 2
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 2)"
elsif argc != 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 2)"
end
# Resolved here rather than left to `sub` because the match below decides
# the return value, and a String pattern is a literal on both paths.
pattern = Regexp.__check_pattern(args[0])
literal = String === pattern
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# Quoting the literal below raises nothing, so asking here is the order the
# original had: after the argument checks, before any search.
raise FrozenError, "can't modify frozen String" if frozen?
# Whether a substitution happened is a question about the match, not about
# the result: `"aaa".sub!(/a/, "a")` returns self even though the string is
# unchanged. A full search and not `match?`, so a failed match clears $~.
# unchanged. The `bang` argument is that question asked of the one search
# `__sub_lit` already makes, so the literal path does not walk the subject
# twice to answer it; a failed search clears $~ there as `__search` does.
if literal && argc == 2
str = Regexp.__sub_lit(pattern, self, args[1].to_s, true)
return nil unless str
return self.replace(str)
end
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# A full search and not `match?`, so a failed match clears $~.
return nil unless Regexp.__search(pattern, self, 0, literal)
# `sub` matches again and publishes its own $~ over this one, leaving the
# caller the match `sub` would have left, a block's own matches included.
Expand All @@ -159,92 +179,68 @@ def sub!(*args, &block)
# Overwriting `self` afterwards is safe: a MatchData snapshots its subject,
# so $~ keeps describing the string as it was matched.
down = literal ? args[0] : pattern
str = args.length == 2 ? self.sub(down, args[1], &block) : self.sub(down, &block)
str = argc == 2 ? self.sub(down, args[1], &block) : self.sub(down, &block)
self.replace(str)
end

def gsub(*args, &block)
unless (1..2).include?(args.length)
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 1..2)"
argc = args.length
unless argc == 1 || argc == 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 1..2)"
end
# Without mruby-enumerator this is core Kernel#to_enum, which raises
# NotImplementedError; every other path here stays usable, so the gem does
# not depend on Enumerator.
return to_enum(:gsub, *args) if args.length == 1 && !block
pattern, replacement = *args
return to_enum(:gsub, *args) if argc == 1 && !block
pattern = args[0]
# After the to_enum return above, so that `"abc".gsub(:b)` yields an
# Enumerator and raises on the first iteration, as CRuby does.
pattern = Regexp.__check_pattern(pattern)
# A String pattern is a literal, as in `sub`, and reaches the subject the
# way CRuby reaches it: byte by byte, with no reading of it as UTF-8.
literal = String === pattern
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# A replacement argument wins over the block, as in CRuby.
if args.length == 2
return Regexp.__gsub_str(pattern, self, replacement.to_s, literal)
# A replacement argument wins over the block, as in CRuby. A literal is
# searched for as bytes and compiles nothing, as in `sub` above.
if argc == 2
replacement = args[1].to_s
return Regexp.__gsub_lit(pattern, self, replacement) if literal
return Regexp.__gsub_str(pattern, self, replacement)
end
# block case: keep in Ruby to avoid VM callback from C
parts = []
pos = 0
len = self.bytesize
binary = Regexp.__binary_string?(self)
# The loop normally ends on a failed __byte_search, which clears $~ and
# the thirteen names that go with it. CRuby leaves the last match behind,
# so keep it and republish it below. A gsub that matched nothing has
# nothing to restore and keeps the cleared state, as CRuby does.
last = nil
while pos <= len
md = Regexp.__byte_search(pattern, self, pos, literal)
break unless md
last = md
# gsub works in byte space (match pos, byteslice). begin/end report
# character offsets (CRuby-compatible), so use the byte accessors.
match_start = md.__byte_begin(0)
match_end = md.__byte_end(0)
parts << self.byteslice(pos, match_start - pos)
parts << block.call(md[0]).to_s
if match_start == match_end
if match_end < len
if binary
parts << self.byteslice(match_end, 1)
pos = match_end + 1
else
rest = self.byteslice(match_end..-1)
char = rest[0]
parts << char
pos = match_end + char.bytesize
end
else
pos = match_end + 1
end
else
pos = match_end
end
end
parts << self.byteslice(pos..-1)
last.__set_globals if last
parts.join
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# The walk and the block call are both in `__gsub_block`. What the loop
# was written here for, the block reading the globals of the match it was
# handed, a C loop publishes just as well, and it pays neither the frame
# per search nor the array of pieces the mrblib one collected.
Regexp.__gsub_block(pattern, self, literal, &block)
end

def gsub!(*args, &block)
# Before the arity check and before the enumerator below, as in CRuby:
# `"abc".freeze.gsub!(/a/)` raises FrozenError rather than handing back an
# Enumerator that fails later.
raise FrozenError, "can't modify frozen String" if frozen?
unless (1..2).include?(args.length)
raise ArgumentError, "wrong number of arguments (given #{args.length}, expected 1..2)"
argc = args.length
unless argc == 1 || argc == 2
raise ArgumentError, "wrong number of arguments (given #{argc}, expected 1..2)"
end
return to_enum(:gsub!, *args) if args.length == 1 && !block
return to_enum(:gsub!, *args) if argc == 1 && !block
pattern = Regexp.__check_pattern(args[0])
literal = String === pattern
pattern = Regexp.new(Regexp.escape(pattern)) if literal
# As in `sub!`: the match decides the return value, and a failed search
# clears $~. What it publishes on success is replaced right away by the
# last match of the `gsub` below, which is the one CRuby leaves behind.
# A literal goes down as the String it was, for the reason `sub!` gives.
# A literal goes down as the String it was, for the reason `sub!` gives,
# and a literal with a replacement asks the question of `__gsub_lit`
# itself rather than searching once to ask and again to substitute.
if literal && argc == 2
str = Regexp.__gsub_lit(pattern, self, args[1].to_s, true)
return nil unless str
return self.replace(str)
end
pattern = Regexp.new(Regexp.escape(pattern)) if literal
return nil unless Regexp.__search(pattern, self, 0, literal)
down = literal ? args[0] : pattern
str = args.length == 2 ? self.gsub(down, args[1], &block) : self.gsub(down, &block)
str = argc == 2 ? self.gsub(down, args[1], &block) : self.gsub(down, &block)
self.replace(str)
end

Expand Down
Loading
Loading