mruby-regexp: publish $~ alone and derive $&, ` $ `, $', $+ and $1` onward on read - #7281

Every successful search published fourteen names: $~, then $&, $`, $', $+ and $1 to $9, each cut from the subject as a string of its own and each written with mrb_gv_set in set_match_globals(). Almost nothing read them: a block given to gsub reads $~ or $1 at most, and the loops of gsub, scan, split and sub published once per match. On the 480 byte subject of #7267, searched 72 times a call with Regexp.__byte_search, set_match_globals() was 3,000 of the 15,600 instructions a search cost under callgrind, and create_matchdata() as a whole 5,300; the match itself is a small part of either.

This is the lazy publishing from the review of #7267, where you said it was on your list from #7148 and #7149: if you have a version of it in hand, or this is not the shape you meant, say so and I will withdraw it.

The thirteen are readings of $~

CRuby keeps no variables for them: $1 is getspecial, which asks rb_reg_nth_match of the backref at the moment of the read, and the same goes for $& (rb_reg_last_match), $` (rb_reg_match_pre), $' (rb_reg_match_post) and $+ (rb_reg_match_last). That is why assigning $~ moves all of them, why $~ = nil clears them, and why $10 reads the tenth group. None of that held here, because each name was a global in its own right that only a search wrote:

/(b)/ =~ "abc"; md = $~
$~ = nil; $1        # CRuby nil, mruby "b"
/(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)/ =~ "abcdefghij"; $10   # CRuby "j", mruby nil

The compiler now writes each of them as a read of $~ and, where that is not nil, a send on it: __group with the number for $& (0) and $n, __pre_match for $`, __post_match for $' and __last_group for $+. All four are private readings of the match, __pre_match and __post_match being the functions the public pre_match and post_match are defined from under a second name, so that the five names read the match itself, the way CRuby's rb_reg_nth_match and the like do, and not a method a program can redefine. $1 is

GETGV   R4  $~
JMPNIL  R4  L
LOADI_1 R5
SEND    R4  :__group  n=1
L:

where it was one GETGV. Where $~ is nil, which it is in a build without mruby-regexp and after a miss, the name reads as nil, as an unset global did, so a build without the gem does not change. A number the parser cannot hold (it hands over 0 for those) reads as nil, where CRuby warns and answers nil.

The gem then publishes $~ alone: set_match_globals() is one mrb_gv_set, clear_match_globals() one of nil, and the symbol tables for the other names go. A MatchData published again at the end of a loop, by Regexp.__gsub_block and MatchData#__republish, no longer rebuilds the strings either. Regexp.last_match(n) already read $~.

What changes for a reader

mrb_gv_get() of $1 from C answers nil (nothing in the tree reads the names that way), global_variables no longer lists them after a match, and bytecode compiled before this reads them with OP_GETGV and so reads nil; source recompiles to the derived form, and the RITE format is unchanged. $~ = 1; $1 raises NoMethodError, where CRuby refuses the assignment with TypeError; mruby has no hook on a global write to raise from, and the assignment was accepted before this too, with the stale name of an earlier match following it. Redefining MatchData#[], #pre_match or #post_match moves $~[n], $~.pre_match and $~.post_match and leaves the five names where they were, as it does in CRuby. A private name is not a lock, though: a $~ that answers __group answers $1.

A differential over 10 patterns and 5 subjects through 27 call shapes, reading $~.to_a, $&, $`, $', $+, $1, $2, $3, $9 and $10 after each call and inside the blocks of sub!, gsub!, gsub, scan and sub, and after $~ = md, $~ = nil, a miss after an assignment and Regexp.last_match, is 1,545 lines under CRuby 4.0.6, under master and under this branch: master differs from CRuby on 154 of them, the assignment forms and the $10 reads, and this branch on 0.

Cost

Wall clock, the cases of the review on #7267 and the cases that read the names in the block, minimum of 7 alternating runs, -O3, default configuration, master addc03bc5:

s480 = "hello world foo bar " * 24
30000.times { s480.gsub(/o/) { } }                 # gsub480
3000.times { (s480 * 10).gsub(/o/) { } }           # gsub4800
30000.times { s480.scan(/o/) { } }                 # scan480
100000.times { "abc".gsub(/b/) { } }               # gsub_abc
100000.times { "abc".scan(/b/) { } }               # scan_abc
100000.times { "abc".sub!(/b/) { } }               # sub_bang_abc
100000.times { "abc".gsub(/b/, "0") }              # gsub_str
100000.times { "abc".gsub(/z/) { } }               # gsub_none
100000.times { "abc".gsub(/(b)/) { $1 } }          # read1
100000.times { "abc".gsub(/(b)/) { $& + $` + $' + $1 } }   # read4
30000.times { s480.gsub(/(o)/) { $1 } }            # read480
30000.times { s480.split(/o/) }                    # split480
30000.times { s480.scan(/o/) }                     # scan480_arr

A read of one of the names is a send on $~ now where it was a global read, and the block that reads four of them per match still comes out ahead, since the match that published them cost more than the reads do. scan480_arr is the one case that comes out behind: the call makes 74 objects instead of 77 and the heap is two pages, so the free slots run out at a different point of the collector's cycle and the call falls into a full collection more often (100 in 2000 calls against 72 under callgrind, the rest of the difference being that); one more allocation in the loop body, a subject of 240 or 960 bytes, or 5000 live objects beside it each put this branch 10% or more ahead on the same case.

The third commit does not show in the wall clock: the same thirteen cases between it and the commit before it are within 2% either way, the send it writes costing no more than the OP_GETIDX it replaces.

Under callgrind, Ir(2N) - Ir(N) on the Regexp.__byte_search loop above (72 searches a call): a search is 15,641 instructions on master and 12,942 here (-17%), of which create_matchdata() is 5,310 and 2,351, and set_match_globals() 3,032 and 101.

Size

.text of bin/mruby, build_config/ci/gcc-clang.rb, each side from a clean build directory.

codegen.o and regexp.o are the objects that change, and together they are the whole delta in every build (to 8 bytes of alignment in full-debug). In bintest the first commit adds 464 to codegen.o for gen_match_ref() and 192 to regexp.o for matchdata_last_group(), the second takes 2,224 out of regexp.o with the publisher and its symbol tables, and the third adds 64 to codegen.o for the send and 320 to regexp.o for matchdata_group() and the two names beside it.

On the default configuration, the one the wall clock was measured on, .text is 1,201,422 on master and 1,200,238 here (-1,184).

Testing

Full suite green at each of the three commits: rake -m test on the default configuration, MRUBY_CONFIG=build_config/ci/gcc-clang.rb rake -m test in a fresh build directory at each commit, and MRUBY_CONFIG=build_config/gcc-asan.rb rake -m test at the tip, no sanitizer report. No new compiler warnings.

New assertions in match_data.rb pin the three readings: the names follow $~ through an assignment of nil and of a MatchData an earlier search made, and $10 and $11 read their groups. They fail on master ($1 is "b" after $~ = nil, $10 is nil) and pass from the first commit on. Every other test that reads one of the names, of which the gem has many, now reads it through the derived form.

The third commit adds six more: __group itself, including the numbers that read as no group at all; that __pre_match, __post_match and __last_group answer what the compiler sends them for; that the five names stand where a read of a global stands, beside other values, inside a literal, in a call and under a block with locals of its own; that redefining [], pre_match and post_match moves $~[n], $~.pre_match and $~.post_match and nothing else; that a $~ that is not a MatchData raises on all five; and that a rewritten __group, or an object that merely answers it, is read all the same, which is the limit of what a private name buys. The one that redefines methods puts them back with remove_method, so the gem asks for mruby-metaprog as a test dependency where the build has it; a build without that gem is green too (stdlib and stdlib-ext, 1,649 assertions, 0 KO), which is the state the test skips its cleanup in.

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/regexp.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK mrbgems/mruby-regexp/src/regexp.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/regexp.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/regexp.c
# ascii-ctype
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CTYPE -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/regexp.c
# gcc-asan
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -fsanitize=address,undefined -g3 -O0 -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER mrbgems/mruby-regexp/src/regexp.c
# default (no MRUBY_CONFIG), the build every figure above was measured on
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRBGEM_MRUBY_REGEXP_VERSION=0.0.0 -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DMRB_USE_COMPLEX -DMRB_USE_BIGINT -DMRB_USE_DEBUG_HOOK mrbgems/mruby-regexp/src/regexp.c

Summary by CodeRabbit

• New Features

  • Regular-expression match variables now dynamically reflect the current match, including full, pre-match, post-match, last-capture, and numbered captures.
  • Match variables correctly update when the current match changes or is cleared.
  • Numbered captures beyond the available groups return nil.

• Documentation

  • Added documentation for the expanded regular-expression match variables and their behavior when no match exists.