mruby-regexp: reject a hex escape with no digit - #7260

parse_escape in re_compile.c reads \x by taking up to two hex digits and returning whatever it has, so \x followed by no digit came out as \x00. A pattern handed to Regexp.new then compiled quietly to a NUL (a regexp literal never gets this far, since the parser refuses /\x/ as a syntax error, as CRuby's does): \x{41} was a NUL and a {41} quantifier and matched 41 NUL bytes, \x and \xZ were a NUL, and [\x] held a NUL because the class reads its escapes through the same function. This is the spelling the README's "No \x{...} hex escape" note steers a reader away from, and it did not fail, it matched something else.

Regexp.new("\\x{41}") =~ "\0" * 41
# CRuby:        RegexpError (invalid hex escape: /\x{41}/)
# mruby before: 0
# mruby after:  RegexpError (invalid hex escape: /\x{41}/)

Regexp.new("\\x") =~ "\0"
# CRuby:        RegexpError (invalid hex escape: /\x/)
# mruby before: 0
# mruby after:  RegexpError (invalid hex escape: /\x/)

Regexp.new("[\\x]") =~ "\0"
# CRuby:        RegexpError (invalid hex escape: /[\x]/)
# mruby before: 0
# mruby after:  RegexpError (invalid hex escape: /[\x]/)

Regexp.new("\\x4Z") =~ "\x04Z"
# CRuby / mruby before / mruby after: 0

Fix

parse_escape raises RegexpError with CRuby's message, invalid hex escape, when the digit loop read nothing. One digit is still enough (\x4, \x4Z), as before, and the check sits in the shared function, so a class atom ([\x], [\xZ], [a-\x]) is refused by the same line. The README's limitation note now says \x{...} raises, since the brace is not a hex digit, and the escape list says \x with no digit raises.

The \x{...} form itself is still not implemented; this only makes the missing form an error rather than a silent NUL, which is what CRuby does with it too.

Testing

mrbgems/mruby-regexp/test/regexp_syntax.rb gains "a hex escape needs at least one digit": the message for \x{41}, \x, and [\x], plus \xZ, a\x, \x{}, [\xZ], [\x{41}], [a-\x], [\x-z] raising, and \x4, \x4Z, [\x4] still matching "\x04". Every pattern in the block was run against CRuby 4.0.6 first and behaves the same there.

Full suite green at every commit (one commit).

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK src/string.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# ascii-case
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c

Summary by CodeRabbit

• Bug Fixes

  • Improved regular expression validation for hexadecimal escapes.
  • Invalid forms such as bare \x, \xZ, and \x{...} now raise RegexpError.
  • Valid one- and two-digit hexadecimal escapes continue to work as expected.

• Documentation

  • Clarified supported hexadecimal and Unicode escape syntax.