mruby-string-ext: carry String#succ across characters and step UTF-8 by code point - #7266

String#succ / succ! / next in mruby-string-ext walked the string by bytes whatever the build read it as, and stepped a single non-alphanumeric byte and stopped. A wrap never carried into the byte before it, an alphanumeric run never carried across a non-alphanumeric one, and the last byte of a multibyte UTF-8 character went up on its own, which spells no character.

# CRuby 4.0.6                                 mruby before             mruby after
"a-z".succ            #=> "b-a"               "a-aa"                   "b-a"
"1.9".succ            #=> "2.0"               "1.10"                   "2.0"
"1-z".succ            #=> "1-aa"              "1-aa"                   "1-aa"
"\xff\xff".b.succ     #=> "\x01\x00\x00"      "\x01\xFF\x00"           "\x01\x00\x00"
"a\xff".b.succ        #=> "b\xFF"             "b\xFF"                  "b\xFF"
"\x7f".succ           #=> "\x01\x00"          "\x80"                   "\x01\x00"
"ÿ".succ              #=> "Ā"                 "\xC3\xC0"               "Ā"
"aÿ".succ             #=> "aĀ"                "bÿ"                     "aĀ"
"あ".succ             #=> "ぃ"                 "ぃ"                     "ぃ"
"\u{7FF}".succ        #=> "\x01\u{80}"        "\xDF\xC0"               "\x01\u{80}"
"\u{FFFF}".succ       #=> "\x01\u{800}"       "\xEF\xBF\xC0"           "\x01\u{800}"
"\u{10FFFF}".succ     #=> "\x01\u{10000}"     "\xF4\x8F\xBF\xC0"       "\x01\u{10000}"
"\u{D7FF}".succ       #=> "\u{E000}"          "\xED\x9F\xC0"           "\u{E000}"
"\xff".succ           #=> "\x01\xFF"          "\x01\x00"               "\x01\xFF"
"a\xff".succ          #=> "b\xFF"             "b\xFF"                  "b\xFF"

The cause is str_succ_bang(): it looked for the last ASCII alphanumeric, and on any other byte it did (*e)++ (or 0xff to 0x00 with a "\x01" prefix) and break, so nothing to the left was ever reached, and a wrapped alphanumeric run only carried into the byte directly before it if that was alphanumeric too.

Fix

str_succ_bang() is rewritten after CRuby's str_succ() (enc_succ_char() / enc_succ_alnum_char()), on the UTF-8 helpers core already has (mrb_utf8_char_head(), mrb_utf8len(), mrb_utf8_decode(), mrb_utf8_to_buf()).

• The walk goes character by character from the end: by byte for a binary string (RSTR_BINARY_P) and on a build without MRB_UTF8_STRING, by UTF-8 character otherwise. A run of bytes that spells no character is stepped over without being touched, as CRuby does.
• The rightmost ASCII alphanumeric steps. One that wraps ('9', 'z', 'Z') carries into the alphanumeric before it across whatever is not one, except that a letter does not carry into a digit nor a digit into a letter across such a gap ("1.9" is "2.0", "a-z" is "b-a", "1-z" is "1-aa").
• A string with no alphanumeric steps its last character instead: a byte to the next byte, 0xFF wrapping to 0x00; a UTF-8 character to the next code point of the same byte length, skipping the surrogates, and wrapping to the first character of that length (U+0000, U+0080, U+0800, U+10000) where the next would take one more.
• When everything that could carry has wrapped, the carry ("1", "a", "A", or "\x01") goes in before the leftmost character that did, with one mrb_str_resize() and one memmove(); the old code built the result in a second string and copied it back.

CRuby asks the encoding what a letter or digit is, so a Unicode letter steps within its script there and wraps at the script's end ("ת".succ is "אא", "９".succ is "１０"). mruby has no table of the letters. It has one of what is not a letter: succ_symbol_bmp and succ_symbol_smp in src/string.c list runs of code points above ASCII that are neither letter nor digit, over the punctuation, symbol and mark blocks beside Latin and CJK text (Latin-1 Supplement, Combining Diacritical Marks, General Punctuation through Miscellaneous Symbols and Arrows, Supplemental Punctuation through CJK Symbols and Punctuation, the kana marks, Enclosed CJK Letters and Months, the variation selectors, the fullwidth punctuation, Specials), the Private Use Area, and the emoji of plane 1; from plane 14 up (the tags, the variation selectors supplement, the private use planes) a comparison answers. Each run holds only code points that CRuby's succ (Unicode 17.0.0) steps as neither, and reaches as far as that holds; what CRuby steps inside these blocks (々 〆 〇, the Suzhou numerals, the letterlike symbols, the Roman numerals, the circled letters) lies between two runs. 37 runs, 148 bytes as uint16_t pairs, in a UTF-8 build; a build without MRB_UTF8_STRING compiles none of it. Which is checkable from CRuby alone:

steps = ->(c) { s = c.chr("UTF-8"); ("9" + s).succ != "10" + s }
# for each run [a, b]: (a..b).none?(&steps) && steps[a - 1] && steps[b + 1]
# and (0xE0000..0x10FFFF).none?(&steps)

succ_alnum() consults the table before the same-byte-length test. A character in a run is not a letter, and the walk passes over it to the letter or digit before it. A character outside the runs steps as a letter when the next code point, or the one after it over a run of one, is a character of the same byte length outside the runs. That gives CRuby's answer for "a、", "a😀", "e\u0301", "aÿ", "1あ", "Ö" (over "×" to "Ø", as enc_succ_alnum_char() does with max_gaps), and every character whose successor is a letter of the same script. A letter with a run after it is at its script's end; CRuby wraps it to the script's start, which this build cannot name, so the letter stays and the walk goes on to the left. Where CRuby wraps at a script's end the table does not reach, or skips a gap in a script, mruby steps to the code point past the end or in the gap:

"a、".succ   # CRuby: "b、", mruby: "b、"
"aｚ".succ   # CRuby: "bａ", mruby: "bｚ"
"ת".succ    # CRuby: "אא", mruby: "\u05EB"

This is recorded in the README and in a comment on succ_alnum(), which say which characters step and which carry.

succ! on an empty receiver returned before mrb_str_modify(), so "".freeze.succ! came back unchanged where CRuby raises FrozenError; the check now comes first (second commit). The README states where the carry stops (third commit). The table, and the two rules it gives, are the fourth commit, after review; the fifth adds the private use areas and plane 14 to it.

Testing

mrbgems/mruby-string-ext/test/string.rb:

• String#succ: "\xff".succ was pinned to "\x01\x00" for every build; that is the byte answer, and a UTF-8 string spelling no character keeps its bytes and takes the carry in front, so the line moved into the encoding-branched block below. Added the carry-across-gap cases ("a-z", "1.9", "-9", "1-z", "a-9", "9-z", "Zz", "zz99", "***").
• String#succ: "".freeze.succ! and "a".freeze.succ! raise FrozenError.
• New String#succ steps a string with no alphanumeric by character: binary strings in every build ("\xff".b, "\xff\xff".b, "\x7f".b, "a\xff".b, "z\xff".b, "ÿ".b, succ!); under UTF8STRING the code point cases ("ÿ", "aÿ", "あ", "1あ", "\u{80}", "\u{D7FF}", "\x7f", "\x7f\x7f", "\u{7FF}", "\u{FFFF}", "\u{10FFFF}", each with an ASCII letter in front, broken "\xff", "a\xff", "z\xff", a Range#to_a over "ÿ".."ā", succ!); otherwise the byte answers for the same literals.
• New String#succ steps over punctuation and symbols above ASCII (UTF8STRING only): the walk passes over punctuation, symbols, combining marks, the variation selector, ZWJ, the emoji, the byte order mark and U+FFFD to the letter or digit before them ("a、", "9、", "z、", "1、9", "・§z", "\u{80}z", "a«", "a¿z", "a×", "a€", "a！", "a㈱", "a⼀", "e\u0301", "a😀", "a🇯🇵", "a👍🏽", "a\u{E000}", "a\u{E0100}", "\u{845B}\u{E0100}", a subdivision flag, "a\u{F0000}", "a\u{10FFFD}", a Range#to_a over "a、".."c、"); a letter steps over one symbol ("Ö", "ö", "aÖ"); the letters inside these blocks step ("a々", "a〆", "aⅠ", "aℊ", "aⓐ", "a🅰", "aａ", "a０"); a letter at its run's end stays ("aｚ", "a〇"); with no letter or digit the last character steps ("、", "😀", "×"); succ!.

Every expected value was taken from CRuby 4.0.6. A 6000-case random comparison against CRuby (3000 binary strings over ASCII alnum / punctuation / 0x00 / 0x7f / 0xff / stray UTF-8 bytes, 3000 UTF-8 strings over ASCII, kana, ÿ, the four length-boundary code points, U+D7FF, U+0800, U+10000, U+10001, 0x7f, 0x00, broken bytes) matched byte for byte.

A second comparison, in the shape of the review's: 600 strings of 1 to 4 characters per pool, built with chr("UTF-8") on both sides, diverging lines against CRuby 4.0.6. The ASCII pool is a z A Z 0 9 m 5 - . _; every other pool holds those as well as its own characters, so that the non-ASCII characters sit beside letters and digits.

The 41 and 32 that remain are ת, which CRuby wraps to "אא"; the 129 are ｚ, ９ and 〇 at their run's end, put in that pool for the purpose, which CRuby wraps and this build leaves.

Cost, size -A of mruby-string-ext string.o in the bintest build (-O3, MRB_UTF8_STRING), commit 3 to commit 5: .rodata +156 (37 runs at 4 bytes, and alignment), .text +192 (a linear scan and the plane 14 comparison; a binary search was 64 bytes more). The byte-string build compiles none of it.

Five commits; full suite green at each (the table is the tip; the third commit changes only the README).

Environment

# full-debug
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -g3 -O0 -DMRB_GC_STRESS -DMRB_USE_DEBUG_HOOK -DMRB_DEBUG -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# bintest
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_GC_FIXED_ARENA -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER -DMRB_USE_DEBUG_HOOK src/string.c
# cxx_abi
gcc -g -O3 -Wall -Wundef -Wwrite-strings -x c++ -std=gnu++03 -DMRB_GC_FIXED_ARENA -DMRB_USE_CXX_EXCEPTION -DMRB_USE_CXX_ABI -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# byte-string
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c
# ascii-case
gcc -std=gnu99 -g -O3 -Wall -Wundef -Werror-implicit-function-declaration -Wwrite-strings -DMRB_USE_ASCII_CASE -DMRB_USE_BIGINT -DMRB_USE_COMPLEX -DHAVE_MRUBY_ENCODING_GEM -DMRB_UTF8_STRING -DHAVE_MRUBY_IO_GEM -DMRB_USE_RATIONAL -DHAVE_MRUBY_REGEXP_GEM -DMRB_USE_SET -DMRB_USE_TASK_SCHEDULER src/string.c

Summary by CodeRabbit

• Enhancements

  • Improved String#succ and String#next behavior for alphanumeric carry rules, punctuation, symbols, Unicode characters, binary data, and invalid UTF-8 sequences.
  • Added more consistent handling when characters wrap or require carry insertion.
  • Expanded String#succ! behavior across UTF-8, binary strings, frozen strings, and additional edge cases.

• Documentation

  • Added guidance and examples covering encoding-specific string successor behavior.