mruby-string-ext: read String#casecmp's bytes as unsigned - #7177

String#casecmp reads each byte into an int through a plain char:

int c1 = p1[i], c2 = p2[i];
if (ISASCII(c1) && ISUPPER(c1)) c1 = TOLOWER(c1);
if (ISASCII(c2) && ISUPPER(c2)) c2 = TOLOWER(c2);
if (c1 > c2) return mrb_fixnum_value(1);
if (c1 < c2) return mrb_fixnum_value(-1);

char is signed on most targets, so a byte of 0x80 or above arrives negative and orders below every ASCII one.

"\xC3".casecmp("a")   #=> -1, and 1 in CRuby
"\xC3" <=> "a"        #=> 1

String#<=> compares the same bytes with memcmp() at src/string.c:1530, which reads them as unsigned char, so the two orderings disagree about a pair of strings neither of them folds anything in. Read the bytes as unsigned char, which is the reading memcmp() gives them.

What does not move

The fold below the read is untouched. ISASCII(c1) already kept TOLOWER() off a byte of 0x80 or above, and the sign kept it off them a second time: a negative int fails ISASCII() as surely as a large one does. Only the comparison of two bytes neither side folds changes.

Verified

The read sits above what a build indexes a string by, so both builds answer the same way. On master, ci/gcc-clang:

$ bintest/bin/mruby -e 'p "\xC3".casecmp("a"); p("\xC3" <=> "a")'
-1
1
$ byte-string/bin/mruby -e 'p "\xC3".casecmp("a"); p("\xC3" <=> "a")'
-1
1

CRuby 4.0.6 answers 1 for both.

MRUBY_CONFIG=ci/gcc-clang rake -m test, all four builds and the bintests, KO 0, Crash 0 and Warning 0. The .text of libmruby.a moves by 24 bytes down on bintest and on byte-string, and not at all on full-debug or cxx_abi.

Summary by CodeRabbit

• Bug Fixes

  • Improved String#casecmp ordering for extended byte values, ensuring consistent comparisons across platforms.
  • Prevented non-ASCII bytes from sorting incorrectly below ASCII characters.

• Tests

  • Added coverage for comparisons involving byte values at or above 0x80.