Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/guides/language.md
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,7 @@ Key compile-time macros that affect language behavior:
| -------------------- | ---------------------------------- |
| `MRB_NO_FLOAT` | Remove all float support |
| `MRB_USE_FLOAT32` | Use 32-bit float instead of double |
| `MRB_UTF8_STRING` | Enable UTF-8 string handling |
| `MRB_UTF8_STRING` | UTF-8 strings and Unicode case |
| `MRB_INT32` | Force 32-bit integer |
| `MRB_INT64` | Force 64-bit integer |
| `MRB_STR_LENGTH_MAX` | Max string length (default 1MB) |
Expand Down
6 changes: 6 additions & 0 deletions doc/guides/mrbconf.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,12 @@ end
`MRB_UTF8_STRING`

- Adds UTF-8 encoding support to character-oriented String instance methods.
- Case conversion follows Unicode: `String#downcase`, `#upcase`, `#capitalize`
and `#swapcase` map every character Unicode gives a case, and a mapping may
spell several characters (`"ß".upcase` is `"SS"`). `String#casecmp?` folds
by the same data rather than converting.
- A string read as bytes (`String#b`) converts and folds ASCII alone, and one
holding bytes that spell no character is refused with `ArgumentError`.
- If it isn't defined, they only support the US-ASCII encoding.

`MRB_STR_LENGTH_MAX`
Expand Down
3 changes: 2 additions & 1 deletion doc/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,8 @@ Module refinements (`refine`, `using`) are not supported in mruby.

mruby does not have an `Encoding` class. Strings are treated as
byte sequences by default. UTF-8 aware string operations can be
enabled with the `MRB_UTF8_STRING` compile flag.
enabled with the `MRB_UTF8_STRING` compile flag, which is also what
makes case conversion follow Unicode rather than ASCII.

## Integer Precision Varies by Boxing Mode

Expand Down
26 changes: 26 additions & 0 deletions include/mruby/internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,32 @@ mrb_enc_decode(const char *p, const char *e, mrb_int *lenp)
return (uint8_t)*p;
#endif
}
/* What a case conversion makes of each character. `capitalize` asks two things
of one string, title case at the front and lower case behind it, and `swap`
asks per character, so a mode is what a method does rather than one case. */
enum mrb_case_mode {
MRB_CASE_DOWN,
MRB_CASE_UP,
MRB_CASE_CAPITALIZE,
MRB_CASE_SWAP,
/* Case folding, which is what two strings are compared under rather than
something a method hands back: it spells "ß" as "ss" so that the two
compare equal, which is no lower case of anything. */
MRB_CASE_FOLD
};

/* Convert every character of `str` in place where Unicode has something to say
about it, answering 1 if any character changed, 0 if none did, and -1 for a
string this walk is not the one to convert: nothing but ASCII, read as bytes,
or empty. A caller takes -1 as "the ASCII loop I have is the whole answer",
which is what every build without the tables answers to every string.
`swapcase` lives in mruby-string-ext and reaches the tables through this, so
they are asked about in one place. */
#ifdef MRB_UTF8_STRING
int mrb_str_case_convert_unicode(mrb_state *mrb, mrb_value str, enum mrb_case_mode mode);
#else
#define mrb_str_case_convert_unicode(mrb, str, mode) (-1)
#endif

/* attr accessor bodies (class.c); the VM compares function pointers against
these to run attr calls without a full method-call frame */
Expand Down
14 changes: 14 additions & 0 deletions mrbgems/mruby-encoding/test/string.rb
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,20 @@
end
end

assert('a byte-read string converted case') do
# Bytes read as bytes spell no characters, so a case conversion has nothing
# above ASCII to map and hands back the bytes it was given, still read as
# bytes. The same bytes read as UTF-8 spell "Ä", which does map.
if UTF8STRING
s = "\xC3\x84B".b
assert_equal [195, 132, 98], s.downcase.bytes
assert_equal [195, 132, 66], s.upcase.bytes
assert_equal [195, 132, 98], s.capitalize.bytes
assert_equal Encoding::BINARY, s.downcase.encoding
assert_equal [195, 164, 98], "\xC3\x84B".downcase.bytes
end
end

assert('a byte-read string cut in three') do
# `partition` and `rpartition` cut their pieces out of the receiver's bytes,
# so the head and the tail are read the way the receiver was. The middle
Expand Down
13 changes: 11 additions & 2 deletions mrbgems/mruby-string-ext/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ Example:

### `String#swapcase`

Returns a copy of `str` with uppercase alphabetic characters converted to lowercase and lowercase characters converted to uppercase. Effective only in ASCII region.
Returns a copy of `str` with uppercase alphabetic characters converted to lowercase and lowercase characters converted to uppercase. On a build defining `MRB_UTF8_STRING` every character Unicode gives a case is swapped, and a swap may spell a character as several (`"ß".swapcase` is `"SS"`); otherwise, and for a string read as bytes, only the ASCII region is affected.

```ruby
str.swapcase #=> new_str
Expand Down Expand Up @@ -779,7 +779,7 @@ b.delete_suffix!("hel") #=> nil

### `String#casecmp`

Case-insensitive version of `String#<=>`. Returns -1, 0, or +1. Returns `nil` if `other_str` is not a String.
Case-insensitive version of `String#<=>`. Returns -1, 0, or +1. Returns `nil` if `other_str` is not a String. Only ASCII case is ignored, whatever the build reads a string as; `casecmp?` is the one that folds Unicode.

```ruby
str.casecmp(other_str) #=> -1, 0, +1 or nil
Expand Down Expand Up @@ -809,6 +809,15 @@ Example:
"aBcDeF".casecmp?("abcdeg") #=> false
```

On a build defining `MRB_UTF8_STRING`, folding follows Unicode, and one folding may spell a character as several:

```ruby
"ä".casecmp?("Ä") #=> true
"ß".casecmp?("ss") #=> true
```

A string holding bytes that spell no character is refused with `ArgumentError`, since bytes that spell nothing have no folding. One read as bytes folds ASCII alone, having no characters to fold.

### `String#+@` (Unary Plus)

Returns `self` if `self` is not frozen. Otherwise returns a mutable (not frozen) duplicate of `self`.
Expand Down
61 changes: 58 additions & 3 deletions mrbgems/mruby-string-ext/src/string.c
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,13 @@ int_chr_utf8(mrb_state *mrb, mrb_value num)
*
* Equivalent to `String#swapcase`, but modifies the receiver in
* place, returning *str*, or `nil` if no changes were made.
* Note: case conversion is effective only in ASCII region.
*/
static mrb_value
str_swapcase_bang(mrb_state *mrb, mrb_value str)
{
int uc = mrb_str_case_convert_unicode(mrb, str, MRB_CASE_SWAP);
if (uc >= 0) return uc ? str : mrb_nil_value();

int modify = 0;
struct RString *s = mrb_str_ptr(str);

Expand Down Expand Up @@ -97,8 +99,10 @@ str_swapcase_bang(mrb_state *mrb, mrb_value str)
* str.swapcase -> new_str
*
* Returns a copy of *str* with uppercase alphabetic characters converted
* to lowercase and lowercase characters converted to uppercase.
* Note: case conversion is effective only in ASCII region.
* to lowercase and lowercase characters converted to uppercase. A build that
* reads a string as characters swaps every character Unicode gives a case,
* which can spell more characters than it was handed ("ß" to "SS"); one that
* reads it as bytes swaps ASCII letters alone.
*
* "Hello".swapcase #=> "hELLO"
* "cYbEr_PuNk11".swapcase #=> "CyBeR_pUnK11"
Expand Down Expand Up @@ -1428,16 +1432,67 @@ str_casecmp(mrb_state *mrb, mrb_value self)
}
#undef lesser

#ifdef MRB_UTF8_STRING
/* Whether a string holds anything the fold table could speak about. A string
of nothing but ASCII does not, and one read as bytes spells no characters
at all, so neither needs the walk. */
static mrb_bool
str_folds_beyond_ascii(mrb_value str)
{
struct RString *s = mrb_str_ptr(str);
return RSTR_CODERANGE(s) != MRB_STR_CODERANGE_7BIT && !RSTR_BINARY_P(s);
}

/* Fold the one side the tables have nothing to say about. Only one of the two
has to hold a character above ASCII for both to be folded, and folding is
ASCII's lower case where it is nothing more: "SS" has to reach "ss" for
`"ß".casecmp?("SS")` to be true, and the walk in core hands such a string
back untouched. */
static void
str_fold_ascii(mrb_state *mrb, mrb_value str)
{
struct RString *s = mrb_str_ptr(str);
mrb_str_modify(mrb, s);
char *p = RSTR_PTR(s);
for (char *pend = p + RSTR_LEN(s); p < pend; p++) {
if (ISUPPER(*p)) *p = TOLOWER(*p);
}
}
#endif

/*
* call-seq:
* str.casecmp?(other) -> true, false, or nil
*
* Returns true if str and other_str are equal after case folding,
* false if they are not equal, and nil if other is not a string.
*
* Folding is what makes this wider than `casecmp`, which orders strings by
* ASCII case alone: a build that reads a string as characters folds every
* character Unicode gives a folding, and one folding spells a character as
* several ("ß" as "ss").
*
* "ä".casecmp("Ä") #=> 1
* "ä".casecmp?("Ä") #=> true
* "ß".casecmp?("ss") #=> true
*/
static mrb_value
str_casecmp_p(mrb_state *mrb, mrb_value self)
{
#ifdef MRB_UTF8_STRING
mrb_value other = mrb_get_arg1(mrb);
if (!mrb_string_p(other)) return mrb_nil_value();

/* Nothing above ASCII on either side leaves nothing for the tables to fold,
and the two strings order by their bytes as they always have. */
if (str_folds_beyond_ascii(self) || str_folds_beyond_ascii(other)) {
mrb_value a = mrb_str_dup(mrb, self);
mrb_value b = mrb_str_dup(mrb, other);
if (mrb_str_case_convert_unicode(mrb, a, MRB_CASE_FOLD) < 0) str_fold_ascii(mrb, a);
if (mrb_str_case_convert_unicode(mrb, b, MRB_CASE_FOLD) < 0) str_fold_ascii(mrb, b);
return mrb_bool_value(mrb_str_equal(mrb, a, b));
}
#endif
mrb_value c = str_casecmp(mrb, self);
if (mrb_nil_p(c)) return c;
return mrb_bool_value(mrb_fixnum(c) == 0);
Expand Down
55 changes: 55 additions & 0 deletions mrbgems/mruby-string-ext/test/string.rb
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,24 @@ def assert_upto(exp, receiver, *args)
assert_equal s.swapcase, t
end

assert('String#swapcase - Unicode') do
skip unless UTF8STRING
assert_equal "äÖ", "Äö".swapcase
# A character with a lower case swaps down, one without swaps up, so a
# mapping that spells more than one character comes back here too.
assert_equal "SSa", "ßA".swapcase
assert_equal "FI", "fi".swapcase
assert_equal "I", "ı".swapcase
# A title case character swaps to what neither of its cases spells: U+01C5
# upper cases to U+01C4 and lower cases to U+01C6, and swaps to "dŽ".
assert_equal "dŽ", "Dž".swapcase
assert_equal "DŽ", "dž".swapcase
assert_equal "dž", "DŽ".swapcase
# A script without case has nothing to swap.
assert_equal "日本", "日本".swapcase
assert_nil "日本".swapcase!
end

assert('String#concat') do
assert_equal "Hello World!", "Hello " << "World" << 33
assert_equal "Hello World!", "Hello ".concat("World").concat(33)
Expand Down Expand Up @@ -298,6 +316,43 @@ def assert_upto(exp, receiver, *args)
assert_equal(-1, "a".casecmp("\xC3"))
end

assert('String#casecmp?') do
assert_true "aBcDeF".casecmp?("abcdef")
assert_false "abcdef".casecmp?("abcde")
assert_nil "abcdef".casecmp?(1)
end

assert('String#casecmp? - Unicode') do
skip unless UTF8STRING
# `casecmp` orders strings by ASCII case alone, which is CRuby's answer
# there too; `casecmp?` folds instead, so it sees past the case.
assert_equal 1, "ä".casecmp("Ä")
assert_true "ä".casecmp?("Ä")
# A folding can spell a character as several, which is what makes this
# wider than comparing one character against one.
assert_true "ß".casecmp?("ss")
assert_true "ß".casecmp?("SS")
# Only one side has to hold a character above ASCII for both to be folded,
# and the other side is folded whether or not a walk over it has already
# settled what it holds.
ss = "SS"
ss.length
assert_true "ß".casecmp?(ss)
assert_true ss.casecmp?("ß")
assert_true "fi".casecmp?("fi")
# U+212A folds to "k", so the two spell the same string folded.
assert_true "\u{212a}".casecmp?("k")
# U+0130 folds to "i" plus U+0307, which "i" alone does not match.
assert_false "İ".casecmp?("i")
assert_false "日本".casecmp?("日")
assert_true "日本".casecmp?("日本")
# Bytes that spell no character have no folding, so the comparison refuses
# them; `casecmp` orders the same bytes without asking what they spell.
assert_raise(ArgumentError) { "\xC3ABC".casecmp?("a") }
assert_equal 0, "\xC3ABC".casecmp("\xC3abc")
assert_raise(ArgumentError) { "\xC3ABC".swapcase }
end

assert('String#count') do
s = "abccdeff123"
assert_equal 0, s.count("")
Expand Down
Loading
Loading