Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions build_config/ci/gcc-clang.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,14 @@
conf.cc.defines += %w(MRB_GC_STRESS MRB_USE_DEBUG_HOOK)

# Widen the regexp /i flag from ASCII letters to the 1:1 Unicode case
# foldings. The option is off by default because of the table it carries, so
# foldings, which it reads off the case table core already carries. The
# option is off by default because of the walks it adds over that table, so
# mruby-regexp/test/unicode_case.rb is only compiled into a build that turns
# it on, and without one here the generated table ships untested. It goes on
# this build rather than a job of its own so it costs no runner; the other
# two builds in this file keep the default, which is what
# it on, and without one here those walks ship untested. It goes on this
# build rather than a job of its own so it costs no runner; the other two
# builds in this file keep the default, which is what
# mruby-regexp/test/ascii_case.rb needs, so both sides stay covered.
conf.cc.defines << 'MRB_REGEXP_UNICODE_CASE'
conf.cc.defines << 'MRB_UNICODE_CASE'

conf.enable_test
end
Expand Down
2 changes: 1 addition & 1 deletion doc/guides/language.md
Original file line number Diff line number Diff line change
Expand Up @@ -420,7 +420,7 @@ Key compile-time macros that affect language behavior:
| -------------------- | ---------------------------------- |
| `MRB_NO_FLOAT` | Remove all float support |
| `MRB_USE_FLOAT32` | Use 32-bit float instead of double |
| `MRB_UTF8_STRING` | Enable UTF-8 string handling |
| `MRB_UTF8_STRING` | UTF-8 strings and Unicode case |
| `MRB_INT32` | Force 32-bit integer |
| `MRB_INT64` | Force 64-bit integer |
| `MRB_STR_LENGTH_MAX` | Max string length (default 1MB) |
Expand Down
16 changes: 16 additions & 0 deletions doc/guides/mrbconf.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,8 +215,24 @@ end
`MRB_UTF8_STRING`

- Adds UTF-8 encoding support to character-oriented String instance methods.
- Case conversion follows Unicode: `String#downcase`, `#upcase`, `#capitalize`
and `#swapcase` map every character Unicode gives a case, and a mapping may
spell several characters (`"ß".upcase` is `"SS"`). `String#casecmp?` folds
by the same data rather than converting.
- A string read as bytes (`String#b`) converts and folds ASCII alone, and one
holding bytes that spell no character is refused with `ArgumentError`.
- If it isn't defined, they only support the US-ASCII encoding.

`MRB_UNICODE_CASE`

- Widens the regexp `i` flag from ASCII letters to the Unicode case foldings
that pair one codepoint with one other, read off the case table
`MRB_UTF8_STRING` carries.
- Without it, `i` folds ASCII alone and a pattern holding a character that
needs one of those foldings raises `RegexpError` rather than answering as if
the character had no case.
- Takes `MRB_UTF8_STRING`, there being no table to read otherwise.

`MRB_STR_LENGTH_MAX`

- The maximum length of strings (default 1048576).
Expand Down
3 changes: 2 additions & 1 deletion doc/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,8 @@ Module refinements (`refine`, `using`) are not supported in mruby.

mruby does not have an `Encoding` class. Strings are treated as
byte sequences by default. UTF-8 aware string operations can be
enabled with the `MRB_UTF8_STRING` compile flag.
enabled with the `MRB_UTF8_STRING` compile flag, which is also what
makes case conversion follow Unicode rather than ASCII.

## Integer Precision Varies by Boxing Mode

Expand Down
83 changes: 83 additions & 0 deletions include/mruby/internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,89 @@ mrb_enc_decode(const char *p, const char *e, mrb_int *lenp)
return (uint8_t)*p;
#endif
}
/* What a case conversion makes of each character. `capitalize` asks two things
of one string, title case at the front and lower case behind it, and `swap`
asks per character, so a mode is what a method does rather than one case. */
enum mrb_case_mode {
MRB_CASE_DOWN,
MRB_CASE_UP,
MRB_CASE_CAPITALIZE,
MRB_CASE_SWAP,
/* Case folding, which is what two strings are compared under rather than
something a method hands back: it spells "ß" as "ss" so that the two
compare equal, which is no lower case of anything. */
MRB_CASE_FOLD
};

/* Convert every character of `str` in place where Unicode has something to say
about it, answering 1 if any character changed, 0 if none did, and -1 for a
string this walk is not the one to convert: nothing but ASCII, read as bytes,
or empty. A caller takes -1 as "the ASCII loop I have is the whole answer",
which is what every build without the tables answers to every string.
`swapcase` lives in mruby-string-ext and reaches the tables through this, so
they are asked about in one place. */
#ifdef MRB_UTF8_STRING
int mrb_str_case_convert_unicode(mrb_state *mrb, mrb_value str, enum mrb_case_mode mode);
#else
#define mrb_str_case_convert_unicode(mrb, str, mode) (-1)
#endif

#ifdef MRB_UTF8_STRING
/* What case a character has, from the tables in unicase.c. A string is
converted through mrb_str_case_convert_unicode() above; these are for a
caller holding a codepoint rather than a string, which is mruby-regexp
under /i. */

/* Which table a character is looked up in. The last three hold a difference
rather than a mapping: title case against upper case, swapping against the
rule that a character with a lower case swaps down, and folding against the
lowercase mapping. */
enum mrb_case_kind {
MRB_CASE_KIND_LOWER,
MRB_CASE_KIND_UPPER,
MRB_CASE_KIND_TITLE,
MRB_CASE_KIND_SWAP,
MRB_CASE_KIND_FOLD
};

/* The buffer mrb_uni_case_map() writes into. A mapping may spell several
characters, so this is wider than one of them; unicase.c asserts that the
table it carries fits. */
#define MRB_UNI_CASE_MAX_BYTES 8

/* The `kind` mapping of `cp`, written into `buf` as UTF-8, answering how many
bytes it took, or 0 for a character that maps to itself. */
mrb_int mrb_uni_case_map(enum mrb_case_kind kind, uint32_t cp, char *buf);

#ifdef MRB_UNICODE_CASE
/* The foldings below are what /i reads under MRB_UNICODE_CASE, and the walks
over the table cost more than the table itself, so a build that does not
ask for them does not carry them. */

/* Simple case folding: the folded codepoint, or cp itself when it folds to
nothing else. A codepoint whose folding spells several characters (U+FB00
to "ff") folds to itself here, which is what makes this the simple folding
rather than the full one mrb_uni_case_map() answers with. */
uint32_t mrb_uni_case_fold(uint32_t cp);

/* At most this many codepoints share one folded form. */
#define MRB_UNI_MAX_UNFOLD 4

/* Write every other codepoint sharing cp's folded form into out, at most max
of them, and answer how many were written. */
int mrb_uni_case_unfold(uint32_t cp, uint32_t *out, int max);

/* The same two directions over a span rather than one codepoint, reporting
what they find by calling add() with each span of it: fold_range the folds
of the sources in [lo, hi], unfold_range the sources of the folds in
[lo, hi]. Spans may repeat or overlap what the caller already holds; the
caller merges. */
void mrb_uni_case_fold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
void mrb_uni_case_unfold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
#endif /* MRB_UNICODE_CASE */
#endif

/* attr accessor bodies (class.c); the VM compares function pointers against
these to run attr calls without a full method-call frame */
Expand Down
1 change: 1 addition & 0 deletions lib/mruby/amalgam.rb
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ class Amalgam
array.c
hash.c
string.c
unicase.c
range.c
numeric.c
numops.c
Expand Down
14 changes: 14 additions & 0 deletions mrbgems/mruby-encoding/test/string.rb
Original file line number Diff line number Diff line change
Expand Up @@ -476,6 +476,20 @@
end
end

assert('a byte-read string converted case') do
# Bytes read as bytes spell no characters, so a case conversion has nothing
# above ASCII to map and hands back the bytes it was given, still read as
# bytes. The same bytes read as UTF-8 spell "Ä", which does map.
if UTF8STRING
s = "\xC3\x84B".b
assert_equal [195, 132, 98], s.downcase.bytes
assert_equal [195, 132, 66], s.upcase.bytes
assert_equal [195, 132, 98], s.capitalize.bytes
assert_equal Encoding::BINARY, s.downcase.encoding
assert_equal [195, 164, 98], "\xC3\x84B".downcase.bytes
end
end

assert('a byte-read string cut in three') do
# `partition` and `rpartition` cut their pieces out of the receiver's bytes,
# so the head and the tail are read the way the receiver was. The middle
Expand Down
32 changes: 19 additions & 13 deletions mrbgems/mruby-regexp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ its own, and the last one can still open a range: `/[\u{61 62}-z]/` is
### Flags

- `i` (`Regexp::IGNORECASE`) case-insensitive matching (ASCII, or Unicode
with `MRB_REGEXP_UNICODE_CASE`)
with `MRB_UNICODE_CASE`)
- `m` (`Regexp::MULTILINE`) `.` matches newline; `^`/`$` match at line boundaries
- `x` (`Regexp::EXTENDED`) free-spacing mode; unescaped whitespace ignored, `#` starts comments

Expand Down Expand Up @@ -181,11 +181,11 @@ pattern analysis.
whose ends are a byte and a character (`[\x80-µ]`) names neither and raises
`RegexpError`.
- **ASCII case folding by default**: The `i` flag handles ASCII letters
only unless the build defines `MRB_REGEXP_UNICODE_CASE`, which adds the
Unicode foldings that pair one codepoint with one other. Without the
option, a pattern holding a character that needs one of those raises
`RegexpError` rather than answering as if the character had no case; see
Configuration. A codepoint with no single counterpart to fold to (`ff` to
only unless the build defines `MRB_UNICODE_CASE`, which reads the Unicode
foldings that pair one codepoint with one other off core's case table.
Without the option, a pattern holding a character that needs one of those
raises `RegexpError` rather than answering as if the character had no case;
see Configuration. A codepoint with no single counterpart to fold to (`ff` to
`ff`) is never folded by either build.
- **Case-insensitive backreferences match a superset**: `\1` under `i`
folds each side and compares, so it matches where the capture and the
Expand Down Expand Up @@ -236,21 +236,27 @@ there.
#endif
```

Case folding beyond ASCII is opt-in, since it carries a table of the Unicode
foldings. Define `MRB_REGEXP_UNICODE_CASE` to enable it:
Case folding beyond ASCII is opt-in, since it carries the walks over core's
case table. Define `MRB_UNICODE_CASE` to enable it:

```ruby
conf.cc.defines << 'MRB_REGEXP_UNICODE_CASE'
conf.cc.defines << 'MRB_UNICODE_CASE'
```

It costs about 4KB of text, of which roughly 2.5KB is the table itself. With
it, `/Ā/i` matches `"ā"`, `/Σ/i` matches `"σ"`, and `[^Ā]` under `/i` stops
accepting `"ā"`.
The table itself is core's, carried by any build that defines
`MRB_UTF8_STRING`, which is what `String#downcase` and the four case methods
beside it read. What this option adds is the two directions /i needs over that
table, at about 4KB of text. It therefore takes `MRB_UTF8_STRING` to do
anything: without it there is no table under the walks, and a pattern read as
bytes has no character to fold in the first place.

With the option, `/Ā/i` matches `"ā"`, `/Σ/i` matches `"σ"`, and `[^Ā]` under
`/i` stops accepting `"ā"`.

Without it, those same patterns do not compile:

```ruby
/Ā/i # RegexpError: /i needs MRB_REGEXP_UNICODE_CASE for this character
/Ā/i # RegexpError: /i needs MRB_UNICODE_CASE for this character
```

The test is whether a character has a case folding, not whether it is
Expand Down
46 changes: 17 additions & 29 deletions mrbgems/mruby-regexp/include/re_internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -184,10 +184,19 @@ mrb_bool mrb_re_is_word_char(uint32_t c);
#define RE_FOLD_LONG_S 0x017F /* to 's' */
#define RE_FOLD_KELVIN 0x212A /* to 'k' */

/* The Unicode foldings the option adds are core's table, which only a build
reading its strings as characters carries. The option therefore answers
where the build reads characters and nowhere else, a pattern read as bytes
having no character to fold in the first place. */
#if defined(MRB_UNICODE_CASE) && defined(MRB_UTF8_STRING)
# define RE_UNICODE_CASE
#endif

/* Simple case folding: the folded codepoint, or cp itself when it folds to
nothing else. With MRB_REGEXP_UNICODE_CASE that is ASCII plus every 1:1
Unicode folding; without it, ASCII plus the two above. Neither build folds a
codepoint that has no single counterpart to fold to (U+FB00 to "ff"). */
nothing else. With RE_UNICODE_CASE that is ASCII plus every 1:1 Unicode
folding, read off core's table; without it, ASCII plus the two above.
Neither build folds a codepoint that has no single counterpart to fold to
(U+FB00 to "ff"). */
uint32_t mrb_re_case_fold(uint32_t cp);

/* True when [lo, hi] holds a codepoint that carries case folding data this
Expand All @@ -203,38 +212,17 @@ uint32_t mrb_re_case_fold(uint32_t cp);
nothing to refuse, so the test compiles away there. The arguments are
evaluated at most once, but only by the definition that uses them, so pass
plain values. */
#ifdef MRB_REGEXP_UNICODE_CASE
#ifdef RE_UNICODE_CASE
#define mrb_re_needs_case_data(lo, hi) FALSE
#else
mrb_bool mrb_re_needs_case_data(uint32_t lo, uint32_t hi);
#endif

#ifdef MRB_REGEXP_UNICODE_CASE
/* Walking the table takes data only this build has. Without it the compiler
reaches the same two foldings directly, since there are only two.

mrb_re_case_unfold() writes every other codepoint sharing cp's folded form
into out, at most max of them, and returns how many it wrote. The two range
forms do the same two directions over a span rather than one codepoint,
reporting what they find by calling add() with each span of it:
mrb_re_case_fold_range the folds of the sources in [lo, hi],
mrb_re_case_unfold_range the sources of the folds in [lo, hi]. Spans may
repeat or overlap what the caller already holds; the caller merges. */
#define RE_MAX_UNFOLD 4
int mrb_re_case_unfold(uint32_t cp, uint32_t *out, int max);
void mrb_re_case_fold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
void mrb_re_case_unfold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
#endif
/* Walking a table takes data only the option build has, and there the table is
core's: mrb_uni_case_unfold() and the two range walks beside it in
mruby/internal.h are what the compiler reaches for. Without the option the
compiler reaches the same two foldings directly, since there are only two. */

/* The byte length of the character at `s`, and its codepoint: every read of a
run of bytes in this gem goes through these two. A subject handed over as
binary is one character per byte; everything else is whatever core says a
run of bytes spells, which is one character per byte too on a build that
indexes Strings by byte. So the engine reads pattern and subject the way
the build reads a String, and the compiler passes FALSE for `binary`, a
pattern being read that same way. */
static inline int
mrb_re_charlen(const char *s, const char *end, mrb_bool binary)
{
Expand Down
5 changes: 4 additions & 1 deletion mrbgems/mruby-regexp/mrbgem.rake
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,11 @@ MRuby::Gem::Specification.new('mruby-regexp') do |spec|
# had its say, which is what `build_settings` waits for; this gem sets no
# build command in the block above, so the reset that comes with it drops
# nothing.
# The pair below is what `RE_UNICODE_CASE` is defined from in re_internal.h:
# the foldings are core's table, which only a build reading characters
# carries, so the option alone does not put them within /i's reach.
spec.build_settings do
if build.has_define?('MRB_REGEXP_UNICODE_CASE')
if build.has_define?('MRB_UNICODE_CASE') && build.has_define?('MRB_UTF8_STRING')
spec.test_rbfiles -= ["#{spec.dir}/test/ascii_case.rb"]
else
spec.test_rbfiles -= ["#{spec.dir}/test/unicode_case.rb"]
Expand Down
4 changes: 2 additions & 2 deletions mrbgems/mruby-regexp/src/re_cased.h
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
/*
** re_cased.h - codepoints /i cannot answer without Unicode case data
**
** Generated by tools/gen_casefold.rb from Unicode 17.0.0
** Generated by tools/gen_cased.rb from Unicode 17.0.0
** as carried by ruby 4.0.6. Do not edit by hand.
**
** A build without MRB_REGEXP_UNICODE_CASE refuses to compile an /i pattern
** A build without MRB_UNICODE_CASE refuses to compile an /i pattern
** holding one of these, rather than folding ASCII and answering wrongly.
** The test is whether a codepoint has a case folding, not whether it is
** non-ASCII: a script without case has nothing to fold, so /日本/i and the
Expand Down
Loading
Loading