Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 16 additions & 10 deletions build_config/ci/gcc-clang.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,6 @@
conf.gembox 'full-core'
conf.cc.defines += %w(MRB_GC_STRESS MRB_USE_DEBUG_HOOK)

# Widen the regexp /i flag from ASCII letters to the 1:1 Unicode case
# foldings, which it reads off the case table core already carries. The
# option is off by default because of the walks it adds over that table, so
# mruby-regexp/test/unicode_case.rb is only compiled into a build that turns
# it on, and without one here those walks ship untested. It goes on this
# build rather than a job of its own so it costs no runner; the other two
# builds in this file keep the default, which is what
# mruby-regexp/test/ascii_case.rb needs, so both sides stay covered.
conf.cc.defines << 'MRB_UNICODE_CASE'

conf.enable_test
end

Expand Down Expand Up @@ -68,3 +58,19 @@

conf.enable_test
end

MRuby::Build.new('ascii-case') do |conf|
conf.toolchain

# The one build here that indexes by character and converts case by ASCII.
# Both halves of that pair are what it covers: core's ASCII conversion, and
# the refusal mruby-regexp answers a pattern with when /i is asked for a
# folding the build has no table for. The refusal has no other home, since a
# build reading its strings as bytes has no character to refuse, so
# mruby-regexp/test/ascii_case.rb skips its assertions there.
# Tests only, for the reason byte-string gives above.
conf.gembox 'full-core'
conf.cc.defines << 'MRB_USE_ASCII_CASE'

conf.enable_test
end
1 change: 1 addition & 0 deletions doc/guides/language.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ Key compile-time macros that affect language behavior:
| `MRB_NO_FLOAT` | Remove all float support |
| `MRB_USE_FLOAT32` | Use 32-bit float instead of double |
| `MRB_UTF8_STRING` | UTF-8 strings and Unicode case |
| `MRB_USE_ASCII_CASE` | Keep UTF-8, convert case by ASCII |
| `MRB_INT32` | Force 32-bit integer |
| `MRB_INT64` | Force 64-bit integer |
| `MRB_STR_LENGTH_MAX` | Max string length (default 1MB) |
Expand Down
29 changes: 20 additions & 9 deletions doc/guides/mrbconf.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,17 +221,28 @@ end
by the same data rather than converting.
- A string read as bytes (`String#b`) converts and folds ASCII alone, and one
holding bytes that spell no character is refused with `ArgumentError`.
- The regexp `i` flag reads the same data, folding every character Unicode
pairs with one other. Without this macro it folds ASCII letters, and a
pattern holding a character that needs one of the Unicode foldings raises
`RegexpError` rather than answering as if the character had no case.
- `MRB_USE_ASCII_CASE` narrows the case half back to ASCII, leaving the
indexing.
- If it isn't defined, they only support the US-ASCII encoding.

`MRB_UNICODE_CASE`

- Widens the regexp `i` flag from ASCII letters to the Unicode case foldings
that pair one codepoint with one other, read off the case table
`MRB_UTF8_STRING` carries.
- Without it, `i` folds ASCII alone and a pattern holding a character that
needs one of those foldings raises `RegexpError` rather than answering as if
the character had no case.
- Takes `MRB_UTF8_STRING`, there being no table to read otherwise.
`MRB_USE_ASCII_CASE`

- Narrows case conversion back to ASCII, so `String#downcase`, `#upcase`,
`#capitalize`, `#swapcase` and `#casecmp?` answer for `'A'` to `'Z'` and hand
every other character back as it stands.
- Drops the Unicode case table the build would otherwise carry. That is what
the option is for: a target counting its bytes buys the UTF-8 indexing of
`MRB_UTF8_STRING` without the table beside it.
- The regexp `i` flag reads that table too, so it narrows with the rest: it
folds ASCII letters, and a pattern holding a character that needs one of the
Unicode foldings raises `RegexpError` rather than answering as if the
character had no case.
- Nothing to narrow without `MRB_UTF8_STRING`: a build reading its strings as
bytes converts ASCII alone whatever this says.

`MRB_STR_LENGTH_MAX`

Expand Down
3 changes: 2 additions & 1 deletion doc/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,8 @@ Module refinements (`refine`, `using`) are not supported in mruby.
mruby does not have an `Encoding` class. Strings are treated as
byte sequences by default. UTF-8 aware string operations can be
enabled with the `MRB_UTF8_STRING` compile flag, which is also what
makes case conversion follow Unicode rather than ASCII.
makes case conversion follow Unicode rather than ASCII; `MRB_USE_ASCII_CASE`
narrows that half back without giving up the indexing.

## Integer Precision Varies by Boxing Mode

Expand Down
19 changes: 11 additions & 8 deletions include/mruby/internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -376,11 +376,13 @@ int mrb_str_case_convert_unicode(mrb_state *mrb, mrb_value str, enum mrb_case_mo
#define mrb_str_case_convert_unicode(mrb, str, mode) (-1)
#endif

#ifdef MRB_UTF8_STRING
#if defined(MRB_UTF8_STRING) && !defined(MRB_USE_ASCII_CASE)
/* What case a character has, from the tables in unicase.c. A string is
converted through mrb_str_case_convert_unicode() above; these are for a
caller holding a codepoint rather than a string, which is mruby-regexp
under /i. */
under /i. A build converting case by ASCII compiles none of this, the
table under it being what it asked to leave behind, so a caller reaching
for one of these there is a compile error rather than a link one. */

/* Which table a character is looked up in. The last three hold a difference
rather than a mapping: title case against upper case, swapping against the
Expand All @@ -403,10 +405,11 @@ enum mrb_case_kind {
bytes it took, or 0 for a character that maps to itself. */
mrb_int mrb_uni_case_map(enum mrb_case_kind kind, uint32_t cp, char *buf);

#ifdef MRB_UNICODE_CASE
/* The foldings below are what /i reads under MRB_UNICODE_CASE, and the walks
over the table cost more than the table itself, so a build that does not
ask for them does not carry them. */
#ifdef HAVE_MRUBY_REGEXP_GEM
/* The four below are the foldings /i reads off the same table, in the two
directions a pattern needs them. A build without mruby-regexp has nothing
that reads them, so a caller reaching for one there is a compile error
rather than a link one. */

/* Simple case folding: the folded codepoint, or cp itself when it folds to
nothing else. A codepoint whose folding spells several characters (U+FB00
Expand All @@ -430,8 +433,8 @@ void mrb_uni_case_fold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
void mrb_uni_case_unfold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
#endif /* MRB_UNICODE_CASE */
#endif
#endif /* HAVE_MRUBY_REGEXP_GEM */
#endif /* MRB_UTF8_STRING && !MRB_USE_ASCII_CASE */

/* attr accessor bodies (class.c); the VM compares function pointers against
these to run attr calls without a full method-call frame */
Expand Down
3 changes: 2 additions & 1 deletion mrbgems/mruby-encoding/test/string.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# String(Ext) Test

UTF8STRING = __ENCODING__ == "UTF-8"
UNICODECASE = "\u00C4".downcase == "\u00E4"

assert('String#valid_encoding? survives what the string goes through') do
# The answer is remembered on the string, either way it came out, so every
Expand Down Expand Up @@ -486,7 +487,7 @@
assert_equal [195, 132, 66], s.upcase.bytes
assert_equal [195, 132, 98], s.capitalize.bytes
assert_equal Encoding::BINARY, s.downcase.encoding
assert_equal [195, 164, 98], "\xC3\x84B".downcase.bytes
assert_equal [195, 164, 98], "\xC3\x84B".downcase.bytes if UNICODECASE
end
end

Expand Down
63 changes: 31 additions & 32 deletions mrbgems/mruby-regexp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ its own, and the last one can still open a range: `/[\u{61 62}-z]/` is

### Flags

- `i` (`Regexp::IGNORECASE`) case-insensitive matching (ASCII, or Unicode
with `MRB_UNICODE_CASE`)
- `i` (`Regexp::IGNORECASE`) case-insensitive matching (Unicode, or ASCII
where the build converts case by ASCII)
- `m` (`Regexp::MULTILINE`) `.` matches newline; `^`/`$` match at line boundaries
- `x` (`Regexp::EXTENDED`) free-spacing mode; unescaped whitespace ignored, `#` starts comments

Expand Down Expand Up @@ -180,16 +180,17 @@ pattern analysis.
pattern's encoding and raises `RegexpError` for either spelling. A range
whose ends are a byte and a character (`[\x80-µ]`) names neither and raises
`RegexpError`.
- **ASCII case folding by default**: The `i` flag handles ASCII letters
only unless the build defines `MRB_UNICODE_CASE`, which reads the Unicode
foldings that pair one codepoint with one other off core's case table.
Without the option, a pattern holding a character that needs one of those
raises `RegexpError` rather than answering as if the character had no case;
see Configuration. A codepoint with no single counterpart to fold to (`ff` to
`ff`) is never folded by either build.
- **Case folding follows the build**: The `i` flag reads the Unicode
foldings that pair one codepoint with one other off core's case table,
which a build converting case by ASCII does not carry. There `i` folds
ASCII letters, and a pattern holding a character that needs one of those
foldings raises `RegexpError` rather than answering as if the character had
no case. What it refuses is held as ranges, which take in some uncased
characters as well; see Configuration. A codepoint with no single
counterpart to fold to (`ff` to `ff`) is never folded by either build.
- **Case-insensitive backreferences match a superset**: `\1` under `i`
folds each side and compares, so it matches where the capture and the
repeat hold the same characters in different widths (`k` and `K`).
repeat hold the same characters in different widths (`k` and ``).
CRuby declines to fold across a width change there.
- **Step limit on backtracking**: Patterns that require the
backtracking engine are subject to a step limit.
Expand Down Expand Up @@ -236,36 +237,34 @@ there.
#endif
```

Case folding beyond ASCII is opt-in, since it carries the walks over core's
case table. Define `MRB_UNICODE_CASE` to enable it:
Case folding beyond ASCII is not this gem's to configure. The table is
core's, carried by any build that defines `MRB_UTF8_STRING` without
`MRB_USE_ASCII_CASE`, and is what `String#downcase` and the four case methods
beside it read; `/i` reads the two directions it needs over that same table.
So `/i` folds what the build's own case conversion folds, and a build
converting case by ASCII has nothing for it to fold beyond ASCII either,
whether the conversion was narrowed there or the strings are read as bytes and
hold no character to fold in the first place.

```ruby
conf.cc.defines << 'MRB_UNICODE_CASE'
```

The table itself is core's, carried by any build that defines
`MRB_UTF8_STRING`, which is what `String#downcase` and the four case methods
beside it read. What this option adds is the two directions /i needs over that
table, at about 4KB of text. It therefore takes `MRB_UTF8_STRING` to do
anything: without it there is no table under the walks, and a pattern read as
bytes has no character to fold in the first place.
Where the build converts case by Unicode, `/Ā/i` matches `"ā"`, `/Σ/i` matches
`"σ"`, and `[^Ā]` under `/i` stops accepting `"ā"`.

With the option, `/Ā/i` matches `"ā"`, `/Σ/i` matches `"σ"`, and `[^Ā]` under
`/i` stops accepting `"ā"`.

Without it, those same patterns do not compile:
Where it converts by ASCII, those same patterns do not compile:

```ruby
/Ā/i # RegexpError: /i needs MRB_UNICODE_CASE for this character
/Ā/i # RegexpError: /i needs Unicode case folding for this character
```

The test is whether a character has a case folding, not whether it is
non-ASCII, so a script without case is unaffected and `/日本/i`, `/العربية/i`
and `/😀/i` go on working. Patterns like `/Ā/i` were answering wrongly rather
than narrowly before this: `[Ā]` under `/i` missed `"ā"`, and `[^Ā]` accepted
it. Reaching this error means the option is what you want.

`/k/i` matching `"K"` (U+212A) and `/s/i` matching `"ſ"` need no option.
and `/😀/i` go on working. The codepoints that do have one are held as ranges
rather than one by one, and those ranges are coarse: the uncased codepoints
inside them are refused with the rest, `ƻ` (U+01BB) among them.
Patterns like `/Ā/i` were answering wrongly rather than narrowly before this:
`[Ā]` under `/i` missed `"ā"`, and `[^Ā]` accepted it. Reaching this error
means the pattern wants a build that converts case by Unicode.

`/k/i` matching `"K"` (U+212A) and `/s/i` matching `"ſ"` need no table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7d67933: the character is U+212A now, where it had been spelled with an ASCII K since before this PR.

The backreference entry above named it the same way, contrasting k with what was meant to be the Kelvin sign as the same character in different widths. Both spellings are corrected.

Those two are the only foldings whose result is an ASCII letter, and both
builds carry them, so that folding "ASCII only" covers the whole of the
equivalence class an ASCII letter belongs to rather than the part of it that
Expand Down
11 changes: 6 additions & 5 deletions mrbgems/mruby-regexp/include/re_internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -184,11 +184,12 @@ mrb_bool mrb_re_is_word_char(uint32_t c);
#define RE_FOLD_LONG_S 0x017F /* to 's' */
#define RE_FOLD_KELVIN 0x212A /* to 'k' */

/* The Unicode foldings the option adds are core's table, which only a build
reading its strings as characters carries. The option therefore answers
where the build reads characters and nowhere else, a pattern read as bytes
having no character to fold in the first place. */
#if defined(MRB_UNICODE_CASE) && defined(MRB_UTF8_STRING)
/* The Unicode foldings /i reads are core's table, which only a build reading
its strings as characters and converting their case by Unicode carries. /i
therefore folds the way that build's own case conversion does and no other
way, a pattern read as bytes having no character to fold in the first
place. */
#if defined(MRB_UTF8_STRING) && !defined(MRB_USE_ASCII_CASE)
# define RE_UNICODE_CASE
#endif

Expand Down
14 changes: 11 additions & 3 deletions mrbgems/mruby-regexp/mrbgem.rake
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@ MRuby::Gem::Specification.new('mruby-regexp') do |spec|
spec.authors = 'mruby developers'
spec.summary = 'Regexp class (built-in NFA engine)'

# The two directions over core's case table that /i reads are compiled for
# this gem and for nothing else, and they sit in the same object as the
# mapping every build's `String#downcase` calls, so the linker brings them
# along whether or not anything calls them. Saying the gem is here is what
# lets core leave them out where it is not.
spec.build.defines << 'HAVE_MRUBY_REGEXP_GEM'

spec.add_dependency 'mruby-string-ext', :core => 'mruby-string-ext'

# Enumerator is optional: only String#gsub without a block reaches `to_enum`,
Expand Down Expand Up @@ -43,10 +50,11 @@ MRuby::Gem::Specification.new('mruby-regexp') do |spec|
# build command in the block above, so the reset that comes with it drops
# nothing.
# The pair below is what `RE_UNICODE_CASE` is defined from in re_internal.h:
# the foldings are core's table, which only a build reading characters
# carries, so the option alone does not put them within /i's reach.
# the foldings are core's table, which only a build reading its strings as
# characters carries, and only where it converts their case by Unicode.
spec.build_settings do
if build.has_define?('MRB_UNICODE_CASE') && build.has_define?('MRB_UTF8_STRING')
if build.has_define?('MRB_UTF8_STRING') &&
!build.has_define?('MRB_USE_ASCII_CASE')
spec.test_rbfiles -= ["#{spec.dir}/test/ascii_case.rb"]
else
spec.test_rbfiles -= ["#{spec.dir}/test/unicode_case.rb"]
Expand Down
4 changes: 2 additions & 2 deletions mrbgems/mruby-regexp/src/re_cased.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
** Generated by tools/gen_cased.rb from Unicode 17.0.0
** as carried by ruby 4.0.6. Do not edit by hand.
**
** A build without MRB_UNICODE_CASE refuses to compile an /i pattern
** holding one of these, rather than folding ASCII and answering wrongly.
** A build narrowing /i to ASCII refuses to compile a pattern holding one
** of these, rather than folding ASCII and answering wrongly.
** The test is whether a codepoint has a case folding, not whether it is
** non-ASCII: a script without case has nothing to fold, so /日本/i and the
** like are unaffected and stay out of the table.
Expand Down
4 changes: 2 additions & 2 deletions mrbgems/mruby-regexp/src/re_compile.c
Original file line number Diff line number Diff line change
Expand Up @@ -789,7 +789,7 @@ compile_charclass(re_compiler *c)
uint32_t lo = cc->ranges[2 * i], hi = cc->ranges[2 * i + 1];
if (lo & RE_CLASS_BYTE) continue;
if (mrb_re_needs_case_data(lo, hi)) {
compile_error(c, "/i needs MRB_UNICODE_CASE for this character class");
compile_error(c, "/i needs Unicode case folding for this character class");
}
if (lo <= RE_FOLD_LONG_S && RE_FOLD_LONG_S <= hi) class_set_bit(cc, 's');
if (lo <= RE_FOLD_KELVIN && RE_FOLD_KELVIN <= hi) class_set_bit(cc, 'k');
Expand Down Expand Up @@ -1021,7 +1021,7 @@ static mrb_bool
emit_cp_folded(re_compiler *c, uint32_t cp)
{
if (mrb_re_needs_case_data(cp, cp)) {
compile_error(c, "/i needs MRB_UNICODE_CASE for this character");
compile_error(c, "/i needs Unicode case folding for this character");
}
#ifdef RE_UNICODE_CASE
uint32_t alt[MRB_UNI_MAX_UNFOLD];
Expand Down
14 changes: 8 additions & 6 deletions mrbgems/mruby-regexp/test/ascii_case.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Only compiled into mrbtest when the build does NOT define
# MRB_UNICODE_CASE; see the gem's mrbgem.rake. With the option every
# pattern refused here compiles and matches instead.
# Only compiled into mrbtest when the build converts case by ASCII, whether by
# MRB_USE_ASCII_CASE or by reading its strings as bytes; see the gem's
# mrbgem.rake. Where it converts by Unicode, every pattern refused here
# compiles and matches instead.
assert("Regexp - /i refuses what ASCII folding cannot answer") do
# Folding ASCII and carrying on would answer wrongly rather than narrowly:
# the missing fold is a missed match in the plain and class forms, and the
Expand All @@ -23,12 +24,13 @@
# which the split leaves in the codepoint list; the ASCII half it could have
# answered on its own does not save it.
assert_raise(RegexpError) { Regexp.new("[a-Ā]", Regexp::IGNORECASE) }
# A source the option build cannot fold either is refused all the same: the
# A source a Unicode build cannot fold either is refused all the same: this
# build has no data to tell it apart from one that would have folded.
assert_raise(RegexpError) { Regexp.new("ß", Regexp::IGNORECASE) }
assert_raise(RegexpError) { Regexp.new("ff", Regexp::IGNORECASE) }
# The message names the option, so hitting this says what to do about it.
refused = "/i needs MRB_UNICODE_CASE for this character"
# The message names what is missing rather than the option that would have
# supplied it, there being two ways to reach it and no one name for both.
refused = "/i needs Unicode case folding for this character"
assert_raise_with_message(RegexpError, "#{refused}: /Ā/") do
Regexp.new("Ā", Regexp::IGNORECASE)
end
Expand Down
7 changes: 4 additions & 3 deletions mrbgems/mruby-regexp/test/unicode_case.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Only compiled into mrbtest when the build defines MRB_UNICODE_CASE;
# see the gem's mrbgem.rake. Without the option every assertion here would
# fail, since /i then folds ASCII letters and nothing else.
# Only compiled into mrbtest when the build converts case by Unicode, which is
# where a build reading its strings as characters stands unless it says
# otherwise; see the gem's mrbgem.rake. Converting by ASCII, every pattern
# below would be refused instead.
assert("Regexp - Unicode case folding under /i") do
# Every source and counterpart here lies above ASCII, so they are characters
# to fold only where the pattern and the subject are read as characters. A
Expand Down
4 changes: 2 additions & 2 deletions mrbgems/mruby-regexp/tools/gen_cased.rb
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@
** Generated by tools/gen_cased.rb from Unicode #{RbConfig::CONFIG['UNICODE_VERSION'] || 'data'}
** as carried by ruby #{RUBY_VERSION}. Do not edit by hand.
**
** A build without MRB_UNICODE_CASE refuses to compile an /i pattern
** holding one of these, rather than folding ASCII and answering wrongly.
** A build narrowing /i to ASCII refuses to compile a pattern holding one
** of these, rather than folding ASCII and answering wrongly.
** The test is whether a codepoint has a case folding, not whether it is
** non-ASCII: a script without case has nothing to fold, so /#{'日本'}/i and the
** like are unaffected and stay out of the table.
Expand Down
2 changes: 1 addition & 1 deletion mrbgems/mruby-string-ext/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -809,7 +809,7 @@ Example:
"aBcDeF".casecmp?("abcdeg") #=> false
```

On a build defining `MRB_UTF8_STRING`, folding follows Unicode, and one folding may spell a character as several:
On a build defining `MRB_UTF8_STRING`, folding follows Unicode, and one folding may spell a character as several. `MRB_USE_ASCII_CASE` narrows it back to ASCII:
Comment thread
coderabbitai[bot] marked this conversation as resolved.

```ruby
"ä".casecmp?("Ä") #=> true
Expand Down
Loading
Loading