Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions build_config/ci/gcc-clang.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@
conf.gembox 'full-core'
conf.cc.defines += %w(MRB_GC_STRESS MRB_USE_DEBUG_HOOK)

# Widen the regexp /i flag from ASCII letters to the 1:1 Unicode case
# foldings. The option is off by default because of the table it carries, so
# mruby-regexp/test/unicode_case.rb is only compiled into a build that turns
# it on, and without one here the generated table ships untested. It goes on
# this build rather than a job of its own so it costs no runner; the other
# two builds in this file keep the default, which is what
# mruby-regexp/test/ascii_case.rb needs, so both sides stay covered.
conf.cc.defines << 'MRB_REGEXP_UNICODE_CASE'

conf.enable_test
end

Expand Down
45 changes: 42 additions & 3 deletions mrbgems/mruby-regexp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ simulation) with backtracking fallback.

### Flags

- `i` (`Regexp::IGNORECASE`) case-insensitive matching (ASCII)
- `i` (`Regexp::IGNORECASE`) case-insensitive matching (ASCII, or Unicode
with `MRB_REGEXP_UNICODE_CASE`)
- `m` (`Regexp::MULTILINE`) `.` matches newline; `^`/`$` match at line boundaries
- `x` (`Regexp::EXTENDED`) free-spacing mode; unescaped whitespace ignored, `#` starts comments

Expand Down Expand Up @@ -132,8 +133,17 @@ pattern analysis.
Maximum 255 bytes.
- **No Unicode properties**: `\p{Alpha}`, `\p{L}`, etc. are not
supported.
- **ASCII case folding only**: The `i` flag handles ASCII letters
only.
- **ASCII case folding by default**: The `i` flag handles ASCII letters
only unless the build defines `MRB_REGEXP_UNICODE_CASE`, which adds the
Unicode foldings that pair one codepoint with one other. Without the
option, a pattern holding a character that needs one of those raises
`RegexpError` rather than answering as if the character had no case; see
Configuration. A codepoint with no single counterpart to fold to (`ff` to
`ff`) is never folded by either build.
- **Case-insensitive backreferences match a superset**: `\1` under `i`
folds each side and compares, so it matches where the capture and the
repeat hold the same characters in different widths (`k` and `K`).
CRuby declines to fold across a width change there.
- **Step limit on backtracking**: Patterns that require the
backtracking engine are subject to a step limit.
- **No inline extended mode**: `(?x)` and `(?x:...)` raise a
Expand Down Expand Up @@ -174,6 +184,35 @@ there.
#endif
```

Case folding beyond ASCII is opt-in, since it carries a table of the Unicode
foldings. Define `MRB_REGEXP_UNICODE_CASE` to enable it:

```ruby
conf.cc.defines << 'MRB_REGEXP_UNICODE_CASE'
```

It costs about 4KB of text, of which roughly 2.5KB is the table itself. With
it, `/Ā/i` matches `"ā"`, `/Σ/i` matches `"σ"`, and `[^Ā]` under `/i` stops
accepting `"ā"`.

Without it, those same patterns do not compile:

```ruby
/Ā/i # RegexpError: /i needs MRB_REGEXP_UNICODE_CASE for this character
```

The test is whether a character has a case folding, not whether it is
non-ASCII, so a script without case is unaffected and `/日本/i`, `/العربية/i`
and `/😀/i` go on working. Patterns like `/Ā/i` were answering wrongly rather
than narrowly before this: `[Ā]` under `/i` missed `"ā"`, and `[^Ā]` accepted
it. Reaching this error means the option is what you want.

`/k/i` matching `"K"` (U+212A) and `/s/i` matching `"ſ"` need no option.
Those two are the only foldings whose result is an ASCII letter, and both
builds carry them, so that folding "ASCII only" covers the whole of the
equivalence class an ASCII letter belongs to rather than the part of it that
is ASCII.

## License

MIT License. See the mruby license file for details.
Expand Down
52 changes: 52 additions & 0 deletions mrbgems/mruby-regexp/include/re_internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,58 @@ int mrb_re_utf8_charlen(const char *s, const char *end);
uint32_t mrb_re_utf8_decode(const char *s, const char *end, int *len);
mrb_bool mrb_re_is_word_char(uint32_t c);

/* The two foldings whose result is an ASCII letter. Every build carries them,
whether or not it has the Unicode table, so that folding "ASCII only" covers
the whole of the equivalence class an ASCII letter belongs to rather than
the part of it that is ASCII: without them /k/i would miss U+212A and, the
sign flipped, [^k] under /i would accept it. */
#define RE_FOLD_LONG_S 0x017F /* to 's' */
#define RE_FOLD_KELVIN 0x212A /* to 'k' */

/* Simple case folding: the folded codepoint, or cp itself when it folds to
nothing else. With MRB_REGEXP_UNICODE_CASE that is ASCII plus every 1:1
Unicode folding; without it, ASCII plus the two above. Neither build folds a
codepoint that has no single counterpart to fold to (U+FB00 to "ff"). */
uint32_t mrb_re_case_fold(uint32_t cp);

/* True when [lo, hi] holds a codepoint that carries case folding data this
build does not have. A pattern reaching one of those under /i is refused at
compile time, since folding ASCII and carrying on would answer wrongly: the
missing fold shows up as a missed match in `[X]` and, with the sign flipped,
as a false accept in `[^X]`. The test is having the data rather than being
foldable, so two kinds fall inside it that no build folds: a source whose
fold expands into several codepoints (U+FB00 to "ff"), and the uncased
neighbours the coarse ranges close over. A build with the table compiles
both and matches them literally, so what the two builds differ in there is
what they refuse rather than what they answer. A build with the data has
nothing to refuse, so the test compiles away there. The arguments are
evaluated at most once, but only by the definition that uses them, so pass
plain values. */
#ifdef MRB_REGEXP_UNICODE_CASE
#define mrb_re_needs_case_data(lo, hi) FALSE
#else
mrb_bool mrb_re_needs_case_data(uint32_t lo, uint32_t hi);
#endif

#ifdef MRB_REGEXP_UNICODE_CASE
/* Walking the table takes data only this build has. Without it the compiler
reaches the same two foldings directly, since there are only two.

mrb_re_case_unfold() writes every other codepoint sharing cp's folded form
into out, at most max of them, and returns how many it wrote. The two range
forms do the same two directions over a span rather than one codepoint,
reporting what they find by calling add() with each span of it:
mrb_re_case_fold_range the folds of the sources in [lo, hi],
mrb_re_case_unfold_range the sources of the folds in [lo, hi]. Spans may
repeat or overlap what the caller already holds; the caller merges. */
#define RE_MAX_UNFOLD 4
int mrb_re_case_unfold(uint32_t cp, uint32_t *out, int max);
void mrb_re_case_fold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
void mrb_re_case_unfold_range(uint32_t lo, uint32_t hi,
void (*add)(void *, uint32_t, uint32_t), void *user);
#endif

static inline int
mrb_re_charlen(const char *s, const char *end, mrb_bool binary)
{
Expand Down
10 changes: 10 additions & 0 deletions mrbgems/mruby-regexp/mrbgem.rake
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,14 @@ MRuby::Gem::Specification.new('mruby-regexp') do |spec|
if build.gems.any? {|g| g.name == 'mruby-symbol-ext'}
spec.add_dependency 'mruby-symbol-ext', :core => 'mruby-symbol-ext'
end

# The two case folding test files assert opposite things about the same
# patterns (one that /i folds them, the other that /i refuses to compile
# them), so each belongs to exactly one of the two builds. Everything /i
# does the same way in both is in test/regexp.rb and always runs.
if build.cc.defines.include?('MRB_REGEXP_UNICODE_CASE')
spec.test_rbfiles -= ["#{spec.dir}/test/ascii_case.rb"]
else
spec.test_rbfiles -= ["#{spec.dir}/test/unicode_case.rb"]
end
end
66 changes: 66 additions & 0 deletions mrbgems/mruby-regexp/src/re_cased.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
** re_cased.h - codepoints /i cannot answer without Unicode case data
**
** Generated by tools/gen_casefold.rb from Unicode 17.0.0
** as carried by ruby 4.0.6. Do not edit by hand.
**
** A build without MRB_REGEXP_UNICODE_CASE refuses to compile an /i pattern
** holding one of these, rather than folding ASCII and answering wrongly.
** The test is whether a codepoint has a case folding, not whether it is
** non-ASCII: a script without case has nothing to fold, so /日本/i and the
** like are unaffected and stay out of the table.
**
** U+017F and U+212A are out of the table as well, for the opposite reason:
** they fold to "s" and "k", so every build carries them and can answer.
**
** Ranges are coarse. 309 uncased codepoints fall inside them and are
** refused too, which costs a pattern ASCII folding would have got right
** and buys a table a tenth the size of the mapping.
**
** See Copyright Notice in mruby.h
*/

/* Inclusive (lo, hi) pairs, ascending and disjoint. */
static const uint32_t re_cased_ranges[][2] = {
{ 0x000B5, 0x0017E },
{ 0x00180, 0x0029E },
{ 0x00345, 0x00345 },
{ 0x00370, 0x00587 },
{ 0x010A0, 0x010FF },
{ 0x013A0, 0x013FD },
{ 0x01C80, 0x01CBF },
{ 0x01D79, 0x01D8E },
{ 0x01E00, 0x01FFC },
{ 0x02126, 0x02129 },
{ 0x0212B, 0x02132 },
{ 0x0214E, 0x0214E },
{ 0x02160, 0x02184 },
{ 0x024B6, 0x024E9 },
{ 0x02C00, 0x02D2D },
{ 0x0A640, 0x0A66D },
{ 0x0A680, 0x0A69B },
{ 0x0A722, 0x0A7DC },
{ 0x0A7F5, 0x0A7F6 },
{ 0x0AB53, 0x0AB53 },
{ 0x0AB70, 0x0ABBF },
{ 0x0FB00, 0x0FB17 },
{ 0x0FF21, 0x0FF5A },
{ 0x10400, 0x1044F },
{ 0x104B0, 0x104FB },
{ 0x10570, 0x105BC },
{ 0x10C80, 0x10CF2 },
{ 0x10D50, 0x10D85 },
{ 0x118A0, 0x118DF },
{ 0x16E40, 0x16E7F },
{ 0x16EA0, 0x16ED3 },
{ 0x1E900, 0x1E943 },
};

#define RE_CASED_RANGE_COUNT (sizeof(re_cased_ranges) / sizeof(re_cased_ranges[0]))

/* Lowest and highest codepoint covered, so a lookup that cannot hit
anything costs one comparison. */
#define RE_CASED_MIN 0x000B5
#define RE_CASED_MAX 0x1E943

/* 2982 cased codepoints in 32 ranges spanning 3291. */
Loading
Loading