Skip to content

mruby-regexp: say at each block why only a byte-indexed subject can answer - #7121

Merged
matz merged 1 commit into
mruby:masterfrom
takumin:regexp-drop-utf8-byte-case-assertions
Aug 12, 2026
Merged

mruby-regexp: say at each block why only a byte-indexed subject can answer#7121
matz merged 1 commit into
mruby:masterfrom
takumin:regexp-drop-utf8-byte-case-assertions

Conversation

@takumin

@takumin takumin commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Rebased onto master, which carries #7120. The deletions this pull request also
made are already in, so what is left is the writing and the ordering, which is
what the review comment on this pull request asked for.

Test only, and no assertion is added, removed or changed: sorting the assert
lines of the file before and after and diffing them turns up nothing.

What the comments say now

Nine blocks ask what the engine does with a byte that stands for no character,
and since #7115 and #7116 each asks a byte-indexed subject alone. Why that is
the only subject with an answer to give was not stated anywhere. The first of
these blocks now says it once:

A subject read as UTF-8 spells no character where the byte is, so what the
engine does with it is whatever a decoder happens to do with input no rule
covers, and an assertion on one pins that accident rather than the pattern.

Each block below then says in a line which of its subjects carries the byte and
why that one is byte-indexed, at the point it asks.

Three comments described a refusal this tree does not have yet: two say the
whole-character subject "goes through as it always did" and "reaches the engine
as it always did", and one says a subject read as UTF-8 "no longer reaches the
engine at all". Each is now written for what the engine does today.

Ordering

Three blocks left a whole-character assertion among the byte-indexed ones.

block what moves to the front
quantifier on an invalid multibyte literal "Ā".match(/./)
a byte that belongs to no character is a match position the three assert_nil on a whole character
overlong UTF-8 is not the character it spells the pattern side and the RFC 3629 bounds, with the byte-indexed groups below them

The comment introducing the surrogate cases said "those bounds", which named
the group that used to sit above it, so it names the bounds itself now.

a pattern byte that starts no character is a byte in a class keeps its
ordering. Every byte-indexed assertion there is written next to the whole
character carrying the same byte, and the pair is what the block is about: the
byte answers for itself, the character for the character. Grouping them apart
would separate each line comment from the assertion it describes.

Verified

  • rake test on a full-core build (MRB_UTF8_STRING through mruby-encoding):
    2254 tests, all green
  • rake test on the default gembox (no MRB_UTF8_STRING): 2061 tests, all
    green
  • prek run on the file passes

Summary by CodeRabbit

  • Tests
    • Reorganized and clarified UTF-8 regular expression test coverage.
    • Improved grouping and ordering of valid and malformed UTF-8 cases without changing expected behavior.

@matz

matz commented Aug 12, 2026

Copy link
Copy Markdown
Member

Sorry for the timing on this one: I merged #7120 shortly before this arrived, so master already carries the deletions. That was my doing rather than anything wrong with how you split it.

I compared the two before deciding what to ask for. Sorting the assert lines out of both files and diffing them turns up nothing but two trailing comments, and the number of assertions using a byte-indexed subject is 31 either way. So the substance of this pull request is already in.

What is not in is the part I would still like, which is the writing. Your comments here say, at the point each block asks, why a subject read as UTF-8 has no answer to give; #7120 left that reasoning spread across the blocks in a way that reads less clearly. The ordering differs too, keeping the byte-indexed assertions together rather than interleaved with the ones a whole character answers.

It does not apply as it stands, since the two reworked the same lines. Could you rebase it onto master? Whatever survives as a real diff after that is what I want. If that turns out to be only comments and ordering, that is fine and worth a commit of its own.

One thing worth saying about the series rather than this piece. Landing #7115 and #7116 first is what made these deletions easy to accept: every assertion dropped here already had a counterpart putting the same question to the engine through a subject that can answer it, so I could check that the byte-indexed count came out unchanged rather than read 41 deletions one at a time. Deleting tests is the part of a change that is hardest to review on trust, and building the replacement first is what turned it into a count I could verify.

…nswer

Nine blocks ask what the engine does with a byte that stands for no character,
and each of them asks a byte-indexed subject alone. Why that is the only
subject with an answer to give is not stated anywhere, and what stands in its
place is spread over the blocks and does not hold together. Two of them say the
whole-character subject "goes through as it always did" and "reaches the engine
as it always did", which describes a refusal this tree does not have; a third
says a subject read as UTF-8 "no longer reaches the engine at all", which is
not true here either.

Say it once, where the first of these blocks asks: a subject read as UTF-8
spells no character where such a byte is, so what a quantifier binds to there,
whether a class holds the byte, where a match may start and how far a walk
reads are whatever a decoder happens to do with input no rule covers, and an
assertion on one pins that accident rather than the pattern. Each block below
then says in a line which of its subjects carries the byte and why that one is
byte-indexed.

Three blocks left a whole-character assertion among the byte-indexed ones.
Move each to the front of its block, where what a UTF-8 subject is left to
answer reads on its own: `.` takes one atom, no match position opens inside a
character, and the shortest spelling on each side of the RFC 3629 bounds is one
character. That last group used to be introduced as "those bounds", which named
the surrogate cases sitting above it, so it names the bounds itself now.

No assertion is added, removed or changed.
@takumin
takumin force-pushed the regexp-drop-utf8-byte-case-assertions branch from 8b9efe2 to 7fec25d Compare August 12, 2026 13:42
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 675c008b-5ec1-43c7-946c-76f97390be5d

📥 Commits

Reviewing files that changed from the base of the PR and between 8d5ae77 and 7fec25d.

📒 Files selected for processing (1)
  • mrbgems/mruby-regexp/test/regexp_utf8.rb

📝 Walkthrough

Walkthrough

The UTF-8 regexp tests now use clearer comments, improved assertion placement, and reordered malformed and valid UTF-8 cases. Production APIs and test behavior remain unchanged.

Changes

UTF-8 regexp test coverage

Layer / File(s) Summary
UTF-8 matching and malformed-byte coverage
mrbgems/mruby-regexp/test/regexp_utf8.rb
The tests clarify byte-indexed matching, multibyte character boundaries, truncated sequences, and invalid bytes. Related assertions are grouped together, and valid UTF-8 checks precede malformed-byte checks.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: matz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding explanations at test blocks for using byte-indexed subjects.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@takumin takumin changed the title mruby-regexp: stop asking a UTF-8 subject what a byte that spells nothing does mruby-regexp: say at each block why only a byte-indexed subject can answer Aug 12, 2026
@takumin

takumin commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Rebased. What survived is the writing and the ordering, and I checked it the
same way you did: sorting the assert lines before and after and diffing them
turns up nothing.

Two things I did not carry over from the version you read.

The first is that it deleted line comments describing assertions that stay.
quantifier on an invalid multibyte literal had five of them, one per
assertion: "x" is not a continuation byte, the quantifier must not be taken
for one either, a sequence cut short emits its bytes one by one. I meant to
drop only the comments describing deleted lines and took these with them. #7120
kept them, so this rebase does too.

The second is a pattern byte that starts no character is a byte in a class,
which I left interleaved. Every byte-indexed assertion there is written next to
the whole character carrying the same byte, and each pair shares one line
comment: an escape names a byte, \u names a codepoint, an invalid leader is a
byte on both sides. Grouping the byte-indexed half at the end left each of those
comments standing over the character half alone. The other three blocks had a
single whole-character assertion sitting among byte-indexed ones with nothing
paired to it, and those moved to the front.

One thing the rebase turned up. Three comments describe a refusal this tree does
not have yet: two say the whole-character subject "goes through as it always
did" and "reaches the engine as it always did", and one says a subject read as
UTF-8 "no longer reaches the engine at all". I wrote them while the refusal was
still one change, and they read as statements about the engine as it is. They
now say what it does today.

And thank you for saying what made the deletions reviewable. I had thought of
#7115 and #7116 as groundwork to get out of the way, but a count you can check
beats 41 lines to read on trust, and that is the better reason to write them
first.

@matz
matz merged commit 27e97cf into mruby:master Aug 12, 2026
21 checks passed
@takumin
takumin deleted the regexp-drop-utf8-byte-case-assertions branch August 12, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants