Skip to content

Add the alignment directives x! and @! to pack and unpack (Feature #22185) - #10023

Merged
headius merged 1 commit into
jruby:ruby-4.1from
zev:pack-leb128-offset-align
Nov 11, 2026
Merged

headius merged 1 commit into
jruby:ruby-4.1from
zev:pack-leb128-offset-align

Conversation

@zev

@zev zev commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Implements the x! and @! alignment directives from Feature #22185, an item on the #9746 checklist, for both pack and unpack.

This PR started with all of the Ruby 4.1 pack/unpack changes. It was split up during review, and the rest are now:

The test_pack.rb sync will come in its own PR once these land. #10022 covers the excludes it needs.

Call CRuby 4.1 JRuby before JRuby after
[1, 2].pack("C x!4 C") "\x01\x00\x00\x00\x02" ArgumentError: '33' allowed only after types sSiIlLjJ "\x01\x00\x00\x00\x02"
"\x01\x00\x00\x00\x02".unpack("C x!i C") [1, 2] same '33' error [1, 2]
[1, 2].pack("C x!4 C", buffer: +"z") "z\x01\x00\x00\x00\x02" same '33' error "z\x01\x00\x00\x00\x02"
[1, 2].pack("C @!4 C", buffer: +"z") "z\x01\x00\x00\x02" same '33' error "z\x01\x00\x00\x02"
[1].pack("C x!s<") ArgumentError: endian modifier is not allowed for alignment same '33' error same as CRuby
[1].pack("C x!s<>") RangeError: Can't use both '<' and '>' same '33' error same as CRuby

Changes

All in core/src/main/java/org/jruby/util/Pack.java:

  • Parsing: parseAlignment ports pack_alignment_size, pack_alignment and the alignment use of pack_modifiers.
    • alignmentOf ports pack_alignof, using fixed 64-bit alignments with l_ at 8 bytes, like JRuby's native long.
    • alignmentPad ports pack_align_pad.
    • NATINT_ALIGN_CODES is annotated as MRI's natstr/endstr.
  • Main loops: the x!/@! check runs before the _/! native-modifier handling, which used to reject x! with the '33' error.
    • x! counts from the start of the buffer: contents in pack, and from offset: in unpack.
    • @! counts from the start of the string.
  • Errors: the errors and messages match MRI: missing alignment, invalid alignment, alignment too big, the endian modifier error, the unknown alignment directive, x/@ outside of string, and string size too big.
  • Endian modifier order: an endian modifier is rejected only after all modifiers are read, like MRI. That way a repeated one raises RangeError first. A comment says so.

Specs

  • 15 new examples in spec/ruby/core/array/pack/x_spec.rb and spec/ruby/core/string/unpack/x_spec.rb, guarded by ruby_version_is "4.1". They cover:
    • x! counting from buffer: and offset:
    • the _/! modifiers after the alignment type
    • every alignment error, including the x!s<> order discussed above
  • They pass on CRuby 4.1.0-dev and are skipped on 4.0.7.
  • With the parser changed to reject < immediately, the two <> examples fail, so they test the order.
  • "alignment too big" has no spec, because it depends on the size of C's unsigned long and would differ on 32-bit CRuby.

Testing

  • x_spec.rb (pack and unpack): 71 examples, 0 failures, 0 errors, interpreted and with forced JIT. Before, the 6 existing x!/@! examples errored.
  • spec/ruby/core/array/pack, spec/ruby/core/string/unpack, unpack_spec.rb, unpack1_spec.rb: 0 failures. The errors drop from 63 to 57, and there are 15 new examples.
  • test/mri/ruby/test_pack.rb: 41 tests, 0 failures.
  • Comparison with CRuby 4.1.0-dev: 1,320 cases, all identical. They cover every directive type, the _/!/</> modifiers, buffer:, offsets, and invalid and huge alignments. I ran it with all of the split pack PRs applied, because some cases use negative offsets.

Not in this PR

  • q!/Q!: other directives still reject them, because JRuby's native-modifier list has never included qQ. The error also prints the modifier as '33'. The new alignment parser accepts q!/Q! and formats its message like CRuby. I'll send the fix for the other directives as a separate PR to jruby-10.0.

🤖 Generated with Claude Code

@zev

zev commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

@headius A question about spec coverage before I mark this ready.

parseAlignment (the x!/@! parser) follows MRI's pack_modifiers and pack_alignment. It accepts </> while it parses, and only rejects an endian modifier after the loop. That order decides which error a repeated modifier raises:

Format CRuby 4.1 This PR
[1].pack("C x!s<") ArgumentError: endian modifier is not allowed for alignment same
[1].pack("C x!s<>") RangeError: Can't use both '<' and '>' same

Nothing in test_pack.rb or ruby/spec covers this. I only checked it with a local comparison script against CRuby. Raising as soon as the first < is read would pass every existing test, but x!s<> would then raise ArgumentError instead of RangeError.

Should I propose a spec for this to ruby/spec (in core/array/pack/x_spec.rb and core/string/unpack/x_spec.rb)? Or is it too much of an edge case to be worth pinning down?

@zev
zev marked this pull request as ready for review November 11, 2026 00:48
@zev

zev commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@headius This is ready for review. It covers the R/r, ^ and x!/@! pack/unpack items from #9746, plus negative offset: and an unpack1 nil fix.

There are two open questions for you. #10022 covers the five TestPack excludes, where a string is modified during pack/unpack. My comment above asks about a ruby/spec for the endian error order in x!/@!.

@headius headius left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed this, but since there's a lot of ported code I've asked Codex to do an additional review of style and contributor recommendations. There's a few changes needed to bring this up to standards that I see immediately:

  • The @CRuby annotations should include a note when they are only a part of the original method, so that in future audits we can quickly correlate them with the right code.
  • Tests should never be added to test/mri, since those tests are copied from CRuby periodically. If there are gaps in those tests, they can be submitted upstream for ruby/ruby, but otherwise we strongly prefer that Ruby behaviors gain specs in spec/ruby (for the ruby/spec repository).
  • There's a lot of features added here that probably could have been separate, smaller PRs. Large PRs are expensive to review (in time for humans and tokens for agents) and increase the chance of missing flaws.

@headius headius left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make the small traceability update in the inline comment, following the recently added CONTRIBUTING guidance.

**/
private static final String NATIVE_CODES = "sSiIlLjJ";
private static final String MAPPED_CODES = "sSiIqQjJ";
/** MRI's natstr and endstr: the types that take '_'/'!' or '<'/'>' in an x!/@! alignment. */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field explicitly ports MRI's natstr/endstr sets from pack.c. The recently merged CONTRIBUTING recommendation asks for @CRuby annotations on ported fields too. Please annotate this mapping so it is traceable during future syncs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Annotated as @CRuby("natstr") and @CRuby("endstr"), with a note that endstr is defined as natstr when the platform has long long. That is the case on every 64-bit platform JRuby models, so both are "sSiIlLqQjJ". I also replaced the "MRI's natstr and endstr" Javadoc with a description of what the field is for.

@headius headius added this to the JRuby 10.2.0.0 milestone Nov 11, 2026
@headius

headius commented Nov 11, 2026

Copy link
Copy Markdown
Member

A question about spec coverage before I mark this ready.

Any visible behavior that could reasonably be considered a "feature" of Ruby should get a spec in ruby/spec. You can commit these directly to our spec/ruby copy, which is periodically synced with the main repository.

Tests should never be added to our copy of CRuby's tests in test/mri, as mentioned in my review and the recent update to the contributor documents.

My agent will be adding some additional review comments based on that style review.

@zev zev closed this Nov 11, 2026
@zev
zev deleted the pack-leb128-offset-align branch November 11, 2026 05:48
@zev
zev restored the pack-leb128-offset-align branch November 11, 2026 05:49
@zev zev reopened this Nov 11, 2026
@zev
zev force-pushed the pack-leb128-offset-align branch from 85fbd15 to 1c2f40e Compare November 11, 2026 05:49
@zev zev changed the title Add LEB128 (R/r), offset (^) and alignment (x!/@!) directives to pack and unpack Add the alignment directives x! and @! to pack and unpack (Feature #22185) Nov 11, 2026
@zev

zev commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@headius Thanks for the review. I've split this up:

About test/mri: this PR didn't add or change tests there. test_pack.rb was copied unchanged from CRuby master. Following CONTRIBUTING, that sync will be its own PR once these land, so it only needs the five excludes from #10022. Or would you rather handle it as part of a broader suite sync?

The @CRuby notes are in each PR, and NATINT_ALIGN_CODES is annotated as natstr/endstr. Sorry for the close and reopen here: I tried renaming the branch, and GitHub closed the PR.

@zev

zev commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@headius split the pr into smaller prs. It might have lost some of your comments, but they should be addressed.

…2185)

x!N and @!N move to the next multiple of N bytes, and x!T and @!T to the
C alignment of directive T (x!i, x!l_). x! counts from the start of the
buffer: contents in pack and from the offset: argument in unpack; @!
counts from the start of the string. pack pads with NULs; unpack raises
if the boundary is past the end of the string. Alignments follow MRI on
64-bit platforms, with l_ at 8 bytes like JRuby's native long.

Adds specs for x! counting from buffer: and offset:, the modifiers
after an alignment type, and the alignment errors, including MRI's
order for a repeated endian modifier ("x!s<>" raises RangeError before
the endian modifier is rejected).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zev
zev force-pushed the pack-leb128-offset-align branch from 1c2f40e to b3b792d Compare November 11, 2026 20:16
@zev

zev commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@headius Ready for another look, along with the split-out PRs #10034 (unpack1 nil, jruby-10.0), #10035 (negative offset:), #10036 (R/r) and #10037 (^).

@headius headius left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@headius
headius merged commit 237722d into jruby:ruby-4.1 Nov 11, 2026
11 of 121 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants