Skip to content

Fix byte order in MODEXP output and BYTE - #31

Merged
sbillig merged 2 commits into
mainfrom
fix/modexp-byte-order
Nov 16, 2026
Merged

sbillig merged 2 commits into
mainfrom
fix/modexp-byte-order

Conversation

@sbillig

@sbillig sbillig commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Two places assumed little-endian memory.

  • MODEXP: encoded stored each whole output limb as swap_bytes(value) through a *u64, which is big-endian only where memory is little-endian. Compiled for the EVM, each whole eight-byte limb came out reversed: a zero exponent with a 17-byte modulus returned 0000000000000000000100000000000000 instead of 0000000000000000000000000000000001. It now stores to_be(value).
  • BYTE: op_byte read byte 31 - i of the stack word's representation. It now picks the index by the target's byte order.

Native code is unchanged: the O2 IR of modexp::run and op_byte is the same before and after. The tests run natively on little-endian hosts and the precompiles ingot does not build for the EVM, so no test here can fail before the fix; it was found by running the EVM-target export of modexp.fe against go-ethereum's MODEXP vectors (43 of 47 disagreed, all agree now).

`encoded` stored each whole limb as `swap_bytes(value)` through a `*u64`,
which writes the limb's bytes big-endian only where memory is
little-endian. Compiled for the EVM, whose memory is big-endian, each whole
eight-byte limb of the output came out reversed: a zero exponent with a
17-byte modulus returned 0000000000000000000100000000000000 instead of
0000000000000000000000000000000001. The store now writes `to_be(value)`,
which is the same byte swap on native targets: the O2 native IR of
`modexp::run` is unchanged.

fevm's tests run natively, on a little-endian host, where the old code is
right, and the precompiles ingot does not compile for the EVM (`sha256`
uses `std::native::sha256_compress`), so no fevm test can show the bug.
The regression tests are fe-equivm-proofs' `scripts/ModexpVectors.lean`
and `scripts/ModexpDiff.lean`, which run the EVM-target export of this
code on go-ethereum's MODEXP vectors and against EVMLean: before the fix
43 of 47 vectors and 81 of 200 generated inputs disagree, after it all
47 vectors agree.
`op_byte` read byte `31 - i` of the stack word's representation, which is
the byte `i` places from the word's most significant end only where memory
is little-endian. It now reads byte `i` where memory is big-endian, by
a test of `to_le` that is a constant: on native targets the O2 IR of
`op_byte` is unchanged. Native hosts are little-endian, so no behaviour
changes there; the Osaka frame differential passes at O0, O1 and O2.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T17:30:14.607458Z f168864 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sbillig
sbillig merged commit 9c5fa91 into main Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant