Skip to content

test(lfm2): pin the warm-continue comparison to answer-region decisions - #152

Merged
Brooooooklyn merged 1 commit into
mainfrom
fix/lfm2-paged-parity-flake
Sep 15, 2026
Merged

Brooooooklyn merged 1 commit into
mainfrom
fix/lfm2-paged-parity-flake

Conversation

@Brooooooklyn

@Brooooooklyn Brooooooklyn commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

The problem, corrected

main failed e2e lfm2 on three consecutive runs (#148, #149, #150), and it is not a random flake — it is host-dependent determinism:

[warm-cont] turn1:             num_tokens=2 finish=length raw_text="</think> Ok"
[warm-cont] FLAT turn1:        num_tokens=2 finish=length raw_text="</think> Ok"     ← arms identical before the boundary
[warm-cont] turn2 WARM:        num_tokens=32 cached_tokens=23 finish=length
[warm-cont] turn2 FLAT oracle: num_tokens=32 cached_tokens=23 finish=length          ← same reuse, same length
WARM raw_text="<think> Okay, let's see. The user said to keep going from where I left off, but …"
FLAT raw_text="<think> Okay, let's see what the user is asking here. They said, \"Now …"

first_diff_byte=Some(23) is identical on all three red runs, and the same commit passes locally. So the carry path is fine, the arms are comparable, and what split is the first free-form decision inside the thinking span — the near-tie that this file already documents as flippable by the accepted ~1-ULP paged-vs-flat kernel-stack class (models/lfm2/attention.rs, and the "Deliberately NOT asserted: byte-equality of the answer texts" note on the memory-probe test). Byte parity was being asserted over a coin flip.

I first tried downgrading that assertion to a diagnostic; an adversarial review rightly rejected it — "a stale or incorrect conv tensor can preserve turn-1 output, reuse 23 tokens, and generate 32 incorrect tokens", so invariants cannot replace the bite. This is the reviewer's own recommended alternative instead.

The change (one file)

  • Turn 2 is now budget-forced, like turn 1. Both arms therefore commit the same forced </think> first and the compared tokens are answer-region ones, not a sentence-opening word choice. The boundary under test (turn-1/turn-2 carry, ContinuedLivePrefix, 23-token prefix) is untouched.
  • Byte parity stays fatal — the assertion is unchanged in strength.
  • Three invariants added, which the fixture was missing: the arms' turn-1 outputs must be equal (turn 1 is cold, so a mismatch is a fixture lottery, and the message says so instead of sending a reader into the continue path), the reuse lengths must be equal, and the committed token counts must be equal (the qwen3 sibling already asserts that one).

Verification

Check Result
Targeted test locally [PASS] … (warm cached=23, tokens=32), both arms byte-identical
Whole gate file locally 9 passed, 0 failed
New count invariant actually fires confirmed by temporarily shortening the flat oracle's turn 2 → the arms committed different token counts (warm=32 flat=31)
cargo fmt --all touched only this file
On the affected (CI) host CI dispatched on this branch: run 34939712245 — result will be posted here

What is still not proven

The compared region is now answer-region, but a near-tie there is still possible in principle, so this is a fixture pin validated on the affected host, not a proof that the split was harmless rounding. The durable instrument is the one models/qwen3/model.rs uses for its mixed-step gate: logits at turn-2's first forward within test_support::bf16_scaled_tolerance, plus top - second > 2 × tolerance. That needs a logits readout a test can reach — ChatResult carries none and Lfm2Inner is crate-private — so it is a follow-up, not something to fake with a looser text comparison here.

This clears the model-test gate that has been skipping Publish on main.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 97d110b5-7f07-43b3-90ce-adec11805063

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cursor

cursor Bot commented Sep 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_17c69d98-d0ba-46e4-871d-a819adab1a2b)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-15T07:06:41.663572Z 50fd793 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Brooooooklyn
Brooooooklyn enabled auto-merge (squash) September 15, 2026 06:58

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27adc120d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +643 to +647
eprintln!(
"[ACCEPTED-CLASS] warm-continue text drifted from the flat oracle at \
first_diff_byte={first_diff:?} (cached warm={} flat={}, tokens warm={} flat={}). \
The invariants above held, so the prefix and the committed shape agree — treat this as \
the documented ~1-ULP kernel-stack class unless they start failing too.\n\

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore a fatal carried-state correctness check

When the original run_conv_only_prefill attention-skip regression—or any other corruption that leaves cache metadata intact—returns, cached_tokens can still equal 23 and both max-token-limited decodes can still report 32 tokens, so every new assertion passes. This branch then merely logs the differing answers, making the real-model gate green despite incorrect warm-continuation state; retain a fatal oracle such as a bounded logits/state comparison or a deterministic forced-token fixture rather than accepting every output divergence as the ~1-ULP class.

Useful? React with 👍 / 👎.

`lfm2_paged_budget_forced_warm_continue_parity` reddened `main` on three
consecutive runs (#148, #149, #150) with one signature: the warm paged continue
and the flat carried-state oracle agreed for 23 bytes and then split at the
first content word after `<think>` ("keep" vs "what"), `first_diff_byte=Some(23)`
identical on every run, while the same commit passes on a developer host.

The failing run's own log rules out the carry path and the fixture: `turn1` and
`FLAT turn1` are both `"</think> Ok"`, and `cached_tokens` (23) and `num_tokens`
(32) match on both sides. What split is the first *free-form* decision inside the
thinking span — exactly the kind of near-tie the accepted ~1-ULP paged-vs-flat
kernel-stack class flips (`models/lfm2/attention.rs`, plus this file's own
"deliberately NOT asserted" note for long generations). Byte parity was being
asserted over a coin flip.

Fix: run turn 2 budget-forced like turn 1, so both arms commit the same forced
`</think>` first and the compared tokens are answer-region ones. Byte parity
stays fatal; nothing is weakened.

Also asserted now, because they were missing: the arms' turn-1 outputs are equal
(turn 1 is cold, so a mismatch is a fixture lottery and the message says so
instead of sending a reader into the continue path), the reuse lengths are equal,
and the committed token counts are equal (the qwen3 sibling already asserts that
one).

Verified: gate passes locally with both arms byte-identical (`[PASS] … warm
cached=23, tokens=32`); the count invariant was confirmed to fire by temporarily
shortening the flat oracle's turn 2 — "the arms committed different token counts
(warm=32 flat=31)".
@Brooooooklyn
Brooooooklyn force-pushed the fix/lfm2-paged-parity-flake branch from 27adc12 to 50fd793 Compare September 15, 2026 07:02
@cursor

cursor Bot commented Sep 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_bb02f631-49ad-47ea-a387-8b2c633469e5)

@Brooooooklyn Brooooooklyn changed the title test(lfm2): gate the warm-continue boundary on invariants, not on free-form text test(lfm2): pin the warm-continue comparison to answer-region decisions Sep 15, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 50fd7932e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +618 to +622
assert_eq!(
r2_warm.num_tokens, r2_flat.num_tokens,
"the arms committed different token counts (warm={} flat={})",
r2_warm.num_tokens, r2_flat.num_tokens,
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require turn 2 to consume the intended token budget

With the new zero thinking budget, turn 2's first token is forced to </think> independently of the carried state, but this assertion only requires the two arms to return the same count. If both runs stop early on EOS or a repetition cutoff, they can pass after little or no answer-region generation, substantially weakening the regression check this change relies on. Assert that both turns finish by length and commit exactly MAX_NEW_TURN2 tokens, as the test already does for turn 1.

Useful? React with 👍 / 👎.

@Brooooooklyn
Brooooooklyn merged commit b8da5be into main Sep 15, 2026
4 of 9 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/lfm2-paged-parity-flake branch September 15, 2026 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant