Skip to content

fix: address book bucket overflow, decoder panic, stale rollback caches, protoio byte reader, negative-number queries - #6043

Open
gomesalexandre wants to merge 3 commits into
cometbft:mainfrom
gomesalexandre:fix_pex_libs_housekeeping
Open

fix: address book bucket overflow, decoder panic, stale rollback caches, protoio byte reader, negative-number queries#6043
gomesalexandre wants to merge 3 commits into
cometbft:mainfrom
gomesalexandre:fix_pex_libs_housekeeping

Conversation

@gomesalexandre

Copy link
Copy Markdown
Contributor

What it says on the box

Five independent, small correctness bugs found while auditing peer-discovery bookkeeping and a handful of internal libraries — none of these are crashes or DoS vectors on their own. (A separate, genuinely security-sensitive nil-pointer-dereference finding in the mempool recheck path came out of the same audit; it is intentionally not in this PR and is being routed through coordinated disclosure instead.)

p2p/pex — address book bucket capacity + overflow-recovery losing addresses

addToNewBucket/addToOldBucket used > instead of >= against their bucket-size constants (newBucketSize/oldBucketSize, both 64), so a bucket already at capacity would still admit a 65th entry.

Worse: when moveToOld's overflow-recovery path demotes the oldest entry of a full old bucket back to a new bucket, removeFromBucket never resets the entry's BucketType. addToNewBucket's isOld() consistency guard then rejects the demotion outright — silently, only logged — and since removeFromBucket had already dropped the address from addrLookup, it's gone from the book entirely.

E[...] Error adding peer to old bucket  err="failed consistency check! Cannot add pre-existing address ... into new bucket 221"

Receipts (p2p/pex/addrbook_bucket_overflow_test.go):

$ go test ./p2p/pex/... -v -run 'TestOldBucketRejectsEntryAtCapacity|TestMoveToOldDemotionDoesNotLoseAddress'
--- PASS: TestOldBucketRejectsEntryAtCapacity (0.01s)
--- PASS: TestMoveToOldDemotionDoesNotLoseAddress (0.01s)

Both fail with the predicted symptoms when run against the unfixed source (verified via git stash before writing the fix — not just written and assumed correct).

A Codex adversarial review of the first pass caught that this fix was itself incomplete for legacy state: a book persisted by a pre-fix build can already have a bucket at 65+ entries. Against that state, a single demotion only brings the bucket down to exactly 64 — one short of the room the original promotee still needs on its final re-add attempt — so the promotee itself ended up orphaned (BucketType=old, zero bucket membership, uncounted, unreachable). Fixed by looping the demotion until there's actually room, with a last-resort fallback that keeps the address reachable as new rather than losing it if that still somehow fails. The identical shape existed in addToNewBucket's single-expiry-then-add pattern for a legacy oversized new bucket (would stay at 65 forever); fixed the same way. Both have dedicated regression tests, independently verified red against the Codex-reviewed first-pass fix and green against the final one:

$ go test ./p2p/pex/... -v -run 'TestMoveToOldSurvivesLegacyOverfullOldBucket|TestAddToNewBucketShrinksLegacyOverfullBucket'
--- PASS: TestMoveToOldSurvivesLegacyOverfullOldBucket (0.01s)
--- PASS: TestAddToNewBucketShrinksLegacyOverfullBucket (0.00s)

Full package: go test ./p2p/pex/...ok (11.1s, 25 tests).

libs/json — 1-byte input panics the 64-bit-integer decode guard

decodeReflect's int64/uint64 branch checks bz[0] != '"' || bz[len(bz)-1] != '"' without first checking len(bz) >= 2. For a 1-byte input this reads the same byte twice, the guard passes (both comparisons happen to be false when that byte is "), and the following bz[1:len(bz)-1] slice becomes bz[1:0] — panic. The sibling time-value guard 30 lines up already has the len(bz) < 2 check this one is missing.

panic: runtime error: slice bounds out of range [1:0]

Reachable via unauthenticated RPC (e.g. GET /block?height="); caught by RecoverAndLogHandler, so this is a wrong-status-code + log-noise bug, not a node crash — stated plainly, not oversold.

Fix: add the missing len(bz) < 2 ||. Receipt:

$ go test ./libs/json/... -v -run 'TestUnmarshal/int64_single_char'
--- PASS: TestUnmarshal (0.00s)
    --- PASS: TestUnmarshal/int64_single_char (0.00s)

Fails with the exact panic above against the unfixed source. Codex found no issue in this fix or its test.

store — DeleteLatestBlock (the cometbft rollback path) leaks the extended-commit row and never evicts commit caches

DeleteLatestBlock deletes the block/commit/seen-commit/meta rows but never deletes the EC: (extended-commit) row, and never evicts any of the three LRU commit caches (blockCommitCache, seenCommitCache, blockExtendedCommitCache). After a rollback + resync with a different block landing at the same height, LoadBlockExtendedCommit can return the stale pre-rollback value straight from cache — the DB itself gets correctly overwritten by the resync, but the cache doesn't know that.

Admin-triggered (requires an operator running cometbft rollback), not remotely reachable — stated as such.

Fix: delete the EC: key in the batch, and evict all three caches for the height. Receipts:

$ go test ./store/... -v -run TestDeleteLatestBlock
--- PASS: TestDeleteLatestBlockEvictsExtendedCommitCache (0.00s)
--- PASS: TestDeleteLatestBlockDeletesExtendedCommitRow (0.00s)
--- PASS: TestDeleteLatestBlockEvictsCommitAndSeenCommitCaches (0.00s)

The first test proves the cache-staleness scenario end to end via LoadBlockExtendedCommit. Codex correctly flagged that this alone doesn't prove the DB row is actually deleted (a resave overwrites it regardless of whether the delete ran) — added TestDeleteLatestBlockDeletesExtendedCommitRow, which reads the raw DB key directly with no resync involved, and extended coverage to LoadSeenCommit's cache too. All three independently verified red against upstream main (the real pre-fix source, not just the first-pass commit) and green after.

Disclosed, not fixed — flagging for a maintainer rather than expanding scope: Load*Commit's DB-read-then-cache-Add sequence isn't performed under bs.mtx. A concurrent DeleteLatestBlock racing an in-flight Load*Commit call could theoretically repopulate a cache entry just after this fix evicts it. This race predates this PR — every existing cache-eviction call site in this file (e.g. PruneBlocks's own blockExtendedCommitCache.Remove) has the identical gap — and closing it properly means broader locking discipline changes across the whole store package, out of scope for a housekeeping PR like this one.

Full package: go test ./store/...ok (2.1s, 15 tests).

libs/protoio — byteReader mishandles two legal io.Reader return shapes

(0, nil) — legal per the io.Reader doc ("nothing happened yet") — surfaced a stale or fabricated zero byte instead of retrying. (n>0, io.EOF) — also legal, explicitly documented — discarded an already-successfully-read byte along with the error. All existing tests use bytes.Buffer, which happens to never produce either shape.

Fix: loop on (0, nil); return the byte immediately whenever n == 1 regardless of the accompanying error (deferring any non-EOF error to the next call). This exactly matches bufio.Reader's own fill()/readErr() pattern in the standard library — verified by reading the real stdlib source, not assumed. Receipts:

$ go test ./libs/protoio/... -v -run TestByteReader
--- PASS: TestByteReaderIgnoresZeroByteNilErrorReturn (0.00s)
--- PASS: TestByteReaderDoesNotDropByteReturnedAlongsideEOF (0.00s)
--- PASS: TestByteReaderGivesUpOnPersistentZeroByteReader (0.00s)

Codex flagged (correctly) that an unbounded (0, nil) retry loop could hang forever against a pathological reader. Bounded it with the identical maxConsecutiveEmptyReads = 100 → io.ErrNoProgress pattern bufio.Reader itself uses. TestByteReaderGivesUpOnPersistentZeroByteReader uses a goroutine + 5s test timeout to directly prove the unbounded version genuinely hangs (confirmed by temporarily reverting just the bound and watching the test time out) and the bounded version returns cleanly.

Full package: go test ./libs/protoio/...ok (1.3s, 9 tests).

libs/pubsub/query — numeric operators never match a negative event attribute value

extractNum's regex was ^\d+(\.\d+)? — no support for a leading -. A negative event attribute (e.g. "-5") extracted an empty string, parseNumber returned an error, and =/</<=/>/>= all share the same err == nil && ... guard — so every comparison against a negative attribute silently evaluated to false.

Fix: ^-?\d+(\.\d+)?. Receipts:

$ go test ./libs/pubsub/query/... -v -run TestNegativeNumbers
--- PASS: TestNegativeNumbers (0.00s)
    --- PASS: TestNegativeNumbers/01 .. /08

Note on scope: the query literal itself still can't be negative — the grammar's own number scanner doesn't accept a leading - in query text at all, a separate pre-existing limitation this PR doesn't touch. Tests compare only against a non-negative literal (0), which is sufficient to isolate this fix. Codex correctly caught that an earlier draft of this test included a delta.value = 0 case that was false both before and after the fix (-5 != 0 either way) — removed it and left a comment explaining why = isn't independently provable through the query-syntax path even though the fix covers it via the same parseNumber the ordering-operator tests do exercise.

Full package: go test ./libs/pubsub/...ok (all three subpackages).

Review process

Ran Codex (gpt-5.6-sol) as a synchronous adversarial reviewer against the first-pass commit before pushing. It surfaced six concrete, real findings — two led to genuine additional fixes (the legacy-state bucket handling, the busy-spin bound), three led to test-quality improvements (the DB-row-vs-cache distinction, the misleading = subtest, the seen-commit coverage), and one (the byteReader n=1+err handling) was investigated against the real Go stdlib and confirmed to already be correct, matching bufio.Reader's own idiom — kept as-is with the reasoning disclosed above rather than silently ignored or blindly changed.

What's not in this PR

A separate, genuinely security-sensitive finding from the same audit pass — an unrecovered nil-pointer dereference in the mempool recheck path (findNextEntryMatching), reachable under a specific recheck-timeout + stale-response timing condition — is deliberately excluded. Publishing the exact trigger conditions in a public diff before a fix ships would be irresponsible for a consensus node's mempool; it's being routed through coordinated disclosure instead.

…es, protoio byte reader, negative-number queries

Five independent, small correctness bugs found across peer-discovery
bookkeeping and internal libraries:

- p2p/pex: addToNewBucket/addToOldBucket used `>` instead of `>=` against
  their bucket size constants, admitting one entry past the documented
  capacity. Worse, when moveToOld's overflow-recovery path demotes the
  oldest entry of a full old bucket back to a new bucket, removeFromBucket
  never resets BucketType before addToNewBucket's isOld() guard runs, so
  the demotion always fails and the address is silently dropped from the
  book (only logged, never surfaced).

- libs/json: decodeReflect's 64-bit-integer guard read bz[0] and
  bz[len(bz)-1] without checking len(bz) >= 2 first, so a 1-byte JSON
  value ("\"") panics on the following bz[1:len(bz)-1] slice. The
  sibling time-value guard 30 lines up already has the len(bz) < 2
  check this one is missing. Reachable via unauthenticated RPC
  (e.g. GET /block?height="); caught by RecoverAndLogHandler, so this
  is a wrong-status/log-noise bug, not a node crash.

- store: DeleteLatestBlock (the `cometbft rollback` operator path)
  deletes the block/commit/seen-commit rows but never deletes the
  extended-commit ("EC:") row, and never evicts any of the three LRU
  commit caches. After a rollback + resync with a different block at
  the same height, LoadBlockExtendedCommit can return the stale
  pre-rollback value straight from cache.

- libs/protoio: byteReader.ReadByte mishandled two io.Reader return
  shapes that are legal per the interface's own doc: (0, nil) surfaced
  a stale/fabricated byte instead of retrying, and (n>0, io.EOF)
  discarded an already-read byte along with the error. Fixed by
  looping on (0, nil) and returning the byte immediately whenever
  n == 1, regardless of the accompanying error.

- libs/pubsub/query: extractNum's regex only matched `^\d+(\.\d+)?`,
  so a negative event attribute value (e.g. "-5") extracted an empty
  string, parseNumber returned an error, and every numeric comparison
  operator (=, <, <=, >, >=) against it silently evaluated to false.
  Fixed by allowing an optional leading '-'.

Each fix ships with a genuine red-before/green-after regression test
(verified via git stash against the unfixed source, not just written
and assumed correct) plus the corresponding CHANGELOG.md entry.

A separate, genuinely security-sensitive finding from the same
scouting pass (an unrecovered nil-pointer dereference in the mempool
recheck path) is intentionally NOT included here and is being routed
through coordinated disclosure instead of a public PR.
…or, and DoS bound

Codex's adversarial pass on the previous commit surfaced several real,
concrete issues, all addressed here:

- p2p/pex: a single demotion in moveToOld only brings a full old bucket
  down to exactly oldBucketSize, one entry short of the room a promotion
  needs -- a bucket a pre-fix build already left at oldBucketSize+2 (or
  more) would still reject the promotee's final re-add, silently
  orphaning it (BucketType=old, zero bucket membership, uncounted,
  unreachable). Changed the recovery to keep demoting until there's
  room, and added a last-resort fallback to keep the address reachable
  as new if that still somehow fails. The identical issue existed in
  addToNewBucket's single-expiry-then-add pattern for a legacy
  oversized new bucket; changed to loop there too. Added regression
  tests for both, independently verified red against the prior commit's
  fix and green against this one.

- libs/protoio: bounded the (0, nil) retry loop added in the previous
  commit with the same maxConsecutiveEmptyReads=100 -> io.ErrNoProgress
  pattern bufio.Reader itself uses, so a reader that persistently
  returns (0, nil) can't hang ReadByte forever. Added a goroutine+timeout
  regression test proving the unbounded version genuinely hangs and the
  bounded version doesn't. (Reviewed and kept as-is: returning the byte
  with nil error whenever n==1, deferring any accompanying non-EOF error
  to the next call, exactly matches bufio.Reader's own fill()/readErr()
  pattern -- not a regression.)

- store: strengthened the DeleteLatestBlock regression coverage to
  check the extended-commit DB row is actually gone (not just that the
  cache was evicted, which a subsequent overwrite could have hidden
  either way), and added seen-commit cache eviction coverage alongside
  the existing extended-commit cache test.

- libs/pubsub/query: removed a TestNegativeNumbers subtest asserting
  `delta.value = 0` is false -- true both before and after the fix
  (-5 != 0 either way), so it couldn't actually distinguish red from
  green. Replaced with a comment explaining why "=" isn't independently
  testable via query syntax (the query literal itself can't be
  negative, a separate pre-existing scanner limitation) even though the
  fix covers it via the same parseNumber the ordering-operator tests do
  exercise.

Not changed, disclosed instead: Load*Commit's DB-read-then-cache-Add
sequence isn't done under bs.mtx, so a concurrent DeleteLatestBlock
racing an in-flight Load call could theoretically repopulate a cache
entry just after this fix evicts it. This race predates this PR --
every existing cache-eviction call site in this file (e.g. PruneBlocks)
has the identical gap -- and fixing it would mean broader locking
changes across the store package, out of scope for this housekeeping
PR. Flagging it for a maintainer rather than silently expanding scope.
@gomesalexandre
gomesalexandre marked this pull request as ready for review September 1, 2026 21:47
@gomesalexandre
gomesalexandre requested a review from a team as a code owner September 1, 2026 21:47
@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

PR author is not in the allowed authors list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant