Skip to content

bug: /api/public/v2/scoresmeta.totalItems still overcounts (deleted-row residue; follow-up to #6396) #13559

@shawnclybor

Description

@shawnclybor

Describe the bug

GET /api/public/v2/scores returns a meta.totalItems that overcounts the actual data[] length. The maintainer comment on #6396 stated the new /v2/scores route "should not be affected by this" — confirming here that it is still affected, but by a different underlying mechanism than the original report: deleted score rows remain counted in meta.totalItems while being correctly excluded from data[].

This is essentially a regression-follow-up on #6396. The session/dataset-score type-mismatch path described in that thread does not apply here (project has no session-attached or dataset-attached scores).

To reproduce

Langfuse Cloud (US), project cml6xji63001pad08i3odtvh3, observed 2026-05-11.

curl -u "$PK:$SK" "$BASE/api/public/v2/scores?limit=100" \
  | jq '{data_len:(.data|length), total:.meta.totalItems}'
{ "data_len": 68, "total": 82 }

The gap narrows as fromTimestamp is moved forward and collapses once the window excludes a known DELETE event:

fromTimestamp data.length meta.totalItems gap
(all-time) 68 82 14
2026-04-01 67 81 14
2026-05-01 39 52 13
2026-05-08 14 14 0

Per-name decomposition isolates the gap to five score names:

name filter data.length meta.totalItems gap
1a-i-cfu-error-floor 13 23 10
1b-misconception-surfacing 3 4 1
2a-i-bucket-detection 2 3 1
2c-bucket-non-example-callout 1 2 1
2d-bucket-absence-framing 1 2 1
all others (11 names) 49 49 0
total 68 82 14

The 10-row gap on 1a-i-cfu-error-floor matches a known event: on 2026-05-05 we renamed legacy 1a-i-per-turn-error-floor scores by POST /api/public/scores with a new name and DELETE /api/public/scores/{old_id} for the legacy id. The 12 visible rows on the new name carry metadata.renamed_from = "1a-i-per-turn-error-floor" and metadata.renamed_at = "2026-05-05". The 10 DELETEd legacy rows are still counted in totalItems. The other four names show the same pattern at +1 each from one-off DELETEs during scorer iteration.

Control: a different score name (4a-iv-engagement-signal--RETIRED-2026-05-11) was renamed today via an in-place mechanism (no DELETE+POST). That name returns data=12 / totalItems=12 — no phantom residue. So the residue is specifically tied to the DELETE path, not to renames generally.

Per-trace evidence (rules out trace-orphan / type-mismatch explanations): ?traceId=a236f702-e26c-443e-b923-c8c15f3306cd returns data=4 / totalItems=5. The trace itself still exists; the phantom is at the score level.

Pagination is internally consistent with data.length (page 1 with limit=50 returns 50, page 2 returns 18 → 68 total), so the bug is solely in the counter source, not in the page iterator.

SDK and container versions

Cloud (US). No SDK involved — reproduced directly with curl.

Additional information

Practical impact: meta.totalItems is the natural fanout-completion signal for scoring pipelines ("did all my POSTs land?"). Switching to data.length works but is surprising given the field name. We've added a note in our internal docs warning operators not to trust totalItems for assertions.

Suggested fix: compute meta.totalItems against the same view used to populate data[], so soft-deleted/tombstoned rows are excluded from both. If the underlying storage retains tombstones for audit, the counter should filter them out at projection time the same way the data path does.

Are you interested to contribute a fix for this bug?

No (no access to the relevant service layer).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions