Repository navigation
feat(transcript): delta upload via uuid-cursor chunks (v0.24.0, full cutover) - #99
Conversation
…cutover)
Replace the full-file transcript re-upload on every Stop hook with an
incremental delta protocol: send only lines added since the last-sent uuid.
The previous approach re-uploaded the entire growing transcript each turn,
tripping the backend's 6 MB request limit (413) on long sessions.
Client protocol:
- Per-session uuid cursor {transcriptId,lastUuid,seq} persisted under
transcripts/cursors/ (survives chunk deletion and --resume).
- Each Stop seals a delta chunk (only new lines since cursor.lastUuid),
per-line sanitized, split to a byte budget, staged under
transcripts/chunks/ and drained independently; deleted on 2xx.
- Reset handling: when the cursor uuid is gone from the transcript, resend
from scratch with X-Chunk-Reset carrying the new baseline seq.
- Headers: X-Chunk-Seq, X-Chunk-Reset, X-Prompt-ID.
Full cutover — no legacy fallback. Removed the whole-file paths
(stageTranscriptForUpload / uploadPendingTranscript / listPendingTranscripts /
drainPendingTranscripts) and the unused sanitizeTranscript(); the retry daemon
and retryFailedTranscripts() now drain delta chunks. Pairs with the collector's
server-side chunk-merge (skillmeter-claude-code-collector#37).
New pure core scripts/lib/transcript-delta.js (parse/cursor/delta/split/plan)
with 22 unit tests. Verified end-to-end against dev: 15 chunks merged into a
single 5.9 MB object, 3025 lines, 0 invalid JSON, 2756 unique uuids (0 dupes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ELGpc4xqYqmAq9tzL93sJ
WalkthroughThe transcript upload pipeline now computes UUID-based deltas, splits them into durable chunks, persists cursors, uploads with reset metadata, and retries through the chunk queue. Queue accounting, cleanup, session finalization, daemon draining, sanitization exports, configuration, tests, and the plugin version were updated. ChangesTranscript delta uploads
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant SessionEnd
participant Transfer
participant ChunkQueue
participant UploadAPI
SessionEnd->>Transfer: stageTranscriptDelta
Transfer->>ChunkQueue: seal delta chunks and cursor metadata
Transfer->>UploadAPI: uploadDeltaChunk
UploadAPI-->>Transfer: upload response
Transfer->>ChunkQueue: remove successful chunks
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skillmeter/scripts/lib/transcript-delta.js`:
- Around line 136-139: The cursor update in lastContentUuid and the returned
newCursor must persist sufficient terminal-line state to advance past UUID-less
metadata records while preserving rewrite detection. Update
skillmeter/scripts/lib/transcript-delta.js:136-139 around lastContentUuid and
newCursor accordingly. Extend skillmeter/test/transcript-delta.test.js:138-145
to plan a second pass with newCursor and assert it produces no chunks.
- Around line 71-91: Enforce the backend’s actual compressed request limit
across the chunking and transfer flow: in
skillmeter/scripts/lib/transcript-delta.js lines 71-91, prevent oversized
records from bypassing the limit and define whether individually oversized
records are rejected or otherwise handled; in skillmeter/scripts/lib/config.js
lines 105-110, require a positive budget and remove the assumption that 20 MB
raw is inherently safe; in skillmeter/scripts/lib/transfer.js lines 331-355,
validate compressed.length and repartition or reject before sending; update
skillmeter/test/transcript-delta.test.js lines 95-100 to assert the hard-limit
behavior rather than allowing oversized records.
In `@skillmeter/scripts/lib/transfer.js`:
- Around line 615-627: Update the cleanup logic that scans
TRANSCRIPTS_CHUNKS_DIR so it does not age-delete sealed delta chunks while their
advanced cursor is retained. Restrict candidates to temporary or unmatched
artifacts, or transactionally invalidate the corresponding cursor before
deleting any sealed chunk; preserve cursor continuity so prolonged outages
cannot permanently skip unsent lines.
- Around line 370-381: Update drainDeltaChunks to replace the unbounded
Promise.allSettled over files with a small fixed-size worker pool that processes
only a limited number of uploadDeltaChunk calls concurrently. Preserve the
existing ensureFreshLicense, tally, and per-file result handling while ensuring
every queued chunk is eventually processed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4d99d614-2960-4864-8e7b-1777780ae80e
📒 Files selected for processing (8)
skillmeter/.claude-plugin/plugin.jsonskillmeter/scripts/lib/config.jsskillmeter/scripts/lib/paths.jsskillmeter/scripts/lib/sanitize.jsskillmeter/scripts/lib/transcript-delta.jsskillmeter/scripts/lib/transfer.jsskillmeter/scripts/monitors/retry_daemon.jsskillmeter/test/transcript-delta.test.js
💤 Files with no reviewable changes (1)
- skillmeter/scripts/lib/sanitize.js
| * Split serialized JSONL strings into groups whose UNCOMPRESSED size stays | ||
| * under `maxUncompressedBytes` (a conservative proxy so the gzipped body stays | ||
| * well under the backend's hard limit). A single line larger than the budget | ||
| * becomes its own group — it is never dropped. | ||
| */ | ||
| function splitLinesByBudget(lines, maxUncompressedBytes = DEFAULT_CHUNK_MAX_BYTES) { | ||
| const groups = []; | ||
| let current = []; | ||
| let size = 0; | ||
| for (const line of lines) { | ||
| const lineBytes = Buffer.byteLength(line, "utf8") + 1; // +1 for the newline | ||
| if (current.length > 0 && size + lineBytes > maxUncompressedBytes) { | ||
| groups.push(current); | ||
| current = []; | ||
| size = 0; | ||
| } | ||
| current.push(line); | ||
| size += lineBytes; | ||
| } | ||
| if (current.length > 0) groups.push(current); | ||
| return groups; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Enforce the backend’s actual compressed request limit. The current design assumes a gzip ratio and permits single records to exceed the budget, so the primary HTTP 413 failure remains possible.
skillmeter/scripts/lib/transcript-delta.js#L71-L91: ensure no emitted chunk can bypass the limit; define handling for an individually oversized record.skillmeter/scripts/lib/config.js#L105-L110: validate a positive budget and stop treating 20 MB raw as a guaranteed safe value.skillmeter/scripts/lib/transfer.js#L331-L355: checkcompressed.lengthand repartition or reject before sending.skillmeter/test/transcript-delta.test.js#L95-L100: test the resulting hard-limit behavior instead of expecting oversized records to pass through.
📍 Affects 4 files
skillmeter/scripts/lib/transcript-delta.js#L71-L91(this comment)skillmeter/scripts/lib/config.js#L105-L110skillmeter/scripts/lib/transfer.js#L331-L355skillmeter/test/transcript-delta.test.js#L95-L100
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/lib/transcript-delta.js` around lines 71 - 91, Enforce the
backend’s actual compressed request limit across the chunking and transfer flow:
in skillmeter/scripts/lib/transcript-delta.js lines 71-91, prevent oversized
records from bypassing the limit and define whether individually oversized
records are rejected or otherwise handled; in skillmeter/scripts/lib/config.js
lines 105-110, require a positive budget and remove the assumption that 20 MB
raw is inherently safe; in skillmeter/scripts/lib/transfer.js lines 331-355,
validate compressed.length and repartition or reject before sending; update
skillmeter/test/transcript-delta.test.js lines 95-100 to assert the hard-limit
behavior rather than allowing oversized records.
| // Metadata-only delta keeps the previous anchor (no new content uuid). | ||
| const newLastUuid = lastContentUuid(deltaObjs) || (cursor && cursor.lastUuid) || null; | ||
|
|
||
| return { chunks, newCursor: { lastUuid: newLastUuid, seq } }; |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Metadata-only tails are replayed on every staging pass. Retaining the previous UUID does not identify that trailing metadata was already processed.
skillmeter/scripts/lib/transcript-delta.js#L136-L139: persist enough terminal-line state to advance past UUID-less records while retaining rewrite detection.skillmeter/test/transcript-delta.test.js#L138-L145: run a second plan usingnewCursorand assert that it produces no chunks.
📍 Affects 2 files
skillmeter/scripts/lib/transcript-delta.js#L136-L139(this comment)skillmeter/test/transcript-delta.test.js#L138-L145
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/lib/transcript-delta.js` around lines 136 - 139, The
cursor update in lastContentUuid and the returned newCursor must persist
sufficient terminal-line state to advance past UUID-less metadata records while
preserving rewrite detection. Update
skillmeter/scripts/lib/transcript-delta.js:136-139 around lastContentUuid and
newCursor accordingly. Extend skillmeter/test/transcript-delta.test.js:138-145
to plan a second pass with newCursor and assert it produces no chunks.
| async function drainDeltaChunks(timeoutMs) { | ||
| const files = listDeltaChunks(); | ||
| if (files.length === 0) return { ok: 0, errors: [] }; | ||
|
|
||
| const deviceId = credstore.getDeviceId(); | ||
| if (!deviceId) return { ok: 0, errors: [] }; | ||
|
|
||
| console.error(`[skillmeter] Draining ${files.length} transcript chunk(s)`); | ||
| // Best-effort, single-flight refresh once per batch (see drainFailedLogs). | ||
| await ensureFreshLicense(deviceId); | ||
| const results = await Promise.allSettled(files.map((f) => uploadDeltaChunk(f, deviceId, timeoutMs))); | ||
| return tally(results); |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Bound delta-chunk drain concurrency.
Promise.allSettled(files.map(...)) reads, gzips, and posts the entire backlog simultaneously. Large queues of 20 MB chunks can exhaust monitor memory and sockets; use a small worker pool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/lib/transfer.js` around lines 370 - 381, Update
drainDeltaChunks to replace the unbounded Promise.allSettled over files with a
small fixed-size worker pool that processes only a limited number of
uploadDeltaChunk calls concurrently. Preserve the existing ensureFreshLicense,
tally, and per-file result handling while ensuring every queued chunk is
eventually processed.
| // Delta chunk bodies + sidecar metas (and any orphaned .tmp). Cursors are | ||
| // deliberately NOT swept here — they must outlive gaps (vacations, long | ||
| // --resume) so the delta continues instead of full-resending. | ||
| if (fs.existsSync(TRANSCRIPTS_CHUNKS_DIR)) { | ||
| try { | ||
| for (const f of fs.readdirSync(TRANSCRIPTS_CHUNKS_DIR)) { | ||
| candidates.push(path.join(TRANSCRIPTS_CHUNKS_DIR, f)); | ||
| } | ||
| } catch { | ||
| // fall through | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win
Do not age-delete sealed chunks while retaining their advanced cursor.
The cursor advances when chunks are sealed, but this cleanup deletes unsent chunks after 30 days and deliberately preserves the cursor. After a prolonged outage, those lines are permanently skipped. Sweep only temporary/unmatched artifacts, or invalidate the corresponding cursor transactionally.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/lib/transfer.js` around lines 615 - 627, Update the
cleanup logic that scans TRANSCRIPTS_CHUNKS_DIR so it does not age-delete sealed
delta chunks while their advanced cursor is retained. Restrict candidates to
temporary or unmatched artifacts, or transactionally invalidate the
corresponding cursor before deleting any sealed chunk; preserve cursor
continuity so prolonged outages cannot permanently skip unsent lines.
What
Replace the full-file transcript re-upload on every Stop hook with an incremental delta protocol — send only lines added since the last-sent
uuid. The old approach re-uploaded the entire growing transcript each turn, tripping the backend's 6 MB request limit (413) on long sessions.Protocol
{transcriptId,lastUuid,seq}undertranscripts/cursors/(survives chunk deletion +--resume).cursor.lastUuid), per-line sanitized, split to a byte budget, staged undertranscripts/chunks/, drained independently, deleted on 2xx.X-Chunk-Resetcarrying the new baseline seq.X-Chunk-Seq,X-Chunk-Reset,X-Prompt-ID.Full cutover — no legacy fallback
Removed the whole-file paths (
stageTranscriptForUpload/uploadPendingTranscript/listPendingTranscripts/drainPendingTranscripts) and the unusedsanitizeTranscript(). The retry daemon +retryFailedTranscripts()now drain delta chunks. Pairs with the server-side chunk-merge in skillmeter-claude-code-collector#37 (deployed to dev + prod).Version
0.23.0 → 0.24.0(per the repo's one-bump-per-feature convention).Tests / verification
scripts/lib/transcript-delta.js(parse / cursor / delta / split / plan) with 22 unit tests, all green.Rollout ordering (done before this merges)
Prod collector chunk-merge (#37) + collector IAM (#100) + activation env rename (infra #104) are all applied to prod, so prod clients can send chunks safely the moment this releases.
🤖 Generated with Claude Code
https://claude.ai/code/session_014ELGpc4xqYqmAq9tzL93sJ
Summary by CodeRabbit