Skip to content

feat(transcript): delta upload via uuid-cursor chunks (v0.24.0, full cutover) - #99

Merged
minarc merged 1 commit into
mainfrom
feat/transcript-delta-upload
Jul 15, 2026
Merged

minarc merged 1 commit into
mainfrom
feat/transcript-delta-upload

Conversation

@minarc

@minarc minarc commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

What

Replace the full-file transcript re-upload on every Stop hook with an incremental delta protocol — send only lines added since the last-sent uuid. The old approach re-uploaded the entire growing transcript each turn, tripping the backend's 6 MB request limit (413) on long sessions.

Protocol

  • Per-session uuid cursor {transcriptId,lastUuid,seq} under transcripts/cursors/ (survives chunk deletion + --resume).
  • Each Stop seals a delta chunk (new lines since cursor.lastUuid), per-line sanitized, split to a byte budget, staged under transcripts/chunks/, drained independently, deleted on 2xx.
  • Reset: when the cursor uuid is gone from the transcript, resend from scratch with X-Chunk-Reset carrying the new baseline seq.
  • Headers: X-Chunk-Seq, X-Chunk-Reset, X-Prompt-ID.

Full cutover — no legacy fallback

Removed the whole-file paths (stageTranscriptForUpload / uploadPendingTranscript / listPendingTranscripts / drainPendingTranscripts) and the unused sanitizeTranscript(). The retry daemon + retryFailedTranscripts() now drain delta chunks. Pairs with the server-side chunk-merge in skillmeter-claude-code-collector#37 (deployed to dev + prod).

Version

0.23.0 → 0.24.0 (per the repo's one-bump-per-feature convention).

Tests / verification

  • New pure core scripts/lib/transcript-delta.js (parse / cursor / delta / split / plan) with 22 unit tests, all green.
  • Verified end-to-end against dev: 15 chunks merged server-side into one 5.9 MB object — 3,025 lines, 0 invalid JSON, 2,756 unique uuids (0 dupes).

Rollout ordering (done before this merges)

Prod collector chunk-merge (#37) + collector IAM (#100) + activation env rename (infra #104) are all applied to prod, so prod clients can send chunks safely the moment this releases.

🤖 Generated with Claude Code

https://claude.ai/code/session_014ELGpc4xqYqmAq9tzL93sJ

Summary by CodeRabbit

  • New Features
    • Improved transcript uploads using smaller, resumable chunks.
    • Uploads can now continue across turns or sessions without retransmitting completed transcript content.
    • Added automatic handling for large transcripts and configurable upload size limits.
    • Enhanced retry and background queue processing for transcript uploads.
  • Bug Fixes
    • Prevented incomplete or orphaned transcript chunks from being uploaded.
    • Improved handling of malformed transcript lines during processing.
  • Release
    • Updated the plugin version to 0.24.0.

…cutover)

Replace the full-file transcript re-upload on every Stop hook with an
incremental delta protocol: send only lines added since the last-sent uuid.
The previous approach re-uploaded the entire growing transcript each turn,
tripping the backend's 6 MB request limit (413) on long sessions.

Client protocol:
  - Per-session uuid cursor {transcriptId,lastUuid,seq} persisted under
    transcripts/cursors/ (survives chunk deletion and --resume).
  - Each Stop seals a delta chunk (only new lines since cursor.lastUuid),
    per-line sanitized, split to a byte budget, staged under
    transcripts/chunks/ and drained independently; deleted on 2xx.
  - Reset handling: when the cursor uuid is gone from the transcript, resend
    from scratch with X-Chunk-Reset carrying the new baseline seq.
  - Headers: X-Chunk-Seq, X-Chunk-Reset, X-Prompt-ID.

Full cutover — no legacy fallback. Removed the whole-file paths
(stageTranscriptForUpload / uploadPendingTranscript / listPendingTranscripts /
drainPendingTranscripts) and the unused sanitizeTranscript(); the retry daemon
and retryFailedTranscripts() now drain delta chunks. Pairs with the collector's
server-side chunk-merge (skillmeter-claude-code-collector#37).

New pure core scripts/lib/transcript-delta.js (parse/cursor/delta/split/plan)
with 22 unit tests. Verified end-to-end against dev: 15 chunks merged into a
single 5.9 MB object, 3025 lines, 0 invalid JSON, 2756 unique uuids (0 dupes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ELGpc4xqYqmAq9tzL93sJ
@coderabbitai

coderabbitai Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The transcript upload pipeline now computes UUID-based deltas, splits them into durable chunks, persists cursors, uploads with reset metadata, and retries through the chunk queue. Queue accounting, cleanup, session finalization, daemon draining, sanitization exports, configuration, tests, and the plugin version were updated.

Changes

Transcript delta uploads

Layer / File(s) Summary
Delta planning and storage contracts
skillmeter/scripts/lib/config.js, skillmeter/scripts/lib/paths.js, skillmeter/scripts/lib/sanitize.js, skillmeter/scripts/lib/transcript-delta.js, skillmeter/test/transcript-delta.test.js
Chunk-size configuration, durable chunk and cursor paths, JSONL parsing, UUID cursor handling, sanitization, reset planning, byte-budget splitting, sequencing, and pure-function tests were added or updated.
Durable chunk persistence and upload
skillmeter/scripts/lib/transfer.js, skillmeter/test/transcript-delta.test.js
Cursor persistence, sidecar-backed chunk sealing, chunk listing, reset-aware headers, uploads, deletion after success, staging, and filesystem/header tests were added.
Queue lifecycle integration
skillmeter/scripts/lib/transfer.js, skillmeter/scripts/monitors/retry_daemon.js, skillmeter/.claude-plugin/plugin.json
Session finalization, migration, queue counts, draining, retries, cleanup, exports, daemon behavior, and the plugin manifest version were updated for delta chunks.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SessionEnd
  participant Transfer
  participant ChunkQueue
  participant UploadAPI
  SessionEnd->>Transfer: stageTranscriptDelta
  Transfer->>ChunkQueue: seal delta chunks and cursor metadata
  Transfer->>UploadAPI: uploadDeltaChunk
  UploadAPI-->>Transfer: upload response
  Transfer->>ChunkQueue: remove successful chunks
Loading

Possibly related PRs

Poem

A bunny packs each delta tight,
With cursors hopping through the night.
Chunks queue up, then safely fly,
Failed ones wait and retry.
“JSONL carrots, neatly sealed—
Every transcript trail revealed!”

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: transcript uploads now use uuid-cursor delta chunks, and it matches the PR’s v0.24.0 cutover.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/transcript-delta-upload

Comment @coderabbitai help to get the list of available commands.

@minarc
minarc merged commit 6776f68 into main Jul 15, 2026
1 check was pending

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skillmeter/scripts/lib/transcript-delta.js`:
- Around line 136-139: The cursor update in lastContentUuid and the returned
newCursor must persist sufficient terminal-line state to advance past UUID-less
metadata records while preserving rewrite detection. Update
skillmeter/scripts/lib/transcript-delta.js:136-139 around lastContentUuid and
newCursor accordingly. Extend skillmeter/test/transcript-delta.test.js:138-145
to plan a second pass with newCursor and assert it produces no chunks.
- Around line 71-91: Enforce the backend’s actual compressed request limit
across the chunking and transfer flow: in
skillmeter/scripts/lib/transcript-delta.js lines 71-91, prevent oversized
records from bypassing the limit and define whether individually oversized
records are rejected or otherwise handled; in skillmeter/scripts/lib/config.js
lines 105-110, require a positive budget and remove the assumption that 20 MB
raw is inherently safe; in skillmeter/scripts/lib/transfer.js lines 331-355,
validate compressed.length and repartition or reject before sending; update
skillmeter/test/transcript-delta.test.js lines 95-100 to assert the hard-limit
behavior rather than allowing oversized records.

In `@skillmeter/scripts/lib/transfer.js`:
- Around line 615-627: Update the cleanup logic that scans
TRANSCRIPTS_CHUNKS_DIR so it does not age-delete sealed delta chunks while their
advanced cursor is retained. Restrict candidates to temporary or unmatched
artifacts, or transactionally invalidate the corresponding cursor before
deleting any sealed chunk; preserve cursor continuity so prolonged outages
cannot permanently skip unsent lines.
- Around line 370-381: Update drainDeltaChunks to replace the unbounded
Promise.allSettled over files with a small fixed-size worker pool that processes
only a limited number of uploadDeltaChunk calls concurrently. Preserve the
existing ensureFreshLicense, tally, and per-file result handling while ensuring
every queued chunk is eventually processed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4d99d614-2960-4864-8e7b-1777780ae80e

📥 Commits

Reviewing files that changed from the base of the PR and between 3ed2ad2 and e0e5cc9.

📒 Files selected for processing (8)
  • skillmeter/.claude-plugin/plugin.json
  • skillmeter/scripts/lib/config.js
  • skillmeter/scripts/lib/paths.js
  • skillmeter/scripts/lib/sanitize.js
  • skillmeter/scripts/lib/transcript-delta.js
  • skillmeter/scripts/lib/transfer.js
  • skillmeter/scripts/monitors/retry_daemon.js
  • skillmeter/test/transcript-delta.test.js
💤 Files with no reviewable changes (1)
  • skillmeter/scripts/lib/sanitize.js

Comment on lines +71 to +91
* Split serialized JSONL strings into groups whose UNCOMPRESSED size stays
* under `maxUncompressedBytes` (a conservative proxy so the gzipped body stays
* well under the backend's hard limit). A single line larger than the budget
* becomes its own group — it is never dropped.
*/
function splitLinesByBudget(lines, maxUncompressedBytes = DEFAULT_CHUNK_MAX_BYTES) {
const groups = [];
let current = [];
let size = 0;
for (const line of lines) {
const lineBytes = Buffer.byteLength(line, "utf8") + 1; // +1 for the newline
if (current.length > 0 && size + lineBytes > maxUncompressedBytes) {
groups.push(current);
current = [];
size = 0;
}
current.push(line);
size += lineBytes;
}
if (current.length > 0) groups.push(current);
return groups;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Enforce the backend’s actual compressed request limit. The current design assumes a gzip ratio and permits single records to exceed the budget, so the primary HTTP 413 failure remains possible.

  • skillmeter/scripts/lib/transcript-delta.js#L71-L91: ensure no emitted chunk can bypass the limit; define handling for an individually oversized record.
  • skillmeter/scripts/lib/config.js#L105-L110: validate a positive budget and stop treating 20 MB raw as a guaranteed safe value.
  • skillmeter/scripts/lib/transfer.js#L331-L355: check compressed.length and repartition or reject before sending.
  • skillmeter/test/transcript-delta.test.js#L95-L100: test the resulting hard-limit behavior instead of expecting oversized records to pass through.
📍 Affects 4 files
  • skillmeter/scripts/lib/transcript-delta.js#L71-L91 (this comment)
  • skillmeter/scripts/lib/config.js#L105-L110
  • skillmeter/scripts/lib/transfer.js#L331-L355
  • skillmeter/test/transcript-delta.test.js#L95-L100
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/lib/transcript-delta.js` around lines 71 - 91, Enforce the
backend’s actual compressed request limit across the chunking and transfer flow:
in skillmeter/scripts/lib/transcript-delta.js lines 71-91, prevent oversized
records from bypassing the limit and define whether individually oversized
records are rejected or otherwise handled; in skillmeter/scripts/lib/config.js
lines 105-110, require a positive budget and remove the assumption that 20 MB
raw is inherently safe; in skillmeter/scripts/lib/transfer.js lines 331-355,
validate compressed.length and repartition or reject before sending; update
skillmeter/test/transcript-delta.test.js lines 95-100 to assert the hard-limit
behavior rather than allowing oversized records.

Comment on lines +136 to +139
// Metadata-only delta keeps the previous anchor (no new content uuid).
const newLastUuid = lastContentUuid(deltaObjs) || (cursor && cursor.lastUuid) || null;

return { chunks, newCursor: { lastUuid: newLastUuid, seq } };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Metadata-only tails are replayed on every staging pass. Retaining the previous UUID does not identify that trailing metadata was already processed.

  • skillmeter/scripts/lib/transcript-delta.js#L136-L139: persist enough terminal-line state to advance past UUID-less records while retaining rewrite detection.
  • skillmeter/test/transcript-delta.test.js#L138-L145: run a second plan using newCursor and assert that it produces no chunks.
📍 Affects 2 files
  • skillmeter/scripts/lib/transcript-delta.js#L136-L139 (this comment)
  • skillmeter/test/transcript-delta.test.js#L138-L145
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/lib/transcript-delta.js` around lines 136 - 139, The
cursor update in lastContentUuid and the returned newCursor must persist
sufficient terminal-line state to advance past UUID-less metadata records while
preserving rewrite detection. Update
skillmeter/scripts/lib/transcript-delta.js:136-139 around lastContentUuid and
newCursor accordingly. Extend skillmeter/test/transcript-delta.test.js:138-145
to plan a second pass with newCursor and assert it produces no chunks.

Comment on lines +370 to +381
async function drainDeltaChunks(timeoutMs) {
const files = listDeltaChunks();
if (files.length === 0) return { ok: 0, errors: [] };

const deviceId = credstore.getDeviceId();
if (!deviceId) return { ok: 0, errors: [] };

console.error(`[skillmeter] Draining ${files.length} transcript chunk(s)`);
// Best-effort, single-flight refresh once per batch (see drainFailedLogs).
await ensureFreshLicense(deviceId);
const results = await Promise.allSettled(files.map((f) => uploadDeltaChunk(f, deviceId, timeoutMs)));
return tally(results);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win

Bound delta-chunk drain concurrency.

Promise.allSettled(files.map(...)) reads, gzips, and posts the entire backlog simultaneously. Large queues of 20 MB chunks can exhaust monitor memory and sockets; use a small worker pool.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/lib/transfer.js` around lines 370 - 381, Update
drainDeltaChunks to replace the unbounded Promise.allSettled over files with a
small fixed-size worker pool that processes only a limited number of
uploadDeltaChunk calls concurrently. Preserve the existing ensureFreshLicense,
tally, and per-file result handling while ensuring every queued chunk is
eventually processed.

Comment on lines +615 to +627
// Delta chunk bodies + sidecar metas (and any orphaned .tmp). Cursors are
// deliberately NOT swept here — they must outlive gaps (vacations, long
// --resume) so the delta continues instead of full-resending.
if (fs.existsSync(TRANSCRIPTS_CHUNKS_DIR)) {
try {
for (const f of fs.readdirSync(TRANSCRIPTS_CHUNKS_DIR)) {
candidates.push(path.join(TRANSCRIPTS_CHUNKS_DIR, f));
}
} catch {
// fall through
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🔴 Critical | ⚡ Quick win

Do not age-delete sealed chunks while retaining their advanced cursor.

The cursor advances when chunks are sealed, but this cleanup deletes unsent chunks after 30 days and deliberately preserves the cursor. After a prolonged outage, those lines are permanently skipped. Sweep only temporary/unmatched artifacts, or invalidate the corresponding cursor transactionally.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/lib/transfer.js` around lines 615 - 627, Update the
cleanup logic that scans TRANSCRIPTS_CHUNKS_DIR so it does not age-delete sealed
delta chunks while their advanced cursor is retained. Restrict candidates to
temporary or unmatched artifacts, or transactionally invalidate the
corresponding cursor before deleting any sealed chunk; preserve cursor
continuity so prolonged outages cannot permanently skip unsent lines.

@TheStarkor
TheStarkor deleted the feat/transcript-delta-upload branch September 21, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant