Skip to content

feat(transcript): durable upload with retry + async read/gzip (v0.6.0) - #48

Merged
minarc merged 1 commit into
mainfrom
feat/transcript-retry-and-async-upload
Apr 24, 2026
Merged

minarc merged 1 commit into
mainfrom
feat/transcript-retry-and-async-upload

Conversation

@minarc

@minarc minarc commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Transcript uploads are the most critical payload this plugin ships — but they were also the fragile piece. Before this PR:

  • transferTranscript was fire-and-forget with no persistence. Any failure (transient network, 5xx from ingest, process tear-down before the fetch resolved) silently lost the transcript forever.
  • The pre-fetch work (fs.readFileSync + zlib.gzipSync on a multi-MB JSONL) blocked the hook's event loop for tens to hundreds of milliseconds on every Stop.

This PR keeps fire-and-forget (we don't want to block the user's next turn) but makes the transfer durable and moves the heavy I/O off the event loop.

Behavior changes

Durable upload with retry. New logs/transcripts/pending/ directory.

  • stageTranscriptForUpload sanitizes the JSONL and writes the result to pending/<transcript_id> before any network attempt. Long sessions re-stage on every Stop, overwriting the previous snapshot so pending always holds the latest content.
  • uploadPendingTranscript reads pending, gzips, POSTs. On 2xx -> unlink pending. On any failure -> pending stays, log line says kept pending for next session.
  • New retryFailedTranscripts() runs on SessionStart (next to the existing retryFailedLogs) and fires uploads for every pending file.

Async I/O. uploadPendingTranscript now awaits fs/promises.readFile and a promisified zlib.gzip, so the Stop hook stops blocking its event loop while compressing. Call sites still don't await the function, preserving the fire-and-forget contract the user asked for (no blocking their next turn on network).

Version bump. 0.5.1 -> 0.6.0.

Failure scenarios, before vs after

Scenario Before After
Network error mid-POST log + transcript lost log + pending file retained, retry on next session
Ingest returns 502 log + transcript lost same, retained and retried
Process tear-down before fetch settles content lost pending file retained, retried
Auth 401/403 clear token, retry once, lose if still failing clear token, retry once, if still failing -> retained
gzip OOM on huge file hook crashes, transcript lost same OOM risk (see #4 in earlier audit; not fixed here)

Trade-offs / known limitations (intentionally not addressed)

  • Still re-sends the full growing transcript on every Stop (no incremental upload). Server must dedupe by X-Transcript-ID.
  • Still loads the whole file into memory for gzip (async but not streaming). Option B from earlier audit is the next step.
  • No hard size cap. A pathological transcript can still OOM the hook.

Happy to tackle streaming + size cap in a follow-up PR.

Test plan

  • Run a Claude Code session, trigger a Stop, confirm log Transcript transferred: <uuid>.jsonl, pending dir empty.
  • Block the ingest endpoint (e.g. edit the JWT to point at a bad host, or kill network), trigger Stop -> log shows kept pending for next session; pending file exists on disk.
  • Restart Claude Code session -> Retrying N pending transcript(s) appears at SessionStart; successful upload removes the pending file.
  • Multi-Stop in same session (long conversation) -> pending file is overwritten each time, not duplicated; single file per transcript_id.
  • Confirm Stop hook returns quickly (hook-timing logs unaffected by transcript size) -- the async change should eliminate the sync gzip stall.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Enhanced transcript upload reliability with automatic retry mechanism for failed uploads.
    • Staged transcript uploads are now preserved and retried in subsequent sessions.
  • Chores

    • Version bumped to 0.6.0.

Transcript uploads were fire-and-forget with no persistence: any failure
(network blip, 5xx, process exit before fetch completed) silently dropped
the transcript forever. This change makes the upload durable and moves
the heavy I/O off the event loop, without making the Stop hook wait.

logger.js:
  - New logs/transcripts/pending/ directory.
  - stageTranscriptForUpload: sanitize the JSONL and write a snapshot to
    pending/<transcript_id>. Long sessions re-stage on every Stop,
    overwriting the previous snapshot so the pending copy always holds
    the latest content.
  - uploadPendingTranscript: reads the pending snapshot, gzips, POSTs.
    On 2xx the pending file is unlinked; on any failure it is left for
    the next session to retry. Fire-and-forget preserved.
  - uploadPendingTranscript is now async and uses fs/promises.readFile +
    promisified zlib.gzip, so the Stop hook no longer blocks the event
    loop while compressing multi-MB transcripts.
  - retryFailedTranscripts: scans pending/ on SessionStart and fires an
    upload for every leftover. Added to logger.js exports.

session_start.js:
  - SessionStart now calls retryFailedTranscripts() alongside the
    existing retryFailedLogs() call when the user has opted in.

plugin.json: 0.5.1 -> 0.6.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

Plugin version incremented to 0.6.0. Transcript upload refactored to use two-step staging pipeline: stageTranscriptForUpload creates sanitized snapshots in a pending directory, uploadPendingTranscript asynchronously uploads and retries failed uploads, and retryFailedTranscripts rescans pending files on session start.

Changes

Cohort / File(s) Summary
Plugin Manifest
skillmeter/.claude-plugin/plugin.json
Version incremented from 0.5.1 to 0.6.0.
Transcript Upload Resilience
skillmeter/scripts/logger.js
Refactored upload flow to stage transcripts in a pending directory before async upload. Added stageTranscriptForUpload() to create sanitized snapshots, uploadPendingTranscript() to handle async gzip compression and HTTP uploads with error retention, and retryFailedTranscripts() to rescan and retry failed uploads. Pending files are retained on failures and cleaned up only on successful uploads.
Session Startup Integration
skillmeter/scripts/session_start.js
Added import of retryFailedTranscripts and invoke it after retryFailedLogs() in the telemetry-enabled session startup path to resume previously failed transcript uploads.

Sequence Diagram

sequenceDiagram
    participant Hook as Session/Hook
    participant Stage as Staging Layer
    participant FS as Pending Directory
    participant Upload as Upload Service
    participant HTTP as HTTP Endpoint

    Hook->>Stage: stageTranscriptForUpload(path)
    Stage->>FS: Create sanitized snapshot
    FS-->>Stage: Pending path
    Stage-->>Hook: Return pending path

    Hook->>Upload: uploadPendingTranscript(pendingPath)
    Upload->>FS: Read staged file
    FS-->>Upload: File contents
    Upload->>Upload: gzipAsync compress
    Upload->>HTTP: POST compressed data
    
    alt Upload Success (2xx)
        HTTP-->>Upload: Success response
        Upload->>FS: Delete pending file
        FS-->>Upload: Deleted
    else Upload Failure
        HTTP-->>Upload: Error response
        Upload->>FS: Retain pending file
        FS-->>Upload: File retained
        Upload-->>Hook: Log error, will retry
    end

    Hook->>Hook: Session restart
    Hook->>Upload: retryFailedTranscripts()
    Upload->>FS: Scan pending directory
    FS-->>Upload: List of pending files
    Upload->>Upload: Retry uploads for each file
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • jhunlee-dev

Poem

🐰 Transcripts now stage and wait their turn,
Pending files persist till uploads earn,
Sessions restart, retries take flight,
From 0.5.1 to 0.6.0—resilience done right! 📝✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main changes: durable transcript uploads with retry logic, async I/O operations, and version bump to 0.6.0, matching the primary objectives of the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/transcript-retry-and-async-upload

Comment @coderabbitai help to get the list of available commands and usage tips.

@minarc
minarc requested a review from jhunlee-dev April 24, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants