Skip to content

fix(transcript): resilient JSONL parse + disk cleanup sweep (v0.6.1) - #49

Merged
minarc merged 1 commit into
mainfrom
fix/transcript-resilient-parse-and-disk-cleanup
Apr 24, 2026
Merged

minarc merged 1 commit into
mainfrom
fix/transcript-resilient-parse-and-disk-cleanup

Conversation

@minarc

@minarc minarc commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two small-but-real gaps left over after PR #48. Both are data-integrity or disk-hygiene fixes with no product or server-side implications.

1. One bad line no longer loses the whole transcript

sanitizer.js parsed every JSONL line via JSON.parse(line) with no try/catch. A single partial line — easy to hit when a writer crashes mid-flush, or when embedded byte sequences confuse the tokenizer — would throw, propagate up through stageTranscriptForUpload, which catches the error and returns null. Net result: the pending snapshot is never written, the transcript has no retry path, the session's transcript is lost.

Now: the offending line is skipped, the rest of the transcript is sanitized and staged normally, and stderr gets one line:

[skillmeter] transcript: dropped 3 malformed line(s) during sanitize

2. Disk usage no longer grows forever

transferEventLog renames uploaded event logs to events.jsonl.<ts>.sent and never removes them. Pending transcripts behind a permanently-broken ingest accumulate too. On a laptop that's been running SkillMeter for a year with flaky upload, this is gigabytes of orphaned disk.

New cleanupStaleFiles() in logger.js deletes:

  • events.jsonl.*.sent files older than 30 days
  • Files under logs/transcripts/pending/ older than 30 days

Invoked from session_start.js after the retry funcs, so anything the current session just attempted to resend is preserved even on brief failures. mtime-based (not filename timestamp) so pending files that are still being retried (and thus get re-written on each Stop) don't get swept prematurely.

One stderr line when anything gets cleaned:

[skillmeter] Cleaned up 5 stale file(s) older than 30 days

Files changed

File Change
skillmeter/scripts/sanitizer.js try/catch around JSON.parse; drop malformed lines with stderr notice
skillmeter/scripts/logger.js add cleanupStaleFiles() helper; export it
skillmeter/scripts/session_start.js call cleanupStaleFiles() after retries
skillmeter/.claude-plugin/plugin.json 0.6.0 → 0.6.1

Test plan

Verified locally with two Node scripts (in this PR's commit history):

  • Resilient parsing: transcript with 1 good + 1 truncated + 1 good line → 2 valid lines output, cwd hashed, stderr reports 1 dropped. ✅
  • Cleanup: seeded fresh sandbox with one aged .sent file, one aged pending file, and one recent counterpart of each. Only the aged pair deleted. ✅

Regression: normal Stop → upload → pending removed path untouched (no changes to hot upload code).

Deferred audit items (tracked for a follow-up PR)

  • Max-size guard for transcripts (OOM on >100 MB transcript).
  • Retry exponential backoff.
  • Session-crash recovery for transcripts never seen through a Stop hook.
  • Sanitization scope vs. "anonymized" claim in the plugin description — needs a product call.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved error resilience: malformed transcript entries are now skipped gracefully instead of halting processing.
  • Chores

    • Version bumped to 0.6.1.
    • Added automatic cleanup of stale uploaded logs and pending transcripts exceeding 30 days old during session initialization.

Two quick-win gaps in the transcript collection path, both exposed by
the audit after PR #48.

1. One malformed JSONL line no longer kills the whole transcript.
   sanitizer.js wraps JSON.parse in try/catch, drops offending lines,
   and writes a single stderr notice with the count. Previously a
   trailing partial line from a crashed writer would throw, abort
   stageTranscriptForUpload, and lose the entire transcript with no
   retry.

2. Disk usage no longer creeps forever. logger.js gains
   cleanupStaleFiles() which deletes events.jsonl.*.sent files and
   transcripts/pending/* older than 30 days. Wired into
   session_start.js after the retry funcs so failed retries always
   survive at least one more session.

plugin.json: 0.6.0 -> 0.6.1 (patch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

The PR increments the plugin version to 0.6.1 and adds a stale file cleanup routine to the logger module that deletes event logs and transcripts exceeding 30 days. The sanitizer now handles malformed JSONL lines gracefully, and the session initialization flow invokes cleanup after retry logic.

Changes

Cohort / File(s) Summary
Plugin Version Bump
skillmeter/.claude-plugin/plugin.json
Version field incremented from 0.6.0 to 0.6.1.
Stale File Cleanup
skillmeter/scripts/logger.js, skillmeter/scripts/session_start.js
New cleanupStaleFiles() function added to logger module that removes stale event logs (events.jsonl.*.sent) and pending transcripts older than 30 days. Function is called in session initialization after retry logic when telemetry is enabled.
Sanitizer Error Resilience
skillmeter/scripts/sanitizer.js
Transcript processing now wraps JSON parsing and sanitization in try/catch to skip malformed lines instead of aborting. Counts and reports dropped malformed entries to stderr.

Sequence Diagram

sequenceDiagram
    participant SessionStart as Session Start
    participant Logger as Logger Module
    participant FileSystem as File System

    SessionStart->>SessionStart: Telemetry enabled?
    SessionStart->>SessionStart: retryFailedLogs()
    SessionStart->>SessionStart: retryFailedTranscripts()
    SessionStart->>Logger: cleanupStaleFiles()
    Logger->>FileSystem: Identify stale files<br/>(events.jsonl.*.sent,<br/>pending transcripts > 30 days)
    FileSystem-->>Logger: List of stale files
    Logger->>FileSystem: Delete stale files
    FileSystem-->>Logger: Deletion complete
    Logger-->>SessionStart: Cleanup done
    SessionStart->>SessionStart: Return true
Loading

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly Related PRs

Suggested Reviewers

  • jhunlee-dev

Poem

🐰 Old files piling high, what a mess!
Time to tidy up and clean our nest.
Stale transcripts vanish, errors caught with care,
A freshly bumped version floats through the air! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically summarizes the two main changes: resilient JSONL parsing and disk cleanup, with the version bump. It is concise and reflects the primary objectives.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/transcript-resilient-parse-and-disk-cleanup

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@skillmeter/scripts/session_start.js`:
- Around line 21-23: The calls to retryFailedTranscripts(), retryFailedLogs(),
and cleanupStaleFiles() create a race where cleanupStaleFiles() can delete
pending transcript files before uploadPendingTranscript() finishes; change the
sequence so cleanup runs only after retries complete by making
retryFailedTranscripts() return a Promise that awaits its internal
uploadPendingTranscript() operations and then awaiting it (e.g., await
retryFailedTranscripts()) before calling cleanupStaleFiles(); keep
retryFailedLogs() behavior but consider awaiting it in parallel (await
Promise.all([retryFailedLogs(), retryFailedTranscripts()]) then call
cleanupStaleFiles()) so stale-file cleanup cannot drop in-flight transcript
uploads.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c890d0b3-e0ee-490b-af82-6e99315e5259

📥 Commits

Reviewing files that changed from the base of the PR and between b9eb612 and 41c5986.

📒 Files selected for processing (4)
  • skillmeter/.claude-plugin/plugin.json
  • skillmeter/scripts/logger.js
  • skillmeter/scripts/sanitizer.js
  • skillmeter/scripts/session_start.js

Comment on lines 21 to +23
retryFailedLogs();
retryFailedTranscripts();
cleanupStaleFiles();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid retry/cleanup race that can drop pending transcripts.

retryFailedTranscripts() is fire-and-forget, so calling cleanupStaleFiles() immediately after can delete stale pending files before uploadPendingTranscript() finishes opening/reading them. This can silently lose a retry attempt.

💡 Minimal safe ordering fix
       process.stderr.write(`SkillMeter v${PLUGIN_VERSION} (activated)\n`);
-      retryFailedLogs();
-      retryFailedTranscripts();
-      cleanupStaleFiles();
+      cleanupStaleFiles();
+      retryFailedLogs();
+      retryFailedTranscripts();
       return true;
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
retryFailedLogs();
retryFailedTranscripts();
cleanupStaleFiles();
process.stderr.write(`SkillMeter v${PLUGIN_VERSION} (activated)\n`);
cleanupStaleFiles();
retryFailedLogs();
retryFailedTranscripts();
return true;
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@skillmeter/scripts/session_start.js` around lines 21 - 23, The calls to
retryFailedTranscripts(), retryFailedLogs(), and cleanupStaleFiles() create a
race where cleanupStaleFiles() can delete pending transcript files before
uploadPendingTranscript() finishes; change the sequence so cleanup runs only
after retries complete by making retryFailedTranscripts() return a Promise that
awaits its internal uploadPendingTranscript() operations and then awaiting it
(e.g., await retryFailedTranscripts()) before calling cleanupStaleFiles(); keep
retryFailedLogs() behavior but consider awaiting it in parallel (await
Promise.all([retryFailedLogs(), retryFailedTranscripts()]) then call
cleanupStaleFiles()) so stale-file cleanup cannot drop in-flight transcript
uploads.

@minarc
minarc merged commit 41c5986 into main Apr 24, 2026
1 check passed
@TheStarkor
TheStarkor deleted the fix/transcript-resilient-parse-and-disk-cleanup branch September 21, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant