Skip to content

Implement transcript file processing with byte offset - #1

Merged
minarc merged 3 commits into
mainfrom
extract-conversation
Feb 5, 2026
Merged

minarc merged 3 commits into
mainfrom
extract-conversation

Conversation

@jhunlee-dev

@jhunlee-dev jhunlee-dev commented Jan 24, 2026 •

Copy link
Copy Markdown
Collaborator

Main Changes

  • Added byte offset-based transcript processing for efficiency
  • Used per-transcript offset files(saved in logs/offsets/{hash}.offset) to prevent race conditions
  • On each hook event, the conversation is extracted from the transcript file and written to logs/conversations.{sessionId}.jsonl.
  • If the size of conversations.{sessionId}.jsonl exceeds 100 KB, send it to the server.
  • On SessionStart hook, retry the requests of all failed log

Summary by CodeRabbit

  • New Features

    • Incremental transcript processing for faster, partial reads
    • Asynchronous/background conversation transfer with retry support
    • Conversation persistence with content filtering and archival support
  • Refactor

    • Streamlined session lifecycle and transcript finalization flow
    • Centralized transcript handling across session events
  • Chores

    • Updated ignore list to include additional temporary entry

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai

coderabbitai Bot commented Jan 24, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

Adds incremental transcript processing with persisted byte offsets, conversation file assembly and background transfer, exposes new logger APIs (processTranscript, offset/conversation helpers, retry logic), updates hook/session scripts to invoke processing, and adds a standalone conversation transfer CLI.

Changes

Cohort / File(s) Summary
Logger Core
skillmeter/scripts/logger.js
New offset dir and helpers (getOffset, saveOffset, getOffsetFilePath), incremental reader (readFromTranscript, processTranscript), conversation helpers (getConversationFilePath, appendConversation), content filtering (filterMessageContent), transfer orchestration (transferConversationIfNeeded), retry logic (retryFailedLogs), exports MAX_CONVERSATION_SIZE and several new APIs; replaced transfer constant with event/conversation scripts.
Hook & Event Scripts
skillmeter/scripts/post_tool_use.js, skillmeter/scripts/user_prompt_submit.js, skillmeter/scripts/stop.js, skillmeter/scripts/session_start.js
Import processTranscript and conditionally call processTranscript(transcriptPath, ...) when transcript_path present; session_start.js also calls retryFailedLogs. stop.js and others updated to use TRANSFER_EVENT_SCRIPT.
Session End Flow
skillmeter/scripts/session_end.js
Rewrote SessionEnd to log a simplified payload, delegate transcript handling to processTranscript, and, for specific reasons, rename conversation files and spawn transfer_conversation.js instead of prior HTTP/zip dispatch.
Transfer CLI
skillmeter/scripts/transfer_conversation.js
New standalone script: reads NDJSON conversation file, gzips metadata+conversation payload, POSTs to BACKEND_URL (http/https), supports API key and timeout, deletes file on success and reports errors.
Misc
.gitignore
Added CLAUDE.md to ignored files.

Sequence Diagram(s)

sequenceDiagram
    participant Hook as Hook Script\n(e.g., PostToolUse)
    participant Logger as logger.js\n(processTranscript)
    participant Transcript as Transcript File
    participant Convo as Conversation File
    participant Transfer as transfer_conversation.js

    Hook->>Logger: processTranscript(transcriptPath, event, sessionId, deviceId, hookData)
    activate Logger
    Logger->>Logger: getOffset(transcriptPath)
    Logger->>Transcript: readFromTranscript(startOffset)
    Transcript-->>Logger: new lines
    Logger->>Logger: filterMessageContent(messages)
    Logger->>Convo: appendConversation(sessionId, filteredLines)
    Convo-->>Logger: appended
    Logger->>Logger: transferConversationIfNeeded(sessionId, event, deviceId, hookData)
    alt exceeds MAX_CONVERSATION_SIZE
        Logger->>Transfer: spawn transfer_conversation.js(file, event, sessionId, deviceId, hookData)
        activate Transfer
        Transfer->>Convo: read NDJSON file
        Transfer->>Transfer: gzip payload
        Transfer->>Transfer: POST to BACKEND_URL
        Transfer-->>Logger: success / error
        Transfer->>Convo: delete on success
        deactivate Transfer
    end
    deactivate Logger
    Hook-->>Hook: continue
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I nibble offsets, hop through lines so neat,
I stitch chatter into files with tiny feet.
Background paws tote conversations away,
Compressing and posting at break of day—
Little hops, big logs, a carrot-shaped hooray! 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 76.19% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'Implement transcript file processing with byte offset' directly and clearly summarizes the main change: adding byte-offset-based transcript processing to the logger module, which is reflected in the substantial changes to skillmeter/scripts/logger.js and its integration across multiple hook scripts.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@skillmeter/scripts/logger.js`:
- Around line 96-140: The retryFailedLogs function currently wraps the whole
directory scan in one try/catch so a single file error can abort the loop;
change it to perform per-file error isolation by moving the try/catch inside the
for (const file of files) loop (or wrap the body that handles each file) so
exceptions from fs.statSync, the events regex branch, spawn calls for
TRANSFER_EVENT_SCRIPT, or the conversations branch (file.match,
TRANSFER_CONVERSATION_SCRIPT, spawn) are caught per-file and logged/ignored
without breaking the overall scan; keep the initial exists check and readdirSync
outside so you only isolate per-file operations.
- Around line 346-417: readFromTranscript currently converts the read buffer to
a UTF-8 string and uses content.lastIndexOf("\n") which returns a character
index and causes offset drift for multibyte UTF-8; instead search the raw buffer
for the last newline byte using buffer.lastIndexOf(0x0a) (use that byte index as
lastNewlineIdx), slice the buffer up to that byte (buffer.slice(0,
lastNewlineIdx)) and then convert that slice to string for line splitting, and
compute newOffset = startOffset + lastNewlineIdx + 1 so offsets remain
byte-accurate; update the no-newline branch to return the original startOffset
as before and keep existing JSON parsing/filtering logic operating on the string
derived from the byte-accurate slice.

Comment on lines +96 to +140
/**
* Retry failed log transfers
* Finds files matching events.jsonl.* and conversations.*.jsonl.* patterns
* and spawns background processes to retry upload
* @param {string} deviceId - Device ID for conversation uploads
*/
function retryFailedLogs(deviceId, hookEventName) {
if (!fs.existsSync(LOG_DIR)) return;

try {
const files = fs.readdirSync(LOG_DIR);

for (const file of files) {
const filePath = path.join(LOG_DIR, file);

// Skip directories
if (!fs.statSync(filePath).isFile()) continue;

// Match events.jsonl.{timestamp}
if (/^events\.jsonl\.\d+$/.test(file)) {
if (fs.existsSync(TRANSFER_EVENT_SCRIPT)) {
spawn("node", [TRANSFER_EVENT_SCRIPT, filePath], {
detached: true,
stdio: "ignore",
}).unref();
}
continue;
}

// Match conversations.{sessionId}.jsonl.{timestamp}
const conversationMatch = file.match(/^conversations\.(.+)\.jsonl\.(\d+)$/);
if (conversationMatch) {
const sessionId = conversationMatch[1];
if (fs.existsSync(TRANSFER_CONVERSATION_SCRIPT) && deviceId) {
spawn("node", [TRANSFER_CONVERSATION_SCRIPT, filePath, hookEventName, sessionId, deviceId, "{}"], {
detached: true,
stdio: "ignore",
}).unref();
}
}
}
} catch {
// Ignore errors during retry
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Keep retries going when a single file fails.
A per-file failure can stop the entire scan, leaving other retry files untouched. Isolating errors per file keeps the loop resilient.

🛠️ Suggested fix
-  try {
-    const files = fs.readdirSync(LOG_DIR);
-
-    for (const file of files) {
-      const filePath = path.join(LOG_DIR, file);
-
-      // Skip directories
-      if (!fs.statSync(filePath).isFile()) continue;
+  try {
+    const files = fs.readdirSync(LOG_DIR);
+
+    for (const file of files) {
+      const filePath = path.join(LOG_DIR, file);
+      try {
+        // Skip directories
+        if (!fs.statSync(filePath).isFile()) continue;
 
       // Match events.jsonl.{timestamp}
       if (/^events\.jsonl\.\d+$/.test(file)) {
         if (fs.existsSync(TRANSFER_EVENT_SCRIPT)) {
           spawn("node", [TRANSFER_EVENT_SCRIPT, filePath], {
             detached: true,
             stdio: "ignore",
           }).unref();
         }
         continue;
       }
 
       // Match conversations.{sessionId}.jsonl.{timestamp}
       const conversationMatch = file.match(/^conversations\.(.+)\.jsonl\.(\d+)$/);
       if (conversationMatch) {
         const sessionId = conversationMatch[1];
         if (fs.existsSync(TRANSFER_CONVERSATION_SCRIPT) && deviceId) {
           spawn("node", [TRANSFER_CONVERSATION_SCRIPT, filePath, hookEventName, sessionId, deviceId, "{}"], {
             detached: true,
             stdio: "ignore",
           }).unref();
         }
       }
+      } catch {
+        // Ignore per-file errors and keep scanning
+        continue;
+      }
     }
   } catch {
     // Ignore errors during retry
   }
🤖 Prompt for AI Agents
In `@skillmeter/scripts/logger.js` around lines 96 - 140, The retryFailedLogs
function currently wraps the whole directory scan in one try/catch so a single
file error can abort the loop; change it to perform per-file error isolation by
moving the try/catch inside the for (const file of files) loop (or wrap the body
that handles each file) so exceptions from fs.statSync, the events regex branch,
spawn calls for TRANSFER_EVENT_SCRIPT, or the conversations branch (file.match,
TRANSFER_CONVERSATION_SCRIPT, spawn) are caught per-file and logged/ignored
without breaking the overall scan; keep the initial exists check and readdirSync
outside so you only isolate per-file operations.

Comment thread skillmeter/scripts/logger.js Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants