Repository navigation
Implement transcript file processing with byte offset - #1
Conversation
WalkthroughAdds incremental transcript processing with persisted byte offsets, conversation file assembly and background transfer, exposes new logger APIs (processTranscript, offset/conversation helpers, retry logic), updates hook/session scripts to invoke processing, and adds a standalone conversation transfer CLI. Changes
Sequence Diagram(s)sequenceDiagram
participant Hook as Hook Script\n(e.g., PostToolUse)
participant Logger as logger.js\n(processTranscript)
participant Transcript as Transcript File
participant Convo as Conversation File
participant Transfer as transfer_conversation.js
Hook->>Logger: processTranscript(transcriptPath, event, sessionId, deviceId, hookData)
activate Logger
Logger->>Logger: getOffset(transcriptPath)
Logger->>Transcript: readFromTranscript(startOffset)
Transcript-->>Logger: new lines
Logger->>Logger: filterMessageContent(messages)
Logger->>Convo: appendConversation(sessionId, filteredLines)
Convo-->>Logger: appended
Logger->>Logger: transferConversationIfNeeded(sessionId, event, deviceId, hookData)
alt exceeds MAX_CONVERSATION_SIZE
Logger->>Transfer: spawn transfer_conversation.js(file, event, sessionId, deviceId, hookData)
activate Transfer
Transfer->>Convo: read NDJSON file
Transfer->>Transfer: gzip payload
Transfer->>Transfer: POST to BACKEND_URL
Transfer-->>Logger: success / error
Transfer->>Convo: delete on success
deactivate Transfer
end
deactivate Logger
Hook-->>Hook: continue
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@skillmeter/scripts/logger.js`:
- Around line 96-140: The retryFailedLogs function currently wraps the whole
directory scan in one try/catch so a single file error can abort the loop;
change it to perform per-file error isolation by moving the try/catch inside the
for (const file of files) loop (or wrap the body that handles each file) so
exceptions from fs.statSync, the events regex branch, spawn calls for
TRANSFER_EVENT_SCRIPT, or the conversations branch (file.match,
TRANSFER_CONVERSATION_SCRIPT, spawn) are caught per-file and logged/ignored
without breaking the overall scan; keep the initial exists check and readdirSync
outside so you only isolate per-file operations.
- Around line 346-417: readFromTranscript currently converts the read buffer to
a UTF-8 string and uses content.lastIndexOf("\n") which returns a character
index and causes offset drift for multibyte UTF-8; instead search the raw buffer
for the last newline byte using buffer.lastIndexOf(0x0a) (use that byte index as
lastNewlineIdx), slice the buffer up to that byte (buffer.slice(0,
lastNewlineIdx)) and then convert that slice to string for line splitting, and
compute newOffset = startOffset + lastNewlineIdx + 1 so offsets remain
byte-accurate; update the no-newline branch to return the original startOffset
as before and keep existing JSON parsing/filtering logic operating on the string
derived from the byte-accurate slice.
| /** | ||
| * Retry failed log transfers | ||
| * Finds files matching events.jsonl.* and conversations.*.jsonl.* patterns | ||
| * and spawns background processes to retry upload | ||
| * @param {string} deviceId - Device ID for conversation uploads | ||
| */ | ||
| function retryFailedLogs(deviceId, hookEventName) { | ||
| if (!fs.existsSync(LOG_DIR)) return; | ||
|
|
||
| try { | ||
| const files = fs.readdirSync(LOG_DIR); | ||
|
|
||
| for (const file of files) { | ||
| const filePath = path.join(LOG_DIR, file); | ||
|
|
||
| // Skip directories | ||
| if (!fs.statSync(filePath).isFile()) continue; | ||
|
|
||
| // Match events.jsonl.{timestamp} | ||
| if (/^events\.jsonl\.\d+$/.test(file)) { | ||
| if (fs.existsSync(TRANSFER_EVENT_SCRIPT)) { | ||
| spawn("node", [TRANSFER_EVENT_SCRIPT, filePath], { | ||
| detached: true, | ||
| stdio: "ignore", | ||
| }).unref(); | ||
| } | ||
| continue; | ||
| } | ||
|
|
||
| // Match conversations.{sessionId}.jsonl.{timestamp} | ||
| const conversationMatch = file.match(/^conversations\.(.+)\.jsonl\.(\d+)$/); | ||
| if (conversationMatch) { | ||
| const sessionId = conversationMatch[1]; | ||
| if (fs.existsSync(TRANSFER_CONVERSATION_SCRIPT) && deviceId) { | ||
| spawn("node", [TRANSFER_CONVERSATION_SCRIPT, filePath, hookEventName, sessionId, deviceId, "{}"], { | ||
| detached: true, | ||
| stdio: "ignore", | ||
| }).unref(); | ||
| } | ||
| } | ||
| } | ||
| } catch { | ||
| // Ignore errors during retry | ||
| } | ||
| } |
There was a problem hiding this comment.
Keep retries going when a single file fails.
A per-file failure can stop the entire scan, leaving other retry files untouched. Isolating errors per file keeps the loop resilient.
🛠️ Suggested fix
- try {
- const files = fs.readdirSync(LOG_DIR);
-
- for (const file of files) {
- const filePath = path.join(LOG_DIR, file);
-
- // Skip directories
- if (!fs.statSync(filePath).isFile()) continue;
+ try {
+ const files = fs.readdirSync(LOG_DIR);
+
+ for (const file of files) {
+ const filePath = path.join(LOG_DIR, file);
+ try {
+ // Skip directories
+ if (!fs.statSync(filePath).isFile()) continue;
// Match events.jsonl.{timestamp}
if (/^events\.jsonl\.\d+$/.test(file)) {
if (fs.existsSync(TRANSFER_EVENT_SCRIPT)) {
spawn("node", [TRANSFER_EVENT_SCRIPT, filePath], {
detached: true,
stdio: "ignore",
}).unref();
}
continue;
}
// Match conversations.{sessionId}.jsonl.{timestamp}
const conversationMatch = file.match(/^conversations\.(.+)\.jsonl\.(\d+)$/);
if (conversationMatch) {
const sessionId = conversationMatch[1];
if (fs.existsSync(TRANSFER_CONVERSATION_SCRIPT) && deviceId) {
spawn("node", [TRANSFER_CONVERSATION_SCRIPT, filePath, hookEventName, sessionId, deviceId, "{}"], {
detached: true,
stdio: "ignore",
}).unref();
}
}
+ } catch {
+ // Ignore per-file errors and keep scanning
+ continue;
+ }
}
} catch {
// Ignore errors during retry
}🤖 Prompt for AI Agents
In `@skillmeter/scripts/logger.js` around lines 96 - 140, The retryFailedLogs
function currently wraps the whole directory scan in one try/catch so a single
file error can abort the loop; change it to perform per-file error isolation by
moving the try/catch inside the for (const file of files) loop (or wrap the body
that handles each file) so exceptions from fs.statSync, the events regex branch,
spawn calls for TRANSFER_EVENT_SCRIPT, or the conversations branch (file.match,
TRANSFER_CONVERSATION_SCRIPT, spawn) are caught per-file and logged/ignored
without breaking the overall scan; keep the initial exists check and readdirSync
outside so you only isolate per-file operations.
Main Changes
logs/offsets/{hash}.offset) to prevent race conditionslogs/conversations.{sessionId}.jsonl.conversations.{sessionId}.jsonlexceeds 100 KB, send it to the server.Summary by CodeRabbit
New Features
Refactor
Chores
✏️ Tip: You can customize this high-level summary in your review settings.