Skip to content

feat(hooks): capture StopFailure, TaskCreated, PostCompact, PermissionDenied (v0.8.0) - #52

Merged
minarc merged 1 commit into
mainfrom
feat/high-value-hook-events
Apr 24, 2026
Merged

minarc merged 1 commit into
mainfrom
feat/high-value-hook-events

Conversation

@minarc

@minarc minarc commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add four observation-only hook events that fill visible gaps in the current telemetry pipeline. All non-blocking, ~10 lines per script, zero new dependencies.

Event Pairs with Value
StopFailure the existing Stop Captures API errors: rate_limit, authentication_failed, billing_error, invalid_request, server_error, max_output_tokens, unknown. Surfaces customer-blocking friction (billing, auth, context exhaustion).
TaskCreated the existing TaskCompleted Lets backend compute task lifetime by subtracting timestamps on matching task_id. Core dev-productivity metric.
PostCompact the existing PreCompact Pair gives compaction duration + frequency — proxy for "session hit the context wall".
PermissionDenied standalone Fires when auto-mode classifier rejects a tool call. Gives visibility into workflow-automation friction.

Implementation

Each script follows the existing runHook pattern — mirroring the shape of its sibling (stop_failure.js matches stop.js, task_created.js matches task_completed.js, post_compact.js matches pre_compact.js, permission_denied.js matches permission_request.js).

PermissionDenied runs tool_input through the same sanitizeToolData HMAC path as PreToolUse. All other events carry no path/PII-sensitive fields.

Intentional non-behaviour: PermissionDenied could return {retry: true} to let the model retry the denied tool. We do not — this is observation-only telemetry, no decision-modification.

Files

skillmeter/scripts/stop_failure.js      (+11)
skillmeter/scripts/task_created.js      (+13)
skillmeter/scripts/post_compact.js      (+9)
skillmeter/scripts/permission_denied.js (+12)
skillmeter/hooks/hooks.json             (+48)  # 4 registrations
skillmeter/.claude-plugin/plugin.json   0.7.0 → 0.8.0

Total event count: 18 → 22.

Test plan

  • Hit a billing wall / rate limit / long response → confirm StopFailure event shows up in backend logs with the correct error_type.
  • Use TaskCreate → later complete the task → confirm both TaskCreated and TaskCompleted events land with the same task_id.
  • Force a context compaction (or wait for auto-compact) → confirm PreCompact + PostCompact both fire.
  • Trigger a denied tool in auto mode (e.g. Write to a restricted path) → confirm PermissionDenied event with hashed tool_input.file_path.
  • Regression: 18 existing events continue to flow unchanged.

Deferred (low-value, per audit)

  • CwdChanged — supplementary to the existing repo-scope gate.
  • UserPromptExpansion — slash-command usage; niche.
  • PostToolBatch — parallel-batch completion; low signal.
  • Elicitation / ElicitationResult — MCP-specific; few users.
  • FileChanged — needs explicit matcher; not general-purpose.

Preserves the current fixed-endpoint + license-bypass overrides in lib/jwt.js and runHook. No server-schema migration — just accept the 4 new hook_event_name values additively.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added four new event hooks: PostCompact, StopFailure, TaskCreated, and PermissionDenied to enable additional extensibility points.
  • Chores

    • Bumped version from 0.7.0 to 0.8.0.

…nDenied

Four observation-only hook events added — each filling a visible gap in
the current telemetry pipeline. All follow the existing runHook pattern
(~10 lines per script, no blocking, no new lib/ dependencies).

StopFailure
  Complements Stop (happy-path turn end). Captures error_type (rate_limit,
  authentication_failed, billing_error, invalid_request, server_error,
  max_output_tokens, unknown) so the backend can surface friction: billing
  blocks, rate limits, context exhaustion.

TaskCreated
  Pairs with existing TaskCompleted. Together they let the backend compute
  task lifetime by subtracting timestamps on matching task_id. Same
  attribute shape as TaskCompleted for easy pair-up server-side.

PostCompact
  Pairs with existing PreCompact. Together they measure compaction
  duration and frequency — a proxy for "session hit the context wall".

PermissionDenied
  Fires when auto-mode classifier rejects a tool call. Gives visibility
  into workflow-automation friction (which tools Claude tried that got
  blocked). tool_input paths get the same HMAC-hashed sanitize pass as
  pre_tool_use.js. We intentionally do NOT return {retry: true} —
  capture only, no behaviour change.

Zero risk: no blocking hooks, no modification of inputs/outputs, no new
dependencies. Preserves the existing fixed-endpoint + license-bypass
overrides.

plugin.json: 0.7.0 -> 0.8.0 (minor bump: new event capture capability).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

Walkthrough

This pull request increments the plugin version from 0.7.0 to 0.8.0 and introduces four new event hooks (PostCompact, StopFailure, TaskCreated, PermissionDenied) via entries in the hooks manifest, each backed by a corresponding Node.js script that executes telemetry or state-tracking logic.

Changes

Cohort / File(s) Summary
Plugin Version
skillmeter/.claude-plugin/plugin.json
Version field updated from 0.7.0 to 0.8.0.
Hook Manifest
skillmeter/hooks/hooks.json
Four new event hook arrays added: PostCompact, StopFailure, TaskCreated, and PermissionDenied; each wired to a corresponding Node script with async: true and timeout: 10.
Hook Runner Scripts
skillmeter/scripts/permission_denied.js, skillmeter/scripts/post_compact.js, skillmeter/scripts/stop_failure.js, skillmeter/scripts/task_created.js
Four new Node.js CLI scripts implementing hook handlers: extracting telemetry data (permission denials, task creation details, compaction state, failure errors), processing payloads, and terminating with exit code 1 on rejection.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • jhunlee-dev

Poem

🐰 Four new hooks we plant with care,
Telemetry floating through the air,
PostCompact, TaskCreated, too,
Permission checks and failures true—
Version bumps from point-seven's light! 🌟

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding four new hook events (StopFailure, TaskCreated, PostCompact, PermissionDenied) and updating the version to 0.8.0, which matches the changeset exactly.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/high-value-hook-events

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@skillmeter/scripts/post_compact.js`:
- Around line 6-9: The code currently passes raw free-form text via
input.custom_instructions into runHook("PostCompact"), which can leak secrets;
instead sanitize before sending: replace input.custom_instructions with a
non-content signal (e.g., has_custom_instructions boolean, a length_bucket based
on input.custom_instructions.length, and/or a stable hash/trimmed fingerprint)
so no raw text is emitted. Update the payload construction in the
runHook("PostCompact", ...) call to compute and send only those derived fields
(referencing runHook, "PostCompact", and input.custom_instructions) and ensure
any fallback on empty/null is handled; do not include the original
custom_instructions string in telemetry.

In `@skillmeter/scripts/stop_failure.js`:
- Around line 8-10: The telemetry currently ships raw provider text via
input.error_message in the runHook("StopFailure", ...) payload; remove that
free-form field and instead send only the enum error_type plus a coarse flag
(e.g., error_message_present or has_error_message) to indicate whether a message
existed (or a small bucket like message_length_bucket) to avoid exposing
sensitive content. Update the runHook payload to drop input.error_message and
add the boolean/bucket flag while retaining input.error_type.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 976dba65-63b9-4361-8aa0-585a980f557f

📥 Commits

Reviewing files that changed from the base of the PR and between 5e4d964 and 244b18d.

📒 Files selected for processing (6)
  • skillmeter/.claude-plugin/plugin.json
  • skillmeter/hooks/hooks.json
  • skillmeter/scripts/permission_denied.js
  • skillmeter/scripts/post_compact.js
  • skillmeter/scripts/stop_failure.js
  • skillmeter/scripts/task_created.js

Comment on lines +6 to +9
runHook("PostCompact", (input) => ({
trigger: input.trigger,
custom_instructions: input.custom_instructions,
})).catch(() => process.exit(1));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Do not log raw custom_instructions content.

Line 8 sends free-form instruction text directly to telemetry, which can contain secrets or sensitive business data. Emit a non-content signal instead (presence/length bucket/hash), not raw text.

🔧 Proposed change
 runHook("PostCompact", (input) => ({
   trigger: input.trigger,
-  custom_instructions: input.custom_instructions,
+  custom_instructions_present: Boolean(input.custom_instructions),
 })).catch(() => process.exit(1));
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@skillmeter/scripts/post_compact.js` around lines 6 - 9, The code currently
passes raw free-form text via input.custom_instructions into
runHook("PostCompact"), which can leak secrets; instead sanitize before sending:
replace input.custom_instructions with a non-content signal (e.g.,
has_custom_instructions boolean, a length_bucket based on
input.custom_instructions.length, and/or a stable hash/trimmed fingerprint) so
no raw text is emitted. Update the payload construction in the
runHook("PostCompact", ...) call to compute and send only those derived fields
(referencing runHook, "PostCompact", and input.custom_instructions) and ensure
any fallback on empty/null is handled; do not include the original
custom_instructions string in telemetry.

Comment on lines +8 to +10
runHook("StopFailure", (input) => ({
error_type: input.error_type,
error_message: input.error_message,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid shipping raw provider error text in telemetry.

Line 10 logs input.error_message verbatim. Free-form API error text can include sensitive request/context fragments; this is a data-minimization risk. Prefer enum-only analytics (error_type) plus a coarse flag.

🔧 Proposed change
 runHook("StopFailure", (input) => ({
   error_type: input.error_type,
-  error_message: input.error_message,
+  error_message_present: Boolean(input.error_message),
 })).catch(() => process.exit(1));
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@skillmeter/scripts/stop_failure.js` around lines 8 - 10, The telemetry
currently ships raw provider text via input.error_message in the
runHook("StopFailure", ...) payload; remove that free-form field and instead
send only the enum error_type plus a coarse flag (e.g., error_message_present or
has_error_message) to indicate whether a message existed (or a small bucket like
message_length_bucket) to avoid exposing sensitive content. Update the runHook
payload to drop input.error_message and add the boolean/bucket flag while
retaining input.error_type.

@minarc
minarc merged commit a421936 into main Apr 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant