Skip to content

Integrate Claude harness metadata sanitization - #79

Merged
john-skillbench merged 2 commits into
mainfrom
feat/sbee-165-claude-harness-sanitization
Jun 26, 2026
Merged

john-skillbench merged 2 commits into
mainfrom
feat/sbee-165-claude-harness-sanitization

Conversation

@john-skillbench

@john-skillbench john-skillbench commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add Claude harness metadata collection on SessionStart for instruction files, skills, hooks, plugin/runtime info, and Level 2 unknown markers.
  • Add deterministic Tier 1/Tier 2 sanitizer primitives to support fail-closed harness scanning and policy-versioned metadata.
  • Route Claude harness metadata through sanitization and document the emitted schema/redaction bookkeeping.

Test plan

  • node --test skillmeter/test/harness.test.js
  • node --check skillmeter/scripts/session_start.js

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features

    • Added session-start “harness metadata” to SessionStart event payloads, including structured presence/shape details.
    • Introduced deterministic, nested-data sanitization with Tier 1/Tier 2 redaction bookkeeping and optional hashed skill names.
  • Bug Fixes

    • Improved safety: harness detection and parsing fail closed and never throw on missing/invalid files, preventing event-collection interruptions.
  • Documentation

    • Updated README to document the harness metadata schema, sanitization behavior, and relevant toggles.
  • Tests

    • Added unit tests covering harness detection, sanitization contracts, and repo-root discovery.

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ad6b15c4-b38d-47b9-8b47-3b9ca3c964cf

📥 Commits

Reviewing files that changed from the base of the PR and between 5d2dd50 and 4d230da.

📒 Files selected for processing (2)
  • skillmeter/README.md
  • skillmeter/scripts/lib/sanitize.js
🚧 Files skipped from review as they are similar to previous changes (2)
  • skillmeter/README.md
  • skillmeter/scripts/lib/sanitize.js

Walkthrough

The PR adds harness metadata detection and sanitization, then threads the sanitized harness payload into SessionStart. It also updates the README contract and adds tests for discovery, redaction, hook filtering, repo-root lookup, and failure handling.

Changes

Harness metadata flow

Layer / File(s) Summary
Sanitization policy and redaction
skillmeter/scripts/lib/sanitize.js
Adds policy constants, Tier 1/Tier 2 detectors, recursive redaction, and sanitizeEventData metadata output.
Harness discovery helpers
skillmeter/scripts/harness.js
Adds safe filesystem probes, repo-root lookup, bounded skill discovery, and allow-listed hook discovery.
Harness metadata assembly
skillmeter/scripts/harness.js
Builds the harness object from discovery output, applies truncation and hash/drop bookkeeping, and exports the detection API.
SessionStart wiring and docs
skillmeter/scripts/session_start.js, skillmeter/README.md
Threads sanitized harness metadata into SessionStart, logs redaction counts to stderr when needed, and updates the README contract and privacy sections.
Harness tests
skillmeter/test/harness.test.js
Covers detection defaults, scope discovery, hashing and drop behavior, hook filtering, sanitization boundaries, malformed input handling, and repo-root lookup.

Sequence Diagram(s)

sequenceDiagram
  participant session_start_js as session_start.js
  participant detectHarness
  participant sanitizeEventData
  participant stderr
  session_start_js->>detectHarness: detectHarness(ctx.cwd, options)
  detectHarness-->>session_start_js: harness metadata
  session_start_js->>sanitizeEventData: sanitizeEventData(harness)
  sanitizeEventData-->>session_start_js: sanitized harness + redaction counts
  session_start_js->>stderr: write redaction counts when tiers match
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • jhunlee-dev

Poem

I hopped through harness paths at dawn,
and tucked the secret strings away anon.
With hooks and scopes, I took my cue,
then hashed the names I shouldn’t view.
🐰 Thump-thump—safe metadata, bright and new.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.38% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding Claude harness metadata collection and sanitization.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/sbee-165-claude-harness-sanitization

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 OpenGrep (1.23.0)

OpenGrep fatal error (exit code 2):
┌──────────────┐
│ Opengrep CLI │
└──────────────┘

�[32m✔�[39m �[1mOpengrep OSS�[0m
�[32m✔�[39m Basic security coverage for first-party code vulnerabilities.

�[1m Loading rules from local config...�[0m
[00.14][ERROR]: Error: exception Glob.Lexer.Syntax_error("malformed glob pattern: missing ']'")
Raised at Glob__Lexer.syntax_error in file "libs/glob/Lexer.mll", line 8, characters 2-26
Called from Glob__Lexer.__ocaml_lex_token_rec in file "libs/glob/Lexer.mll", line 29, characters 26-53
Cal


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
skillmeter/test/harness.test.js (1)

31-49: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider tearing down the temp dirs created per test.

tmpDir/makeProject/makeHome create directories under os.tmpdir() on nearly every test but nothing removes them, so each run leaks several trees. A t.after(...) cleanup (or tracking created roots and fs.rmSync(..., { recursive: true, force: true })) keeps the suite hermetic.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/test/harness.test.js` around lines 31 - 49, The test helpers
tmpDir, makeProject, and makeHome create temporary directories under os.tmpdir()
without cleanup, causing leaked trees across runs. Add teardown in the harness
test file by tracking each created root and removing them with fs.rmSync(..., {
recursive: true, force: true }) in a t.after hook (or equivalent per-test
cleanup), and make sure any callers that use makeProject/makeHome are covered so
the suite stays hermetic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skillmeter/README.md`:
- Line 117: The SessionStart documentation row is out of sync with the payload
emitted by the handler and runHook. Update the SessionStart entry in README to
include the missing `model` and `agent_type` fields alongside `session_id`,
`permission_mode`, `source`, and `harness`, using the same naming as returned by
the SessionStart handler and added in runHook.

In `@skillmeter/scripts/lib/sanitize.js`:
- Around line 65-66: The repeated-character shortcut in sanitize.js is too
permissive because it returns arbitrary placeholders like 11111111 or aaaaaaaa
unchanged. Update the repeated single-character handling near the
trim/sanitization logic so only explicit mask-style placeholders such as xxxxx
or ***** are preserved, and do not treat any repeated character sequence as
safe; use the existing sanitize flow in sanitize() to route non-mask repeats
through masking/redaction instead of returning them verbatim.
- Around line 171-184: Update redactDeep in sanitize.js so object field names
are used as context when redacting nested values: when iterating Object.entries,
detect secret-labeled keys such as api_key, token, password, or similar, and
force Tier 1 redaction on their string values instead of only relying on
redactString scanning the leaf text. Keep the recursive handling for arrays and
objects, but pass the key context through the redactDeep path so
sanitizeEventData catches sensitive values even when the string itself has no
obvious secret pattern.

---

Nitpick comments:
In `@skillmeter/test/harness.test.js`:
- Around line 31-49: The test helpers tmpDir, makeProject, and makeHome create
temporary directories under os.tmpdir() without cleanup, causing leaked trees
across runs. Add teardown in the harness test file by tracking each created root
and removing them with fs.rmSync(..., { recursive: true, force: true }) in a
t.after hook (or equivalent per-test cleanup), and make sure any callers that
use makeProject/makeHome are covered so the suite stays hermetic.
🪄 Autofix (Beta)

✅ Autofix completed


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4e93542a-5263-40de-bdd8-e98b792b4acc

📥 Commits

Reviewing files that changed from the base of the PR and between 3ebcb73 and 5d2dd50.

📒 Files selected for processing (5)
  • skillmeter/README.md
  • skillmeter/scripts/harness.js
  • skillmeter/scripts/lib/sanitize.js
  • skillmeter/scripts/session_start.js
  • skillmeter/test/harness.test.js

Comment thread skillmeter/README.md Resolved
Comment thread skillmeter/scripts/lib/sanitize.js Resolved
Comment thread skillmeter/scripts/lib/sanitize.js Resolved
@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Note

Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it.

Fixes Applied Successfully

Fixed 2 file(s) based on 3 unresolved review comments.

Files modified:

  • skillmeter/README.md
  • skillmeter/scripts/lib/sanitize.js

Commit: 4d230dace2255ea50a0d2c5fe24ae00dd95718a8

The changes have been pushed to the feat/sbee-165-claude-harness-sanitization branch.

Time taken: 2m 5s

Fixed 2 file(s) based on 3 unresolved review comments.

Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant