Skip to content

feat(harness): collect full Level-1 harness metadata (SBEE-166) - #80

Merged
john-skillbench merged 1 commit into
mainfrom
SBEE-166-phase-1-implement-harness-metadata-collection-in
Jun 26, 2026
Merged

john-skillbench merged 1 commit into
mainfrom
SBEE-166-phase-1-implement-harness-metadata-collection-in

Conversation

@john-skillbench

@john-skillbench john-skillbench commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Phase 1 (SBEE-166) of the Harness Detection & Collection epic (SBEE-163): implements the locked SBEE-164 metadata contract (spec/harness-metadata-contract.v1.json) in the Claude collector.

detectHarness now emits the contract's flat data.harness field set (replacing the earlier nested shape from the SBEE-165 sanitization work), expanding coverage from presence-only to the full agreed Level-1 surface.

What's newly collected

  • Instruction files: claude_md_count (nested), claude_md_size_bucket, claude_md_import_count, has_claude_local_md, has_user_claude_md
  • Skills: skills_present + skill_source_counts {project,user,plugin}
  • Subagents (.claude/agents/*.md): present / count / hashed names / subagent_used
  • Slash commands (.claude/commands/**/*.md)
  • MCP servers (.mcp.json / ~/.claude.json) — server names only, never command/args/env
  • Plugins + marketplaces (~/.claude/plugins manifests)
  • Runtime: model, session_source, plugin_version

Privacy

Metadata only — no raw file contents, hook command strings, MCP env, or file paths. tier2 identifiers (skill/subagent/command/MCP/private-plugin names) are HMAC-hashed; public-marketplace plugin names stay raw; tier1 secrets are dropped fail-closed; everything is tallied in redactions. Detection is filesystem-only, depth/dir-bounded, and never throws out of SessionStart.

Tests

  • harness.test.js rewritten for the flat shape — 20/20 passing (node --test).
  • A contract-coverage check confirms the emitted keys match the contract exactly; the only contract field not emitted is hook_commands, which the contract marks action: drop.

Notes

  • HARNESS_SCHEMA_VERSION is now "1.0" (string) to match the contract.
  • Name-hashing now defaults on (names dropped when no salt is present) to honor the contract's Tier-2 rule; production session_start always supplies a salt.
  • No plugin-version bump included — leaving that to the team's separate release PR.
  • Level 2 signals (external_orchestration / multi_agent) remain "unknown" pending SBEE-168.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Updated harness detection to better recognize newer metadata layouts and report more accurate counts.
    • Improved handling of missing or unsafe data so sensitive names are dropped instead of exposed.
    • Refined plugin, hook, skill, and MCP detection to produce cleaner, more consistent metadata.
  • New Features

    • Added additional session details to the generated harness metadata for downstream processing.
    • Expanded redaction tracking to make metadata handling more transparent.

Phase 1 of the harness detection epic (SBEE-163): implement the locked
SBEE-164 metadata contract in the Claude collector. detectHarness now emits
the contract's flat data.harness field set instead of the earlier nested
shape, adding detection for:

  - instruction files: claude_md_count, size bucket, @import count,
    has_claude_local_md, has_user_claude_md
  - skills: skills_present + per-source counts (project/user/plugin)
  - subagents (.claude/agents/*.md): present/count/hashed names/used
  - slash commands (.claude/commands/**/*.md)
  - MCP servers (.mcp.json / ~/.claude.json) — names only, never command/env
  - plugins + marketplaces (~/.claude/plugins manifests)
  - runtime: model, session_source, plugin_version

Metadata only — no raw file contents, hook command strings, MCP env, or
file paths. tier2 names are HMAC-hashed (public-marketplace plugin names
kept raw); tier1 secrets dropped fail-closed; all tallied in `redactions`.
Detection stays filesystem-only, depth/dir-bounded, and never throws out of
SessionStart. session_start.js now threads model/source into detection.

Tests rewritten for the flat shape (20 passing); a contract-coverage check
confirms the emitted keys match spec/harness-metadata-contract.v1.json
exactly (hook_commands intentionally dropped per the contract).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The PR changes harness detection to a flat SBEE-164 metadata contract, adds bounded filesystem and JSON scanning with fail-closed redaction, updates hook/MCP/plugin collection, and passes session model/source into harness detection.

Changes

Harness metadata contract update

Layer / File(s) Summary
Contract, version, and scan bounds
skillmeter/scripts/harness.js
HARNESS_SCHEMA_VERSION changes to "1.0", the module docs describe the flat contract, and new scan limits, skip sets, and safeReadJson are added.
Name hashing and flat detector output
skillmeter/scripts/harness.js
hashNames adds fail-closed redaction handling, and detectHarness starts emitting the flat harness fields from bounded skill, command, agent, and instruction scans.
Hook, MCP, plugin, and session wiring
skillmeter/scripts/harness.js, skillmeter/scripts/session_start.js
detectHarness finishes hook counting, MCP server hashing, and plugin metadata collection, exports PUBLIC_MARKETPLACES, and session_start.js forwards model and sessionSource.

Sequence Diagram(s)

sequenceDiagram
  participant SessionStart as "session_start.js"
  participant DetectHarness as "detectHarness"
  participant HashNames as "hashNames"
  participant DetectPlugins as "detectPlugins"

  SessionStart->>DetectHarness: model, sessionSource
  DetectHarness->>HashNames: hash collected names
  DetectHarness->>DetectPlugins: plugin and marketplace metadata
  DetectHarness-->>SessionStart: flat harness metadata
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • jhunlee-dev

Poem

🐇 I hopped through hooks and names tonight,
And flattened metadata to light.
The redactions pruned the thorny bits,
While model and session_source fit.
Now harness fields go leaping by—hop, hop, tidy as the sky!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: collecting flat Level-1 harness metadata for the harness contract.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch SBEE-166-phase-1-implement-harness-metadata-collection-in

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 OpenGrep (1.23.0)

OpenGrep fatal error (exit code 2):
┌──────────────┐
│ Opengrep CLI │
└──────────────┘

�[32m✔�[39m �[1mOpengrep OSS�[0m
�[32m✔�[39m Basic security coverage for first-party code vulnerabilities.

�[1m Loading rules from local config...�[0m
[00.14][ERROR]: Error: exception Glob.Lexer.Syntax_error("malformed glob pattern: missing ']'")
Raised at Glob__Lexer.syntax_error in file "libs/glob/Lexer.mll", line 8, characters 2-26
Called from Glob__Lexer.__ocaml_lex_token_rec in file "libs/glob/Lexer.mll", line 29, characters 26-53
Cal


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skillmeter/scripts/harness.js`:
- Around line 165-167: Both safeReadJson() and countImports() currently read
entire files synchronously before any limit is applied, so oversized
settings.json, .mcp.json, .claude.json, or CLAUDE.md files can still stall or
exhaust memory. Update these helpers to use a bounded read helper that caps
bytes read before parsing/counting, and apply it wherever fs.readFileSync is
used in safeReadJson() and countImports(). Ensure the same bounded approach is
used for the related read paths referenced by the same logic.
- Around line 590-599: The plugin scan in harness.js is using the same capped
key list for both emitted names and install path collection, which causes later
plugins to be missed from skills_count and skill_source_counts.plugin. Separate
the exact iteration used to gather installPaths from the NAMES_LIMIT-capped loop
used to emit plugin names, and keep the install path aggregation in the logic
around the keys/entry processing so all plugins contribute to source counts
while only names remain capped.
- Around line 302-310: The hook counting logic in harness.js is skipping
non-allowlisted events too early, so `hooks_count` and `hooks_source_counts`
miss configured hooks even though `hooks_enabled` stays correct. Update the loop
around `hooks` processing to always count entries for every event, and only gate
the `events.push(event)` behavior behind `KNOWN_HOOK_EVENTS`; keep using the
existing `hooks` iteration and `group.hooks` inner counting logic so unknown
events still contribute to totals.
- Around line 344-347: Normalize the runtime metadata fields in the harness
output so they always serialize as strings. In the harness logic that builds the
flat metadata object, update the handling of model and source-related inputs
(along with the existing agent_type/session_source/plugin_version fields) to
coerce any truthy non-string values into string form before emission, so the
contract from session_start.js is preserved. Use the existing metadata assembly
in the harness script to locate the fix and keep the output schema string-only.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8388233b-28f7-4423-a6bc-5c0268b2c215

📥 Commits

Reviewing files that changed from the base of the PR and between e9279af and 74764d5.

📒 Files selected for processing (3)
  • skillmeter/scripts/harness.js
  • skillmeter/scripts/session_start.js
  • skillmeter/test/harness.test.js

Comment on lines +165 to +167
function safeReadJson(p) {
try {
return JSON.parse(fs.readFileSync(p, "utf8"));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Cap file reads before parsing/counting.

safeReadJson() and countImports() synchronously read the full file. A huge settings.json, .mcp.json, .claude.json, or CLAUDE.md can still stall/OOM the SessionStart hook despite the bounded directory walks.

Proposed bounded read helper
+const MAX_HARNESS_FILE_BYTES = 256 * 1024;
+
+function safeReadText(p) {
+  try {
+    const stat = fs.statSync(p);
+    if (!stat.isFile() || stat.size > MAX_HARNESS_FILE_BYTES) return "";
+    return fs.readFileSync(p, "utf8");
+  } catch {
+    return "";
+  }
+}
+
 function safeReadJson(p) {
   try {
-    return JSON.parse(fs.readFileSync(p, "utf8"));
+    const text = safeReadText(p);
+    return text ? JSON.parse(text) : null;
   } catch {
     return null;
   }
 }
 function countImports(claudeMdPath) {
   try {
-    const text = fs.readFileSync(claudeMdPath, "utf8");
+    const text = safeReadText(claudeMdPath);
     const matches = text.match(/(?:^|\s)@[^\s@]+/g);

Also applies to: 282-285

🧰 Tools
🪛 ast-grep (0.44.0)

[warning] 166-166: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(p, "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(detect-non-literal-fs-filename)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/harness.js` around lines 165 - 167, Both safeReadJson()
and countImports() currently read entire files synchronously before any limit is
applied, so oversized settings.json, .mcp.json, .claude.json, or CLAUDE.md files
can still stall or exhaust memory. Update these helpers to use a bounded read
helper that caps bytes read before parsing/counting, and apply it wherever
fs.readFileSync is used in safeReadJson() and countImports(). Ensure the same
bounded approach is used for the related read paths referenced by the same
logic.

Comment on lines +302 to +310
for (const [event, groups] of Object.entries(hooks)) {
if (!KNOWN_HOOK_EVENTS.has(event)) continue;
events.push(event);
if (Array.isArray(groups)) {
for (const group of groups) {
// Each matcher group may hold an inner `hooks` array of commands; count
// those when present, otherwise count the group itself as one entry.
const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0;
entries += inner > 0 ? inner : 1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Count hook entries even when event names are not emitted.

Line 303 drops unknown/non-allowlisted hook events before counting their entries. That preserves hooks_enabled, but makes hooks_count and hooks_source_counts falsely report fewer configured hooks.

Proposed fix
   for (const [event, groups] of Object.entries(hooks)) {
-    if (!KNOWN_HOOK_EVENTS.has(event)) continue;
-    events.push(event);
+    if (KNOWN_HOOK_EVENTS.has(event)) events.push(event);
     if (Array.isArray(groups)) {
       for (const group of groups) {
         // Each matcher group may hold an inner `hooks` array of commands; count
         // those when present, otherwise count the group itself as one entry.
-        const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0;
-        entries += inner > 0 ? inner : 1;
+        entries += group && Array.isArray(group.hooks) ? group.hooks.length : 1;
       }
     }
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for (const [event, groups] of Object.entries(hooks)) {
if (!KNOWN_HOOK_EVENTS.has(event)) continue;
events.push(event);
if (Array.isArray(groups)) {
for (const group of groups) {
// Each matcher group may hold an inner `hooks` array of commands; count
// those when present, otherwise count the group itself as one entry.
const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0;
entries += inner > 0 ? inner : 1;
for (const [event, groups] of Object.entries(hooks)) {
if (KNOWN_HOOK_EVENTS.has(event)) events.push(event);
if (Array.isArray(groups)) {
for (const group of groups) {
// Each matcher group may hold an inner `hooks` array of commands; count
// those when present, otherwise count the group itself as one entry.
entries += group && Array.isArray(group.hooks) ? group.hooks.length : 1;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/harness.js` around lines 302 - 310, The hook counting
logic in harness.js is skipping non-allowlisted events too early, so
`hooks_count` and `hooks_source_counts` miss configured hooks even though
`hooks_enabled` stays correct. Update the loop around `hooks` processing to
always count entries for every event, and only gate the `events.push(event)`
behavior behind `KNOWN_HOOK_EVENTS`; keep using the existing `hooks` iteration
and `group.hooks` inner counting logic so unknown events still contribute to
totals.

Comment on lines 344 to +347
agent_type: options.agentType || "claude-code",
instructions: { has_claude_md: false, has_claude_md_global: false, has_agents_md: false, has_agents_md_global: false, scopes: [] },
skills: { count: 0, names: [], scopes: [] },
hooks: { enabled: [], scopes: [] },
// Level 2 — architecture-level signals are not detectable from the
// filesystem/transcript, so they are explicitly "unknown" rather than a
// misleading false. subagent_used is derived downstream from the
// SubagentStart/SubagentStop events the collector already emits.
orchestration: {
external_orchestration: "unknown",
multi_agent: "unknown",
},
// Sanitization bookkeeping (SBEE-165, Phase 2): how many harness values
// were HMAC-hashed (Tier 2 names) or dropped (Tier 1 fail-closed) at this
// boundary, broken down by field type. Counts/types only — the original
// names/values never appear here. Complements the central sanitizeEventData
// summary, which only ever sees the already hashed/dropped result.
model: options.model || "",
session_source: options.sessionSource || "",
plugin_version: options.pluginVersion || "",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Normalize runtime fields to contract strings.

session_start.js now forwards input.model and input.source directly; truthy non-strings would be emitted as non-string harness fields and break the flat metadata contract.

Proposed normalization
+const stringOrEmpty = (value) => (typeof value === "string" ? value : "");
+
 function detectHarness(cwd, options = {}) {
   const harness = {
@@
-    agent_type: options.agentType || "claude-code",
-    model: options.model || "",
-    session_source: options.sessionSource || "",
-    plugin_version: options.pluginVersion || "",
+    agent_type: stringOrEmpty(options.agentType) || "claude-code",
+    model: stringOrEmpty(options.model),
+    session_source: stringOrEmpty(options.sessionSource),
+    plugin_version: stringOrEmpty(options.pluginVersion),
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
agent_type: options.agentType || "claude-code",
instructions: { has_claude_md: false, has_claude_md_global: false, has_agents_md: false, has_agents_md_global: false, scopes: [] },
skills: { count: 0, names: [], scopes: [] },
hooks: { enabled: [], scopes: [] },
// Level 2 — architecture-level signals are not detectable from the
// filesystem/transcript, so they are explicitly "unknown" rather than a
// misleading false. subagent_used is derived downstream from the
// SubagentStart/SubagentStop events the collector already emits.
orchestration: {
external_orchestration: "unknown",
multi_agent: "unknown",
},
// Sanitization bookkeeping (SBEE-165, Phase 2): how many harness values
// were HMAC-hashed (Tier 2 names) or dropped (Tier 1 fail-closed) at this
// boundary, broken down by field type. Counts/types only — the original
// names/values never appear here. Complements the central sanitizeEventData
// summary, which only ever sees the already hashed/dropped result.
model: options.model || "",
session_source: options.sessionSource || "",
plugin_version: options.pluginVersion || "",
const stringOrEmpty = (value) => (typeof value === "string" ? value : "");
function detectHarness(cwd, options = {}) {
const harness = {
agent_type: stringOrEmpty(options.agentType) || "claude-code",
model: stringOrEmpty(options.model),
session_source: stringOrEmpty(options.sessionSource),
plugin_version: stringOrEmpty(options.pluginVersion),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/harness.js` around lines 344 - 347, Normalize the runtime
metadata fields in the harness output so they always serialize as strings. In
the harness logic that builds the flat metadata object, update the handling of
model and source-related inputs (along with the existing
agent_type/session_source/plugin_version fields) to coerce any truthy non-string
values into string form before emission, so the contract from session_start.js
is preserved. Use the existing metadata assembly in the harness script to locate
the fix and keep the output schema string-only.

Comment on lines +590 to +599
for (const key of keys.slice(0, NAMES_LIMIT)) {
const at = key.lastIndexOf("@");
const name = at >= 0 ? key.slice(0, at) : key;
const marketplace = at >= 0 ? key.slice(at + 1) : "";
const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key];
const version =
entry && typeof entry.version === "string" ? entry.version : undefined;
if (entry && typeof entry.installPath === "string") {
result.installPaths.push(entry.installPath);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Do not cap plugin install paths with emitted plugin names.

keys.slice(0, NAMES_LIMIT) also limits installPaths, so bundled skills from plugins after the first 64 are excluded from skills_count and skill_source_counts.plugin even though plugins_count remains exact.

Proposed separation of exact scan inputs from capped output
-    for (const key of keys.slice(0, NAMES_LIMIT)) {
+    for (const key of keys) {
       const at = key.lastIndexOf("@");
       const name = at >= 0 ? key.slice(0, at) : key;
       const marketplace = at >= 0 ? key.slice(at + 1) : "";
       const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key];
       const version =
         entry && typeof entry.version === "string" ? entry.version : undefined;
       if (entry && typeof entry.installPath === "string") {
         result.installPaths.push(entry.installPath);
       }
+      if (result.plugins.length >= NAMES_LIMIT) continue;
 
       const rec = {};
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for (const key of keys.slice(0, NAMES_LIMIT)) {
const at = key.lastIndexOf("@");
const name = at >= 0 ? key.slice(0, at) : key;
const marketplace = at >= 0 ? key.slice(at + 1) : "";
const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key];
const version =
entry && typeof entry.version === "string" ? entry.version : undefined;
if (entry && typeof entry.installPath === "string") {
result.installPaths.push(entry.installPath);
}
for (const key of keys) {
const at = key.lastIndexOf("@");
const name = at >= 0 ? key.slice(0, at) : key;
const marketplace = at >= 0 ? key.slice(at + 1) : "";
const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key];
const version =
entry && typeof entry.version === "string" ? entry.version : undefined;
if (entry && typeof entry.installPath === "string") {
result.installPaths.push(entry.installPath);
}
if (result.plugins.length >= NAMES_LIMIT) continue;
const rec = {};
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/harness.js` around lines 590 - 599, The plugin scan in
harness.js is using the same capped key list for both emitted names and install
path collection, which causes later plugins to be missed from skills_count and
skill_source_counts.plugin. Separate the exact iteration used to gather
installPaths from the NAMES_LIMIT-capped loop used to emit plugin names, and
keep the install path aggregation in the logic around the keys/entry processing
so all plugins contribute to source counts while only names remain capped.

@john-skillbench
john-skillbench requested a review from minarc June 26, 2026 15:39
@john-skillbench
john-skillbench merged commit c105bee into main Jun 26, 2026
1 check passed
@TheStarkor
TheStarkor deleted the SBEE-166-phase-1-implement-harness-metadata-collection-in branch September 21, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants