Repository navigation
feat(harness): collect full Level-1 harness metadata (SBEE-166) - #80
john-skillbench merged 1 commit into
Conversation
Phase 1 of the harness detection epic (SBEE-163): implement the locked SBEE-164 metadata contract in the Claude collector. detectHarness now emits the contract's flat data.harness field set instead of the earlier nested shape, adding detection for: - instruction files: claude_md_count, size bucket, @import count, has_claude_local_md, has_user_claude_md - skills: skills_present + per-source counts (project/user/plugin) - subagents (.claude/agents/*.md): present/count/hashed names/used - slash commands (.claude/commands/**/*.md) - MCP servers (.mcp.json / ~/.claude.json) — names only, never command/env - plugins + marketplaces (~/.claude/plugins manifests) - runtime: model, session_source, plugin_version Metadata only — no raw file contents, hook command strings, MCP env, or file paths. tier2 names are HMAC-hashed (public-marketplace plugin names kept raw); tier1 secrets dropped fail-closed; all tallied in `redactions`. Detection stays filesystem-only, depth/dir-bounded, and never throws out of SessionStart. session_start.js now threads model/source into detection. Tests rewritten for the flat shape (20 passing); a contract-coverage check confirms the emitted keys match spec/harness-metadata-contract.v1.json exactly (hook_commands intentionally dropped per the contract). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WalkthroughThe PR changes harness detection to a flat SBEE-164 metadata contract, adds bounded filesystem and JSON scanning with fail-closed redaction, updates hook/MCP/plugin collection, and passes session model/source into harness detection. ChangesHarness metadata contract update
Sequence Diagram(s)sequenceDiagram
participant SessionStart as "session_start.js"
participant DetectHarness as "detectHarness"
participant HashNames as "hashNames"
participant DetectPlugins as "detectPlugins"
SessionStart->>DetectHarness: model, sessionSource
DetectHarness->>HashNames: hash collected names
DetectHarness->>DetectPlugins: plugin and marketplace metadata
DetectHarness-->>SessionStart: flat harness metadata
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 OpenGrep (1.23.0)OpenGrep fatal error (exit code 2): �[32m✔�[39m �[1mOpengrep OSS�[0m �[1m Loading rules from local config...�[0m Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skillmeter/scripts/harness.js`:
- Around line 165-167: Both safeReadJson() and countImports() currently read
entire files synchronously before any limit is applied, so oversized
settings.json, .mcp.json, .claude.json, or CLAUDE.md files can still stall or
exhaust memory. Update these helpers to use a bounded read helper that caps
bytes read before parsing/counting, and apply it wherever fs.readFileSync is
used in safeReadJson() and countImports(). Ensure the same bounded approach is
used for the related read paths referenced by the same logic.
- Around line 590-599: The plugin scan in harness.js is using the same capped
key list for both emitted names and install path collection, which causes later
plugins to be missed from skills_count and skill_source_counts.plugin. Separate
the exact iteration used to gather installPaths from the NAMES_LIMIT-capped loop
used to emit plugin names, and keep the install path aggregation in the logic
around the keys/entry processing so all plugins contribute to source counts
while only names remain capped.
- Around line 302-310: The hook counting logic in harness.js is skipping
non-allowlisted events too early, so `hooks_count` and `hooks_source_counts`
miss configured hooks even though `hooks_enabled` stays correct. Update the loop
around `hooks` processing to always count entries for every event, and only gate
the `events.push(event)` behavior behind `KNOWN_HOOK_EVENTS`; keep using the
existing `hooks` iteration and `group.hooks` inner counting logic so unknown
events still contribute to totals.
- Around line 344-347: Normalize the runtime metadata fields in the harness
output so they always serialize as strings. In the harness logic that builds the
flat metadata object, update the handling of model and source-related inputs
(along with the existing agent_type/session_source/plugin_version fields) to
coerce any truthy non-string values into string form before emission, so the
contract from session_start.js is preserved. Use the existing metadata assembly
in the harness script to locate the fix and keep the output schema string-only.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8388233b-28f7-4423-a6bc-5c0268b2c215
📒 Files selected for processing (3)
skillmeter/scripts/harness.jsskillmeter/scripts/session_start.jsskillmeter/test/harness.test.js
| function safeReadJson(p) { | ||
| try { | ||
| return JSON.parse(fs.readFileSync(p, "utf8")); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Cap file reads before parsing/counting.
safeReadJson() and countImports() synchronously read the full file. A huge settings.json, .mcp.json, .claude.json, or CLAUDE.md can still stall/OOM the SessionStart hook despite the bounded directory walks.
Proposed bounded read helper
+const MAX_HARNESS_FILE_BYTES = 256 * 1024;
+
+function safeReadText(p) {
+ try {
+ const stat = fs.statSync(p);
+ if (!stat.isFile() || stat.size > MAX_HARNESS_FILE_BYTES) return "";
+ return fs.readFileSync(p, "utf8");
+ } catch {
+ return "";
+ }
+}
+
function safeReadJson(p) {
try {
- return JSON.parse(fs.readFileSync(p, "utf8"));
+ const text = safeReadText(p);
+ return text ? JSON.parse(text) : null;
} catch {
return null;
}
} function countImports(claudeMdPath) {
try {
- const text = fs.readFileSync(claudeMdPath, "utf8");
+ const text = safeReadText(claudeMdPath);
const matches = text.match(/(?:^|\s)@[^\s@]+/g);Also applies to: 282-285
🧰 Tools
🪛 ast-grep (0.44.0)
[warning] 166-166: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(p, "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/harness.js` around lines 165 - 167, Both safeReadJson()
and countImports() currently read entire files synchronously before any limit is
applied, so oversized settings.json, .mcp.json, .claude.json, or CLAUDE.md files
can still stall or exhaust memory. Update these helpers to use a bounded read
helper that caps bytes read before parsing/counting, and apply it wherever
fs.readFileSync is used in safeReadJson() and countImports(). Ensure the same
bounded approach is used for the related read paths referenced by the same
logic.
| for (const [event, groups] of Object.entries(hooks)) { | ||
| if (!KNOWN_HOOK_EVENTS.has(event)) continue; | ||
| events.push(event); | ||
| if (Array.isArray(groups)) { | ||
| for (const group of groups) { | ||
| // Each matcher group may hold an inner `hooks` array of commands; count | ||
| // those when present, otherwise count the group itself as one entry. | ||
| const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0; | ||
| entries += inner > 0 ? inner : 1; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Count hook entries even when event names are not emitted.
Line 303 drops unknown/non-allowlisted hook events before counting their entries. That preserves hooks_enabled, but makes hooks_count and hooks_source_counts falsely report fewer configured hooks.
Proposed fix
for (const [event, groups] of Object.entries(hooks)) {
- if (!KNOWN_HOOK_EVENTS.has(event)) continue;
- events.push(event);
+ if (KNOWN_HOOK_EVENTS.has(event)) events.push(event);
if (Array.isArray(groups)) {
for (const group of groups) {
// Each matcher group may hold an inner `hooks` array of commands; count
// those when present, otherwise count the group itself as one entry.
- const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0;
- entries += inner > 0 ? inner : 1;
+ entries += group && Array.isArray(group.hooks) ? group.hooks.length : 1;
}
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for (const [event, groups] of Object.entries(hooks)) { | |
| if (!KNOWN_HOOK_EVENTS.has(event)) continue; | |
| events.push(event); | |
| if (Array.isArray(groups)) { | |
| for (const group of groups) { | |
| // Each matcher group may hold an inner `hooks` array of commands; count | |
| // those when present, otherwise count the group itself as one entry. | |
| const inner = group && Array.isArray(group.hooks) ? group.hooks.length : 0; | |
| entries += inner > 0 ? inner : 1; | |
| for (const [event, groups] of Object.entries(hooks)) { | |
| if (KNOWN_HOOK_EVENTS.has(event)) events.push(event); | |
| if (Array.isArray(groups)) { | |
| for (const group of groups) { | |
| // Each matcher group may hold an inner `hooks` array of commands; count | |
| // those when present, otherwise count the group itself as one entry. | |
| entries += group && Array.isArray(group.hooks) ? group.hooks.length : 1; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/harness.js` around lines 302 - 310, The hook counting
logic in harness.js is skipping non-allowlisted events too early, so
`hooks_count` and `hooks_source_counts` miss configured hooks even though
`hooks_enabled` stays correct. Update the loop around `hooks` processing to
always count entries for every event, and only gate the `events.push(event)`
behavior behind `KNOWN_HOOK_EVENTS`; keep using the existing `hooks` iteration
and `group.hooks` inner counting logic so unknown events still contribute to
totals.
| agent_type: options.agentType || "claude-code", | ||
| instructions: { has_claude_md: false, has_claude_md_global: false, has_agents_md: false, has_agents_md_global: false, scopes: [] }, | ||
| skills: { count: 0, names: [], scopes: [] }, | ||
| hooks: { enabled: [], scopes: [] }, | ||
| // Level 2 — architecture-level signals are not detectable from the | ||
| // filesystem/transcript, so they are explicitly "unknown" rather than a | ||
| // misleading false. subagent_used is derived downstream from the | ||
| // SubagentStart/SubagentStop events the collector already emits. | ||
| orchestration: { | ||
| external_orchestration: "unknown", | ||
| multi_agent: "unknown", | ||
| }, | ||
| // Sanitization bookkeeping (SBEE-165, Phase 2): how many harness values | ||
| // were HMAC-hashed (Tier 2 names) or dropped (Tier 1 fail-closed) at this | ||
| // boundary, broken down by field type. Counts/types only — the original | ||
| // names/values never appear here. Complements the central sanitizeEventData | ||
| // summary, which only ever sees the already hashed/dropped result. | ||
| model: options.model || "", | ||
| session_source: options.sessionSource || "", | ||
| plugin_version: options.pluginVersion || "", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Normalize runtime fields to contract strings.
session_start.js now forwards input.model and input.source directly; truthy non-strings would be emitted as non-string harness fields and break the flat metadata contract.
Proposed normalization
+const stringOrEmpty = (value) => (typeof value === "string" ? value : "");
+
function detectHarness(cwd, options = {}) {
const harness = {
@@
- agent_type: options.agentType || "claude-code",
- model: options.model || "",
- session_source: options.sessionSource || "",
- plugin_version: options.pluginVersion || "",
+ agent_type: stringOrEmpty(options.agentType) || "claude-code",
+ model: stringOrEmpty(options.model),
+ session_source: stringOrEmpty(options.sessionSource),
+ plugin_version: stringOrEmpty(options.pluginVersion),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| agent_type: options.agentType || "claude-code", | |
| instructions: { has_claude_md: false, has_claude_md_global: false, has_agents_md: false, has_agents_md_global: false, scopes: [] }, | |
| skills: { count: 0, names: [], scopes: [] }, | |
| hooks: { enabled: [], scopes: [] }, | |
| // Level 2 — architecture-level signals are not detectable from the | |
| // filesystem/transcript, so they are explicitly "unknown" rather than a | |
| // misleading false. subagent_used is derived downstream from the | |
| // SubagentStart/SubagentStop events the collector already emits. | |
| orchestration: { | |
| external_orchestration: "unknown", | |
| multi_agent: "unknown", | |
| }, | |
| // Sanitization bookkeeping (SBEE-165, Phase 2): how many harness values | |
| // were HMAC-hashed (Tier 2 names) or dropped (Tier 1 fail-closed) at this | |
| // boundary, broken down by field type. Counts/types only — the original | |
| // names/values never appear here. Complements the central sanitizeEventData | |
| // summary, which only ever sees the already hashed/dropped result. | |
| model: options.model || "", | |
| session_source: options.sessionSource || "", | |
| plugin_version: options.pluginVersion || "", | |
| const stringOrEmpty = (value) => (typeof value === "string" ? value : ""); | |
| function detectHarness(cwd, options = {}) { | |
| const harness = { | |
| agent_type: stringOrEmpty(options.agentType) || "claude-code", | |
| model: stringOrEmpty(options.model), | |
| session_source: stringOrEmpty(options.sessionSource), | |
| plugin_version: stringOrEmpty(options.pluginVersion), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/harness.js` around lines 344 - 347, Normalize the runtime
metadata fields in the harness output so they always serialize as strings. In
the harness logic that builds the flat metadata object, update the handling of
model and source-related inputs (along with the existing
agent_type/session_source/plugin_version fields) to coerce any truthy non-string
values into string form before emission, so the contract from session_start.js
is preserved. Use the existing metadata assembly in the harness script to locate
the fix and keep the output schema string-only.
| for (const key of keys.slice(0, NAMES_LIMIT)) { | ||
| const at = key.lastIndexOf("@"); | ||
| const name = at >= 0 ? key.slice(0, at) : key; | ||
| const marketplace = at >= 0 ? key.slice(at + 1) : ""; | ||
| const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key]; | ||
| const version = | ||
| entry && typeof entry.version === "string" ? entry.version : undefined; | ||
| if (entry && typeof entry.installPath === "string") { | ||
| result.installPaths.push(entry.installPath); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not cap plugin install paths with emitted plugin names.
keys.slice(0, NAMES_LIMIT) also limits installPaths, so bundled skills from plugins after the first 64 are excluded from skills_count and skill_source_counts.plugin even though plugins_count remains exact.
Proposed separation of exact scan inputs from capped output
- for (const key of keys.slice(0, NAMES_LIMIT)) {
+ for (const key of keys) {
const at = key.lastIndexOf("@");
const name = at >= 0 ? key.slice(0, at) : key;
const marketplace = at >= 0 ? key.slice(at + 1) : "";
const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key];
const version =
entry && typeof entry.version === "string" ? entry.version : undefined;
if (entry && typeof entry.installPath === "string") {
result.installPaths.push(entry.installPath);
}
+ if (result.plugins.length >= NAMES_LIMIT) continue;
const rec = {};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for (const key of keys.slice(0, NAMES_LIMIT)) { | |
| const at = key.lastIndexOf("@"); | |
| const name = at >= 0 ? key.slice(0, at) : key; | |
| const marketplace = at >= 0 ? key.slice(at + 1) : ""; | |
| const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key]; | |
| const version = | |
| entry && typeof entry.version === "string" ? entry.version : undefined; | |
| if (entry && typeof entry.installPath === "string") { | |
| result.installPaths.push(entry.installPath); | |
| } | |
| for (const key of keys) { | |
| const at = key.lastIndexOf("@"); | |
| const name = at >= 0 ? key.slice(0, at) : key; | |
| const marketplace = at >= 0 ? key.slice(at + 1) : ""; | |
| const entry = Array.isArray(plugins[key]) ? plugins[key][0] : plugins[key]; | |
| const version = | |
| entry && typeof entry.version === "string" ? entry.version : undefined; | |
| if (entry && typeof entry.installPath === "string") { | |
| result.installPaths.push(entry.installPath); | |
| } | |
| if (result.plugins.length >= NAMES_LIMIT) continue; | |
| const rec = {}; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/harness.js` around lines 590 - 599, The plugin scan in
harness.js is using the same capped key list for both emitted names and install
path collection, which causes later plugins to be missed from skills_count and
skill_source_counts.plugin. Separate the exact iteration used to gather
installPaths from the NAMES_LIMIT-capped loop used to emit plugin names, and
keep the install path aggregation in the logic around the keys/entry processing
so all plugins contribute to source counts while only names remain capped.
Summary
Phase 1 (SBEE-166) of the Harness Detection & Collection epic (SBEE-163): implements the locked SBEE-164 metadata contract (
spec/harness-metadata-contract.v1.json) in the Claude collector.detectHarnessnow emits the contract's flatdata.harnessfield set (replacing the earlier nested shape from the SBEE-165 sanitization work), expanding coverage from presence-only to the full agreed Level-1 surface.What's newly collected
claude_md_count(nested),claude_md_size_bucket,claude_md_import_count,has_claude_local_md,has_user_claude_mdskills_present+skill_source_counts{project,user,plugin}.claude/agents/*.md): present / count / hashed names /subagent_used.claude/commands/**/*.md).mcp.json/~/.claude.json) — server names only, nevercommand/args/env~/.claude/pluginsmanifests)model,session_source,plugin_versionPrivacy
Metadata only — no raw file contents, hook command strings, MCP env, or file paths. tier2 identifiers (skill/subagent/command/MCP/private-plugin names) are HMAC-hashed; public-marketplace plugin names stay raw; tier1 secrets are dropped fail-closed; everything is tallied in
redactions. Detection is filesystem-only, depth/dir-bounded, and never throws out ofSessionStart.Tests
harness.test.jsrewritten for the flat shape — 20/20 passing (node --test).hook_commands, which the contract marksaction: drop.Notes
HARNESS_SCHEMA_VERSIONis now"1.0"(string) to match the contract.session_startalways supplies a salt.external_orchestration/multi_agent) remain"unknown"pending SBEE-168.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
New Features