Skip to content

feat(harness): custom-skill content collection (schema v2.1) - #98

Merged
john-skillbench merged 1 commit into
mainfrom
feat/harness-v2.1-skill-contents
Jul 10, 2026
Merged

john-skillbench merged 1 commit into
mainfrom
feat/harness-v2.1-skill-contents

Conversation

@john-skillbench

@john-skillbench john-skillbench commented Jul 10, 2026 •

Copy link
Copy Markdown
Contributor

What & why

Analysis-side request (SBEE-169): for custom (project/user) skills — which have no public catalog to join against — emit the SKILL.md body so the analysis pipeline can do semantic work on custom skills. Public/plugin-marketplace skills stay name-only (catalog join). Additive bump: harness_schema_version 2.0 → 2.1 (no existing fields changed; v2.0 consumers keep working).

New field

"skill_contents": [
  { "name": "deploy", "description": "Ship the app",
    "body": "1. run tests\n2. deploy", "bytes": 52, "truncated": false }
]

Guardrails (raw file content → privacy-sensitive)

  • Custom skills only (project/user). Plugin/public skills excluded by design.
  • Size cap — body capped at 4 KB (truncated flag + original bytes kept), ≤ 50 skills.
  • Secret scrub — bodies pass the central secret/PII sanitizer before upload; a skill whose name contains a secret is dropped entirely (body never read).

⚠️ Review note

This is tier2_raw_content (raw file content leaving the machine) — the step deferred from v2.0. Worth a security/compliance ack on contract v2.1, same as v2.0.

Tests

node --test suite: 59/59 pass (incl. new custom-body, plugin-exclusion, truncation cases). Companion PRs: Codex collector + skillbench-docs (contract v2.1 + eval).

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added custom skill content to harness metadata, including descriptions and safely truncated bodies.
    • Added safeguards to exclude secret-bearing skills and plugin marketplace skill bodies from emitted content.
    • Updated harness metadata to schema version 2.1.
  • Bug Fixes

    • Prevented oversized skill content from exceeding configured limits while preserving original size information.

Analysis-side request (SBEE-169): for CUSTOM (project/user) skills — which
have no public catalog to join against — emit the SKILL.md body so the
analysis pipeline can do semantic work on custom skills.

- new `skill_contents`: [{ name, description, body, bytes, truncated }] for
  project/user skills only; public/plugin skills stay name-only (catalog join)
- body size-capped (MAX_SKILL_BODY_BYTES=4096), <= MAX_SKILL_CONTENTS entries
- a skill dropped for a secret in its NAME never has its body read; bodies
  still pass the central secret/PII sanitizer before egress
- additive bump harness_schema_version 2.0 -> 2.1 (no existing fields changed)
- plugin 0.22.4 -> 0.23.0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 10, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The harness metadata contract advances to schema v2.1 and adds bounded skill_contents output for project and user skills. Skill bodies support frontmatter parsing, truncation, secret filtering, and safe omission; plugin skills remain name-only. Tests and the plugin version are updated.

Changes

Harness skill content emission

Layer / File(s) Summary
Schema contract and payload shape
skillmeter/.claude-plugin/plugin.json, skillmeter/scripts/harness.js, skillmeter/test/harness.test.js
The plugin and harness metadata versions advance to 0.23.0 and 2.1; skill_contents is initialized and schema expectations are updated.
Custom skill discovery and parsing
skillmeter/scripts/harness.js
Project and user skill paths are tracked, SKILL.md bodies are parsed safely, optional descriptions are extracted, and byte/count limits are applied while plugin skills remain name-only.
Filtered content emission and validation
skillmeter/scripts/harness.js, skillmeter/test/harness.test.js
Content is emitted only for retained custom skills in sorted, bounded order, with coverage for secret filtering, plugin exclusion, and body truncation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant detectHarness
  participant collectSkillNames
  participant Filesystem
  participant readSkillContent
  participant HarnessPayload
  detectHarness->>collectSkillNames: collect custom skill names and paths
  collectSkillNames->>Filesystem: locate project/user SKILL.md files
  detectHarness->>readSkillContent: read retained custom skill paths
  readSkillContent->>Filesystem: read and parse SKILL.md
  readSkillContent-->>detectHarness: bounded content object or null
  detectHarness->>HarnessPayload: emit filtered skill_contents
Loading

Possibly related PRs

Suggested reviewers: minarc, jhunlee-dev, TheStarkor

Poem

I nibbled new fields from the skill garden bright,
Parsed little bodies and trimmed them just right.
Secret names vanished beneath moonlit skies,
Plugin skills stayed names, with no text to disguise.
Schema two-one hops merrily onward tonight.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly reflects the main change: custom-skill content collection in the harness with schema v2.1.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/harness-v2.1-skill-contents

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
skillmeter/test/harness.test.js (1)

252-301: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a multi-byte truncation test to catch the byte-vs-length cap bug.

The custom-body and truncation tests are solid, but the truncation test (Lines 288-300) uses only ASCII ("x".repeat(6000)), so it can't detect the .length-vs-byte-cap mismatch in readSkillContent (see harness.js Lines 252-272). A body built from multi-byte characters (e.g. "é".repeat(3000)) exceeding MAX_SKILL_BODY_BYTES in real bytes but not in .length would currently slip through as truncated: false.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/test/harness.test.js` around lines 252 - 301, Issue: The
truncation test only uses ASCII and cannot detect character-length versus
byte-size handling. Extend the test for readSkillContent by using a multi-byte
body such as "é".repeat(3000), then assert the skill is marked truncated, its
body stays within the byte cap, and the original bytes value exceeds the cap.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skillmeter/scripts/harness.js`:
- Around line 252-272: The readSkillContent function truncates body by UTF-16
code units instead of UTF-8 byte length. Update the truncation logic to cap the
encoded body at MAX_SKILL_BODY_BYTES without splitting multibyte characters or
surrogate pairs, while preserving bytes as the original file size; add a test
covering multibyte content at the cap.

---

Nitpick comments:
In `@skillmeter/test/harness.test.js`:
- Around line 252-301: Issue: The truncation test only uses ASCII and cannot
detect character-length versus byte-size handling. Extend the test for
readSkillContent by using a multi-byte body such as "é".repeat(3000), then
assert the skill is marked truncated, its body stays within the byte cap, and
the original bytes value exceeds the cap.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: bef79cbf-7bc6-4cc1-9ca7-39af2958292b

📥 Commits

Reviewing files that changed from the base of the PR and between b7dded5 and db82053.

📒 Files selected for processing (3)
  • skillmeter/.claude-plugin/plugin.json
  • skillmeter/scripts/harness.js
  • skillmeter/test/harness.test.js

Comment on lines +252 to +272
function readSkillContent(name, mdPath) {
let text;
try {
text = fs.readFileSync(mdPath, "utf8");
} catch {
return null;
}
const bytes = Buffer.byteLength(text, "utf8");
let description = "";
let body = text;
const fm = text.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/);
if (fm) {
body = fm[2];
const d = fm[1].match(/^description:\s*(.+)$/m);
if (d) description = d[1].trim().replace(/^["']|["']$/g, "");
}
body = body.trim();
const truncated = body.length > MAX_SKILL_BODY_BYTES;
if (truncated) body = body.slice(0, MAX_SKILL_BODY_BYTES);
return { name, description, body, bytes, truncated };
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the relevant file and nearby tests before reading exact slices.
ast-grep outline skillmeter/scripts/harness.js --view expanded || true
ast-grep outline skillmeter/scripts --view expanded | sed -n '1,220p' || true

# Read the relevant implementation around the reported lines.
sed -n '230,300p' skillmeter/scripts/harness.js

# Find tests that cover readSkillContent / truncation / bytes.
rg -n "readSkillContent|MAX_SKILL_BODY_BYTES|truncat|bytes" skillmeter -g '!**/node_modules/**'

# Show the specific test file(s) if found.
for f in $(rg -l "MAX_SKILL_BODY_BYTES|truncat|readSkillContent" skillmeter -g '!**/node_modules/**'); do
  echo "===== $f ====="
  sed -n '1,260p' "$f"
done

Repository: SkillBench-AI/skillmeter-claude-code-marketplace

Length of output: 6032


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the test that documents truncation behavior.
sed -n '260,320p' skillmeter/test/harness.test.js

# Inspect the call site and surrounding comments for readSkillContent.
sed -n '520,555p' skillmeter/scripts/harness.js

# Look for any other references to the returned `bytes` field or body size expectations.
rg -n "\.bytes\b|bytes > 4096|truncated" skillmeter/scripts skillmeter/test -g '!**/node_modules/**'

Repository: SkillBench-AI/skillmeter-claude-code-marketplace

Length of output: 5027


🏁 Script executed:

node - <<'JS'
const MAX_SKILL_BODY_BYTES = 4096;

const samples = [
  { label: "ASCII", s: "x".repeat(4096) },
  { label: "emoji", s: "🙂".repeat(4096) },
  { label: "cjk", s: "汉".repeat(4096) },
  { label: "accented", s: "é".repeat(4096) },
];

for (const { label, s } of samples) {
  const truncated = s.length > MAX_SKILL_BODY_BYTES;
  const sliced = truncated ? s.slice(0, MAX_SKILL_BODY_BYTES) : s;
  console.log(label, {
    jsLength: s.length,
    utf8Bytes: Buffer.byteLength(s, "utf8"),
    truncated,
    slicedJsLength: sliced.length,
    slicedUtf8Bytes: Buffer.byteLength(sliced, "utf8"),
    lastCodePoint: Array.from(sliced).at(-1),
  });
}
JS

Repository: SkillBench-AI/skillmeter-claude-code-marketplace

Length of output: 738


Truncate body on UTF-8 bytes, not string.length. body.length/slice() count UTF-16 code units, so multi-byte skill bodies can still exceed MAX_SKILL_BODY_BYTES, and slice() can cut a surrogate pair. Keep bytes as the original file size; add a multibyte test for the cap.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 254-254: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(mdPath, "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(detect-non-literal-fs-filename)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skillmeter/scripts/harness.js` around lines 252 - 272, The readSkillContent
function truncates body by UTF-16 code units instead of UTF-8 byte length.
Update the truncation logic to cap the encoded body at MAX_SKILL_BODY_BYTES
without splitting multibyte characters or surrogate pairs, while preserving
bytes as the original file size; add a test covering multibyte content at the
cap.

@john-skillbench
john-skillbench merged commit 3ed2ad2 into main Jul 10, 2026
@TheStarkor
TheStarkor deleted the feat/harness-v2.1-skill-contents branch September 21, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant