Repository navigation
feat(harness): custom-skill content collection (schema v2.1) - #98
Conversation
Analysis-side request (SBEE-169): for CUSTOM (project/user) skills — which
have no public catalog to join against — emit the SKILL.md body so the
analysis pipeline can do semantic work on custom skills.
- new `skill_contents`: [{ name, description, body, bytes, truncated }] for
project/user skills only; public/plugin skills stay name-only (catalog join)
- body size-capped (MAX_SKILL_BODY_BYTES=4096), <= MAX_SKILL_CONTENTS entries
- a skill dropped for a secret in its NAME never has its body read; bodies
still pass the central secret/PII sanitizer before egress
- additive bump harness_schema_version 2.0 -> 2.1 (no existing fields changed)
- plugin 0.22.4 -> 0.23.0
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WalkthroughThe harness metadata contract advances to schema v2.1 and adds bounded ChangesHarness skill content emission
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant detectHarness
participant collectSkillNames
participant Filesystem
participant readSkillContent
participant HarnessPayload
detectHarness->>collectSkillNames: collect custom skill names and paths
collectSkillNames->>Filesystem: locate project/user SKILL.md files
detectHarness->>readSkillContent: read retained custom skill paths
readSkillContent->>Filesystem: read and parse SKILL.md
readSkillContent-->>detectHarness: bounded content object or null
detectHarness->>HarnessPayload: emit filtered skill_contents
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
skillmeter/test/harness.test.js (1)
252-301: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a multi-byte truncation test to catch the byte-vs-length cap bug.
The custom-body and truncation tests are solid, but the truncation test (Lines 288-300) uses only ASCII (
"x".repeat(6000)), so it can't detect the.length-vs-byte-cap mismatch inreadSkillContent(see harness.js Lines 252-272). A body built from multi-byte characters (e.g."é".repeat(3000)) exceedingMAX_SKILL_BODY_BYTESin real bytes but not in.lengthwould currently slip through astruncated: false.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skillmeter/test/harness.test.js` around lines 252 - 301, Issue: The truncation test only uses ASCII and cannot detect character-length versus byte-size handling. Extend the test for readSkillContent by using a multi-byte body such as "é".repeat(3000), then assert the skill is marked truncated, its body stays within the byte cap, and the original bytes value exceeds the cap.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skillmeter/scripts/harness.js`:
- Around line 252-272: The readSkillContent function truncates body by UTF-16
code units instead of UTF-8 byte length. Update the truncation logic to cap the
encoded body at MAX_SKILL_BODY_BYTES without splitting multibyte characters or
surrogate pairs, while preserving bytes as the original file size; add a test
covering multibyte content at the cap.
---
Nitpick comments:
In `@skillmeter/test/harness.test.js`:
- Around line 252-301: Issue: The truncation test only uses ASCII and cannot
detect character-length versus byte-size handling. Extend the test for
readSkillContent by using a multi-byte body such as "é".repeat(3000), then
assert the skill is marked truncated, its body stays within the byte cap, and
the original bytes value exceeds the cap.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: bef79cbf-7bc6-4cc1-9ca7-39af2958292b
📒 Files selected for processing (3)
skillmeter/.claude-plugin/plugin.jsonskillmeter/scripts/harness.jsskillmeter/test/harness.test.js
| function readSkillContent(name, mdPath) { | ||
| let text; | ||
| try { | ||
| text = fs.readFileSync(mdPath, "utf8"); | ||
| } catch { | ||
| return null; | ||
| } | ||
| const bytes = Buffer.byteLength(text, "utf8"); | ||
| let description = ""; | ||
| let body = text; | ||
| const fm = text.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?([\s\S]*)$/); | ||
| if (fm) { | ||
| body = fm[2]; | ||
| const d = fm[1].match(/^description:\s*(.+)$/m); | ||
| if (d) description = d[1].trim().replace(/^["']|["']$/g, ""); | ||
| } | ||
| body = body.trim(); | ||
| const truncated = body.length > MAX_SKILL_BODY_BYTES; | ||
| if (truncated) body = body.slice(0, MAX_SKILL_BODY_BYTES); | ||
| return { name, description, body, bytes, truncated }; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant file and nearby tests before reading exact slices.
ast-grep outline skillmeter/scripts/harness.js --view expanded || true
ast-grep outline skillmeter/scripts --view expanded | sed -n '1,220p' || true
# Read the relevant implementation around the reported lines.
sed -n '230,300p' skillmeter/scripts/harness.js
# Find tests that cover readSkillContent / truncation / bytes.
rg -n "readSkillContent|MAX_SKILL_BODY_BYTES|truncat|bytes" skillmeter -g '!**/node_modules/**'
# Show the specific test file(s) if found.
for f in $(rg -l "MAX_SKILL_BODY_BYTES|truncat|readSkillContent" skillmeter -g '!**/node_modules/**'); do
echo "===== $f ====="
sed -n '1,260p' "$f"
doneRepository: SkillBench-AI/skillmeter-claude-code-marketplace
Length of output: 6032
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the test that documents truncation behavior.
sed -n '260,320p' skillmeter/test/harness.test.js
# Inspect the call site and surrounding comments for readSkillContent.
sed -n '520,555p' skillmeter/scripts/harness.js
# Look for any other references to the returned `bytes` field or body size expectations.
rg -n "\.bytes\b|bytes > 4096|truncated" skillmeter/scripts skillmeter/test -g '!**/node_modules/**'Repository: SkillBench-AI/skillmeter-claude-code-marketplace
Length of output: 5027
🏁 Script executed:
node - <<'JS'
const MAX_SKILL_BODY_BYTES = 4096;
const samples = [
{ label: "ASCII", s: "x".repeat(4096) },
{ label: "emoji", s: "🙂".repeat(4096) },
{ label: "cjk", s: "汉".repeat(4096) },
{ label: "accented", s: "é".repeat(4096) },
];
for (const { label, s } of samples) {
const truncated = s.length > MAX_SKILL_BODY_BYTES;
const sliced = truncated ? s.slice(0, MAX_SKILL_BODY_BYTES) : s;
console.log(label, {
jsLength: s.length,
utf8Bytes: Buffer.byteLength(s, "utf8"),
truncated,
slicedJsLength: sliced.length,
slicedUtf8Bytes: Buffer.byteLength(sliced, "utf8"),
lastCodePoint: Array.from(sliced).at(-1),
});
}
JSRepository: SkillBench-AI/skillmeter-claude-code-marketplace
Length of output: 738
Truncate body on UTF-8 bytes, not string.length. body.length/slice() count UTF-16 code units, so multi-byte skill bodies can still exceed MAX_SKILL_BODY_BYTES, and slice() can cut a surrogate pair. Keep bytes as the original file size; add a multibyte test for the cap.
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 254-254: Filesystem path is not a string literal; a request-/variable-derived path can enable path traversal. Validate and normalize the path before use.
Context: fs.readFileSync(mdPath, "utf8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(detect-non-literal-fs-filename)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skillmeter/scripts/harness.js` around lines 252 - 272, The readSkillContent
function truncates body by UTF-16 code units instead of UTF-8 byte length.
Update the truncation logic to cap the encoded body at MAX_SKILL_BODY_BYTES
without splitting multibyte characters or surrogate pairs, while preserving
bytes as the original file size; add a test covering multibyte content at the
cap.
What & why
Analysis-side request (SBEE-169): for custom (project/user) skills — which have no public catalog to join against — emit the SKILL.md body so the analysis pipeline can do semantic work on custom skills. Public/plugin-marketplace skills stay name-only (catalog join). Additive bump:
harness_schema_version2.0 → 2.1 (no existing fields changed; v2.0 consumers keep working).New field
Guardrails (raw file content → privacy-sensitive)
truncatedflag + originalbyteskept), ≤ 50 skills.This is
tier2_raw_content(raw file content leaving the machine) — the step deferred from v2.0. Worth a security/compliance ack on contract v2.1, same as v2.0.Tests
node --testsuite: 59/59 pass (incl. new custom-body, plugin-exclusion, truncation cases). Companion PRs: Codex collector + skillbench-docs (contract v2.1 + eval).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes