Repository navigation
Migrate skill evaluation scenarios from markdown to JSON format - #95
NoumaanAhamed wants to merge 1 commit into
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Hey Team, Can you have a look at this and couple of other PRs when you get time? Thanks! cc: @mikeyarce @jonathanbossenger ( Saw that you've worked on previous PRs, hence tagged you :) ) |
|
Nothing inherently wrong here, but considering that we don't actually have anything in-repo to execute these evals yet, it's still speculative. |
Resolves #94
Summary
shared/scripts/scaffold-skill.mjsnow generateseval/scenarios/<skill-name>.json(matching the realname/skills/query/expected_behavior/success_criteriaschema) instead of a markdown file that nothing else in the repo useddocs/authoring-guide.mdandCONTRIBUTING.mdupdated to describe the JSON schema instead of "markdown files" — now consistent witheval/scenarios/README.md, which was already correctTest plan
node eval/harness/run.mjspassesnode shared/scripts/scaffold-skill.mjs <name> "<desc>"generates a valideval/scenarios/<name>.jsonmatching the schema (verified locally, then removed as a throwaway test)