Skip to content

feat: honor disable-model-invocation in skill frontmatter - #31147

Merged
ibetitsmike merged 270 commits into
mainfrom
mike/skills-disable-model-invocation
Oct 27, 2026
Merged

ibetitsmike merged 270 commits into
mainfrom
mike/skills-disable-model-invocation

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Extends #30882's model_invocable semantics to workspace skills and to stored skills created through the API.

Claude Code and Xum let skill authors set disable-model-invocation: true in SKILL.md frontmatter, so agents never pick the skill on their own and it runs only when a person invokes it. Coder Agents honored the key only when someone imported a SKILL.md file while adding a project skill (#30884).

What changes

  • Workspace skills: when the pinned SKILL.md sets the key to the YAML boolean true, chatd leaves the skill out of <available-skills> before merging sources, so a listed skill with the same name keeps its bare name, and read_skill reports it as not found. The skill stays in the / menu, and invoking it still loads its body. read_skill_file serves its supporting files under the bare name or workspace/<name> and refuses its SKILL.md (case-insensitive, after path.Clean). Skills without the key behave as on the base.
  • Invoked workspace skills: invoking a workspace skill now stores the pinned skill directory on the invocation part, and the prompt renders it as dir on <invoked-skill>, escaped as an attribute value. Before, only read_skill gave the model a skill's directory, and it refuses a hidden skill, so a hidden skill could not run the scripts it bundles. The part also stores the chat's workspace, and the prompt shows dir only while the chat stays bound to it, as it does for workspace file references. This changes feat: send skill invocations as structured message parts #30861's invocation code. Hook payloads drop the directory and its workspace like the body, and API responses strip both.
  • Stored skills: a create request whose content sets the key and that omits model_invocable stores model_invocable = false. An explicit value wins, and PATCH is unchanged. This covers personal, organization, and project skills, which share createSkill.
  • workspacesdk.SkillDisablesModelInvocation reads the key. The CreateSkillRequest.ModelInvocable doc comment, the generated API docs, and extending-agents.md (the SKILL.md format, a new "Turn off automatic use" section, and the notes on the skill index, the skill tools, and aliases) describe the new behavior.

Follow-ups

Testing

  • Go tests cover the parser, the turn's skill index and aliases, read_skill, read_skill_file, invocation of a hidden skill and the directory it carries (attribute escaping, no directory after the chat changes workspace, no directory in hook payloads), and the create default for personal and organization skills. Each new test fails when its code path is removed.
  • Remote dogfood UAT with a real model failed rounds 1 to 3. Rounds 1 and 2 found the hidden SKILL.md still readable through read_skill_file by symlinks and stale pins, plus a bare-name lookup bug and a docs gap that round 2 fixed. Round 3 found that the content check added against those bypasses refused a listed skill's file whose own frontmatter sets the key. After round 3 we narrowed the rule to the hidden skill's SKILL.md path. The key decides what agents are offered (the index and read_skill), not which files they can read: read_file and execute already open any workspace file, so a content check in read_skill_file protected nothing and kept producing edge cases.
  • Round 4 passed on c9ff25dc29f. Later commits added a behavior-neutral cleanup (cd617533c2e), one test case from the test audit (044c1e6e23a), and a comment fix (9ec3768b18b), which passed validation without another UAT round. Round 5 passed on 2b518c6c2b1, which adds the skill directory. A real model ran hidden skills' scripts from the dir on <invoked-skill>, including directories whose paths contain &, ', ", < and >, and API responses carried no directory. Merging main after feat: show project skill actions above project chat composers #30884 landed (2d7d3cb9674) did not change this PR's diff, so UAT was not rerun. 8f043b2a342, which shows the directory only while the chat stays bound to the skill's workspace, has Go tests but no UAT round.
  • The doc-check skill ran in author self-check mode on 2d7d3cb9674, and 27de7f5c6d7 fixes its findings: the skill index sentence said every skill is listed, the stored-skill sentence claimed any project editor import honors the key and presented the API default as fixed, the API path came before the UI path, and some wording broke the docs style rules. The fix is docs-only, so UAT was not rerun.
  • Polish and the test audit stayed inside the implementation diff. No new dependencies.
  • Codex review found no major issues on 9ec3768b18b, then raised one P2 on the same head when the PR was marked ready: an invoked hidden skill gets no skill directory. A probe confirmed it, and 2b518c6c2b1 fixes it. Its earlier finding on that head (honor CODER_AGENT_EXP_SKILL_META_FILE) was refuted, because the pinned skill snapshot always reads SKILL.md. Both threads are resolved, and Codex found no major issues on 2b518c6c2b1 or on the merge 2d7d3cb9674.
  • On 27de7f5c6d7, Codex raised two P2s. One was confirmed: an invoked skill's directory stayed in the prompt after the chat moved to another workspace, which a prompt test showed on that head, and 8f043b2a342 fixes it. The other was declined: on Windows, scripts\..\SKILL.md gets past the SKILL.md check in read_skill_file, but that check is not an access boundary (read_file opens any path), and Windows has more aliases (SKILL.md., SKILL.md::$DATA, SKILL~1.MD) that normalizing backslashes would not stop. On 8f043b2a342, Codex asked to bind the directory to the workspace build instead. That was declined: every workspace start is a new build with new agent IDs, so the directory would drop after each stop and start, and the pinned body and the live files can diverge without a rebuild anyway, as they do for read_skill. A later request on that head asked to limit a hidden skill's read_skill_file access to invocations in the current workspace. That was declined too: read_skill_file resolves names against the current pin for listed skills on main as well, and the key decides which skills agents are offered, not which files they can read. All four threads are resolved. a697dcd927f adds the format:"uuid" tag that TestSDKFieldsFormatted requires on the new field, and Codex found no major issues on a697dcd927f. ae70b2c533f is docs-only: it applies the doc-check suggestion to say that an invoked skill's directory is included only while the chat stays in that workspace. The author doc-check self-check found no blocking findings on it, and Codex found no major issues on ae70b2c533f.

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

…ts through refetches and handle deleted skills
…downloads, toggles, refetches and deleted skills
…ll details after deletes and refresh after download 404s
…t-skills/project-skills-dialog

# Conflicts:
#	site/src/pages/AgentsPage/components/SkillsTable.tsx
#	site/src/pages/AgentsPage/components/SkillsTableView.tsx
…kill refetches on save and respect the skill cap
…oject-skills/project-skills-details

# Conflicts:
#	site/src/api/queries/chatProjectSkills.ts
…ed skill refetches on save and the skill cap
Skills gain two independent settings. action marks a project skill as a
one-click action above project composers; a check constraint keeps it on
project skills, and personal and organization routes reject the field.
model_invocable controls whether agents see and load a skill on their
own: skills with it off are left out of <available-skills> and read_skill,
but structured skill invocations still load them.
read_skill refuses a workspace skill hidden from the model, and before
this branch it was the only tool that returned a skill's absolute
directory. A person can still invoke such a skill, but its body could not
run the scripts it bundles, because the model never learned where they
are.

Invoking a workspace skill now stores the pinned skill directory on the
invocation part, and the prompt renders it as a `dir` attribute on the
<invoked-skill> block, escaped as an attribute value. The field is
internal only: API responses already strip it, and hook payloads now
drop it next to the skill body.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 2b518c6c2b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Base automatically changed from chat-project-skills/actions-site to main October 23, 2026 12:53
@jaaydenh
jaaydenh requested a review from a team as a code owner October 23, 2026 12:53
…odel-invocation

# Conflicts:
#	coderd/apidoc/docs.go
#	coderd/apidoc/swagger.json
#	coderd/exp_chats.go
#	coderd/userskills.go
#	coderd/x/chatd/ARCHITECTURE.md
#	coderd/x/chatd/chathooks/hooks_internal_test.go
#	coderd/x/chatd/chathooks/trigger.go
#	coderd/x/chatd/chatprompt/chatprompt.go
#	coderd/x/chatd/chatprompt/chatprompt_test.go
#	coderd/x/chatd/chattool/skill.go
#	coderd/x/chatd/project_instructions.go
#	coderd/x/chatd/skill_invocation.go
#	coderd/x/chatd/skill_invocation_internal_test.go
#	codersdk/skills.go
#	docs/ai-coder/agents/extending-agents.md
#	docs/reference/api/chats.md
#	docs/reference/api/schemas.md
#	site/src/api/typesGenerated.ts
#	site/src/pages/AgentsPage/AgentCreatePage/AgentCreatePage.tsx
#	site/src/pages/AgentsPage/components/ChatMessageInput/ChatMessageInput.test.tsx
#	site/src/pages/AgentsPage/components/ChatMessageInput/ChatMessageInput.tsx
#	site/src/pages/AgentsPage/components/ProjectPage/ProjectSkillsDialog.tsx
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 2d7d3cb967

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…invocation

The skill index sentence said every discovered skill is listed, which a
skill that sets disable-model-invocation: true no longer is. The
stored-skill sentence claimed any project editor import honors the key
and presented the API default as fixed, but edit-mode import keeps the
current setting and an explicit model_invocable wins. A new "Turn off
automatic use" section states when the key applies to stored skills,
with the UI path first.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27de7f5c6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/skill_invocation.go Resolved
Comment thread coderd/x/chatd/chattool/skill.go Resolved
An invoked workspace skill stores its directory on the message, and every
later prompt replays it. After the chat was rebound to another workspace,
or unbound, the prompt still pointed the model at that path, which there
either does not exist or holds a different skill.

The invocation now also stores the chat's workspace, and the prompt shows
the directory only while the chat stays bound to it, as workspace file
references already do. The field is internal: API responses strip it and
hook payloads drop it.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f043b2a34

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatprompt/chatprompt.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f043b2a34

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chattool/skill.go Resolved
TestSDKFieldsFormatted requires every uuid.UUID field in codersdk to carry
format:"uuid", including internal fields that Swagger ignores.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: a697dcd927

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@coderagents coderagents Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc-check: 1 non-blocking suggestion after re-review.

@coderagents

coderagents Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

Re-reviewed at a697dcd9. No blocking findings; every new claim checks out against the code. One non-blocking suggestion is raised inline.

Design notes

  • The setting has four names across the API, UI, and frontmatter: disable-model-invocation (frontmatter), model_invocable (API), Agents can use it automatically (project editor label), and Show as action. This makes the new #frontmatter paragraph and the docs/ai-coder/agents/extending-agents.md "Turn off automatic use" section harder to keep accurate.

Related open PRs

Findings for agents
- id: coverage-1
  persona: coverage
  class: suggestion
  blocking: false
  file: docs/ai-coder/agents/extending-agents.md
  line: 140
  quote: "When you invoke a workspace skill, the agent also gets the skill's directory, so the agent can run the scripts that the skill includes."
  evidence:
    - coderd/x/chatd/chatprompt/chatprompt.go:1368
    - coderd/x/chatd/chatprompt/chatprompt.go:1355
  rule: null
  problem: The invoked-skill directory is only included while the chat stays bound to the owning workspace; the docs do not say this.
  fix: "When you invoke a workspace skill, the agent also gets the skill's directory, as long as the chat stays in that workspace, so the agent can run the scripts that the skill includes."
  verified: true

Automated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong.

@linear-code

linear-code Bot commented Oct 26, 2026

Copy link
Copy Markdown

CODAGT-669

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: ae70b2c533

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit 514f239 into main Oct 27, 2026
34 of 35 checks passed
@ibetitsmike
ibetitsmike deleted the mike/skills-disable-model-invocation branch October 27, 2026 00:06
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants