Skip to content

chore: pin docs-check model and add the measured evidence discipline - #29194

Merged
bpmct merged 3 commits into
mainfrom
bpmct/doc-check-solstice-model
Sep 10, 2026
Merged

bpmct merged 3 commits into
mainfrom
bpmct/doc-check-solstice-model

Conversation

@bpmct

@bpmct bpmct commented Sep 10, 2026 •

Copy link
Copy Markdown
Member

This lets us choose which model reviews a PR for documentation, instead of always using the deployment default. Set the repository variable DOC_CHECK_MODEL_CONFIG_ID to a model configuration ID to pick one, and clear it to go back. We want this so doc-check can run on a local LLM at no marginal cost. It also pins the chat organization, because the bot account belongs to many organizations and the model ID only works inside one of them.

The skill file gains the review order that scored best when we benchmarked this job against 48 real merged PRs: list what the user experiences, check the guide and the list page separately, find the exact stale sentence, treat a PR that ships its own docs as done, and write the evidence before the verdict.

The local LLM we tested still scores lower than a frontier model on our benchmark, 89.6 percent against 96.6 percent but is deemed effective for this use case and we'll continue to iterate on models/instructions

Two changes, both aimed at running doc-check on a model we choose and
measure rather than the deployment default.

Workflow:
- model-config-id comes from the DOC_CHECK_MODEL_CONFIG_ID repository
  variable. An unset variable keeps today's behavior, so clearing it is
  the rollback path and needs no code change.
- coder-organization is pinned to coder. The bot account belongs to many
  organizations on this deployment and the action's fallback choice is
  not deterministic. A model configuration ID only resolves inside its
  own organization, so this pin is required by the input above.

Skill:
- Adds the review order that measured best on a 48 PR benchmark of this
  job: enumerate what the user experiences, check the conceptual guide
  and the enumerating page separately, find the exact stale sentence,
  treat a PR that ships its own docs as resolved, and write the evidence
  before the verdict.
The skill states the discipline. It does not need to justify it or point
at an internal repository.
The inline comments on the action inputs already say it.
@bpmct
bpmct requested a review from nickvigilante September 10, 2026 20:55

@nickvigilante nickvigilante left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit: 🚀

Have you tested this on an existing PR, or is merging this PR and then monitoring doc-check the test?

@bpmct

bpmct commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Have you tested this on an existing PR, or is merging this PR and then monitoring doc-check the test?

Gotta test it post-merge sadly. Will let you know shortly 🫡

@bpmct bpmct changed the title chore(doc-check): pin the review model and add the measured evidence discipline chore: pin docs-check model and add the measured evidence discipline Sep 10, 2026
@bpmct
bpmct merged commit 0de1ab3 into main Sep 10, 2026
37 of 40 checks passed
@bpmct
bpmct deleted the bpmct/doc-check-solstice-model branch September 10, 2026 21:15
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 10, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants