Repository navigation
chore: pin docs-check model and add the measured evidence discipline - #29194
Merged
Merged
Conversation
Two changes, both aimed at running doc-check on a model we choose and measure rather than the deployment default. Workflow: - model-config-id comes from the DOC_CHECK_MODEL_CONFIG_ID repository variable. An unset variable keeps today's behavior, so clearing it is the rollback path and needs no code change. - coder-organization is pinned to coder. The bot account belongs to many organizations on this deployment and the action's fallback choice is not deterministic. A model configuration ID only resolves inside its own organization, so this pin is required by the input above. Skill: - Adds the review order that measured best on a 48 PR benchmark of this job: enumerate what the user experiences, check the conceptual guide and the enumerating page separately, find the exact stale sentence, treat a PR that ships its own docs as resolved, and write the evidence before the verdict.
The skill states the discipline. It does not need to justify it or point at an internal repository.
The inline comments on the action inputs already say it.
nickvigilante
approved these changes
Sep 10, 2026
nickvigilante
left a comment
Collaborator
There was a problem hiding this comment.
LGTM
🚀
Have you tested this on an existing PR, or is merging this PR and then monitoring doc-check the test?
Member
Author
Gotta test it post-merge sadly. Will let you know shortly 🫡 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This lets us choose which model reviews a PR for documentation, instead of always using the deployment default. Set the repository variable
DOC_CHECK_MODEL_CONFIG_IDto a model configuration ID to pick one, and clear it to go back. We want this so doc-check can run on a local LLM at no marginal cost. It also pins the chat organization, because the bot account belongs to many organizations and the model ID only works inside one of them.The skill file gains the review order that scored best when we benchmarked this job against 48 real merged PRs: list what the user experiences, check the guide and the list page separately, find the exact stale sentence, treat a PR that ships its own docs as done, and write the evidence before the verdict.
The local LLM we tested still scores lower than a frontier model on our benchmark, 89.6 percent against 96.6 percent but is deemed effective for this use case and we'll continue to iterate on models/instructions