Repository navigation
Update the newsfragment eval cases to the policy from #74589 - #75623
Draft
namanjain24-sudo wants to merge 2 commits into
Draft
namanjain24-sudo wants to merge 2 commits into
namanjain24-sudo wants to merge 2 commits into
Conversation
The cases have not changed since the harness landed in apache#69308, so they still encode the six-type policy. apache#74589 narrowed airflow-core to `significant` only -- "never for an ordinary bug fix, feature, improvement or doc change" -- which leaves the two cases that expect a feature newsfragment failing an agent for answering correctly. Flipping both would leave every case expecting "no", which an agent that always declines would pass, so two changes that do warrant a significant note keep the other direction covered: the asset-triggered run default from apache#68517 and the async Postgres driver default from apache#68496, both taken from newsfragments already on main.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dev/skill-evals/cases/newsfragment.yamlhas not changed since the harness landed in #69308, so it stillencodes the six-type newsfragment policy. #74589 then narrowed
airflow-coretosignificantonly:That change updated
AGENTS.md,contributing-docs/18_contribution_workflow.rstand the PR-draftingskill, but not these cases. Two of them describe additive features — a new endpoint plus a new
[connection_test]config section, and a new coordinator layer plus a config option — and assertshould_create === true. Neither is a behavior change, a removal, a new default or a migration, so anagent following the current guidance answers
falseand the case fails it for being right. The harnessthat exists to catch guidance drift had drifted from the guidance.
Flipping both would leave all six cases expecting "no", which an agent that always declines passes
perfectly, so the suite would stop detecting guidance that is too restrictive. Two cases that do warrant a
significantnote keep the other direction covered, both taken from newsfragments already onmainratherthan invented: the asset-triggered run default (#68517) and the async Postgres driver default (#68496).
That keeps the balance the file had before — six "no" to two "yes".
I may have the policy call wrong on the two flipped cases; if maintainers read "a new config section plus a
new endpoint" as something users must read before upgrading, say so and I will restore them and adjust the
reasoning instead.
The eval hash
Re-ran the harness and pushed the refreshed
last-eval-hash.txt, socheck-eval-hashpasses and this nolonger needs a maintainer to run it.
The run is also direct evidence for the change. My local
mainref happened to predate #74589, so thebaseline arm carried the old wording ("only for major or breaking changes") while the working-tree arm
carried the current one. The two flipped cases came out exactly as the policy change would predict:
Since this branch's
AGENTS.mdis identical to currentmain's, that working-tree arm is the realcurrent guidance — all eight cases pass against it. The two "failures" are the stale baseline answering
under the superseded policy, which is the signal the harness is for rather than a defect.
related: #62500
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines