Skip to content

Update the newsfragment eval cases to the policy from #74589 - #75623

Draft
namanjain24-sudo wants to merge 2 commits into
apache:mainfrom
namanjain24-sudo:fix-newsfragment-eval-cases
Draft

namanjain24-sudo wants to merge 2 commits into
apache:mainfrom
namanjain24-sudo:fix-newsfragment-eval-cases

Conversation

@namanjain24-sudo

@namanjain24-sudo namanjain24-sudo commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

dev/skill-evals/cases/newsfragment.yaml has not changed since the harness landed in #69308, so it still
encodes the six-type newsfragment policy. #74589 then narrowed airflow-core to significant only:

In airflow-core/ use only significant newsfragments, and only for a change users must read before
upgrading: a behavior change, a removal or a new default, or migration steps. [...] Never add one for an
ordinary bug fix, feature, improvement or doc change

That change updated AGENTS.md, contributing-docs/18_contribution_workflow.rst and the PR-drafting
skill, but not these cases. Two of them describe additive features — a new endpoint plus a new
[connection_test] config section, and a new coordinator layer plus a config option — and assert
should_create === true. Neither is a behavior change, a removal, a new default or a migration, so an
agent following the current guidance answers false and the case fails it for being right. The harness
that exists to catch guidance drift had drifted from the guidance.

Flipping both would leave all six cases expecting "no", which an agent that always declines passes
perfectly, so the suite would stop detecting guidance that is too restrictive. Two cases that do warrant a
significant note keep the other direction covered, both taken from newsfragments already on main rather
than invented: the asset-triggered run default (#68517) and the async Postgres driver default (#68496).
That keeps the balance the file had before — six "no" to two "yes".

I may have the policy call wrong on the two flipped cases; if maintainers read "a new config section plus a
new endpoint" as something users must read before upgrading, say so and I will restore them and adjust the
reasoning instead.

The eval hash

Re-ran the harness and pushed the refreshed last-eval-hash.txt, so check-eval-hash passes and this no
longer needs a maintainer to run it.

The run is also direct evidence for the change. My local main ref happened to predate #74589, so the
baseline arm carried the old wording ("only for major or breaking changes") while the working-tree arm
carried the current one. The two flipped cases came out exactly as the policy change would predict:

arm=main (pre-#74589 guidance):  should_create=True,  type='feature'
arm=working tree (current):      should_create=False

Since this branch's AGENTS.md is identical to current main's, that working-tree arm is the real
current guidance — all eight cases pass against it. The two "failures" are the stale baseline answering
under the superseded policy, which is the signal the harness is for rather than a defect.

related: #62500

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5) following the guidelines

The cases have not changed since the harness landed in apache#69308, so they
still encode the six-type policy. apache#74589 narrowed airflow-core to
`significant` only -- "never for an ordinary bug fix, feature,
improvement or doc change" -- which leaves the two cases that expect a
feature newsfragment failing an agent for answering correctly.

Flipping both would leave every case expecting "no", which an agent that
always declines would pass, so two changes that do warrant a significant
note keep the other direction covered: the asset-triggered run default
from apache#68517 and the async Postgres driver default from apache#68496, both
taken from newsfragments already on main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant