Skip to content

flag pull requests likely to be fully generated by an LLM - #20148

Open
zacharyburnett wants to merge 3 commits into
astropy:mainfrom
zacharyburnett:flag_generated_prs
Open

flag pull requests likely to be fully generated by an LLM#20148
zacharyburnett wants to merge 3 commits into
astropy:mainfrom
zacharyburnett:flag_generated_prs

Conversation

@zacharyburnett

@zacharyburnett zacharyburnett commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

adds https://github.com/peakoss/anti-slop workflow, perhaps it will be useful for reviewers? We are also interested in knowing if it will work correctly @pllim

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@zacharyburnett zacharyburnett mentioned this pull request Jul 29, 2026
Closed
zacharyburnett
zacharyburnett commented Jul 29, 2026
View reviewed changes
Comment thread .github/workflows/quality.yml
Comment on lines +19 to +20
close-pr: false
lock-pr: false

@zacharyburnett zacharyburnett Jul 29, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with auto-close and auto-lock set to False this will just make a comment on the PR

@neutrinoceros neutrinoceros Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good strat as a starting point. I hope we can move to auto-close later if this proves reliable and useful.

@pllim pllim added this to the v8.1.0 milestone Jul 29, 2026
@pllim

pllim commented Jul 29, 2026

Copy link
Copy Markdown
Member

While I applaud the enthusiasm, I am hesitant to introduce an experimental workflow (v0) from an author we know nothing of. Is there another repo we can try this on that is not a core library?

@zacharyburnett

zacharyburnett commented Jul 29, 2026

Copy link
Copy Markdown
Author

sure, but we might have to test it ourself with our own agent, since we're not currently getting a huge amount of slop PRs in. Maybe that's a good thing though, and we can wait until it's a bigger problem to revisit this

@pllim

pllim commented Jul 29, 2026

Copy link
Copy Markdown
Member

Surely more popular upstream packages are in more desperate condition. We could see which anti-bot bot they use and go from there.

@pllim pllim added Close? Tell stale bot that this issue/PR is stale Experimental labels Jul 29, 2026
neutrinoceros
neutrinoceros reviewed Jul 29, 2026
View reviewed changes
Comment thread .github/workflows/quality.yml Outdated
neutrinoceros
neutrinoceros reviewed Jul 29, 2026
View reviewed changes
Comment thread .github/workflows/quality.yml Outdated
pull-requests: write

on:
pull_request_target:

@neutrinoceros neutrinoceros Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as pointed by zizmor: this trigger is really a security can-of-worm. I'm surprised this is actually from a documented example.

@neutrinoceros neutrinoceros Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zacharyburnett how do we know that simply changing this to pull_request still works as intended ?

@zacharyburnett zacharyburnett Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as far as I understand it, pull_request_target just runs the workflow from the target branch, so changing it to pull_request should still trigger the workflow normally (from the PR branch). There aren't any checks in the anti-slop workflow itself for github.event_name or pull_request_target so it shouldn't matter what the trigger is

@neutrinoceros neutrinoceros Jul 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced yet. I opened an issue to touch base with the author: peakoss/anti-slop#20

@neutrinoceros neutrinoceros mentioned this pull request Jul 30, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neutrinoceros neutrinoceros neutrinoceros left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Close? Tell stale bot that this issue/PR is stale dev-automation Experimental no-changelog-entry-needed

Projects

None yet

Milestone

v8.1.0

Development

Successfully merging this pull request may close these issues.

3 participants

@zacharyburnett @pllim @neutrinoceros