DOC: Explicitly prohibit bots/agents to post contents - #31026

Of the two examples given, the second apologised and clarified that they had intended those posts for their fork and the spamming was accidental. They took action to correct the error, so I am unclear that a specific policy about automation would have improved that situation.

More generally, how do you practically distinguish between someone running an AI bot and someone repeatedly copy/pasting AI output as described in the first paragraph? If the comments are coming too fast to realistically be from a human, then I think that comes under GitHub’s existing spamming rules.

It would not have prevented the situation in the second case, but it gives me reference point to explain their wrong-doing without having to write much manual text. I also do not see a significant difference between accidental and intentional spamming. Both cases result in extra work on our side. If you run agents it's your responsibility to control them (e.g. possibly do test runs). They have spammed several repositories over multiple days.

Distinguishing: First of all, I want to be very explicit on unacceptable behavior. How to dect/distinguish is the second step. Some cases will be obvious, others won't. I've intentionally written "we reserve the right" and "we may". Thus, there is no atuomatism and we can do a case-by-case decision. In unclear cases we can start with "we have the impresssion that ..." / "it seems that ..." - "please make sure to comply with our AI policy".

Ping. Bots are becoming a real issue, spamming with low-quality contributions, e.g. #31095, #31096, #31098, #31099, #31100.

I really want something like this explicitly written out. It doesn't solve the issue, but hopefully if many projects state this, at least some people realize it's not an acceptable way of working (I assume good intentions and that people want to help and feel now they can through commanding a bot).

Also. our minimal defense opportunity is to make rejecting such PRs as little effort as possilbe. I could see a tag like "AI spam" or "AI policy violation" or "low quality PR" (name t.b.d.) that we attach and immediately close without any comment.

Tagging "release critical" to give this a sense of priority - even though it's not release critical as such.

@rcomer pointed this out on gitter but Scikit learn flags this in a way I really like scikit-learn/scikit-learn#31679

Also we might want to do something like have an "agents.md" file that says please don't open a PR or the like. Something like https://github.com/scikit-learn/scikit-learn/blob/main/AGENTS.md

• Do we know from ScikitLearn whether agents.md works?

• I haven't really found the flagging AI tools like Copilot Coding Agent don't know about / don't respect our Automated Contributions Policy scikit-learn/scikit-learn#31679 do you mean the autoclose

• Idea: Is there / can we write a bot, that checks whether an author is doing mass PRs?

  If somebody opens N (N~>5-10) on a day in multiple repos, it's highly likely that it's automated tooling, and suitable human crosscheck has probably not happend. I would want to auto-close such a PR (with a friendly message in case that was still a human).
  Such a bot could run e.g. for new contributors and on request (like the backport bot).

  Single low quality PRs are a burden, but with more automation and things like openclaw, I fear, we'll be spammed much more in the future. Like today with 8 PRs from the same account. It's a first line of defense if that could get auto-closed and no maintainer has to look at it at all.

• @story645 are you aware of any joint activity in the scientific / numfocus / python world? We all have the same issues and finding common practices and tooling may be worthwhile.

Do we know from ScikitLearn whether agents.md works?

No, I can ask though

do you mean the autoclose

yes

are you aware of any joint activity in the scientific / numfocus / python world? We all have the same issues and finding common practices and tooling may be worthwhile.

No but I think GitHub is considering rolling out tooling b/c maintainers from just about all the projects are complaining to them about slop. @melissawm might know more on the scientific/NumFocus side of things.

I thing I have done several times now is identify a change that I need to make to a bunch of repos (e.g. adding SHA pinning to yamls or bulk updates of versioneer for new Python versions) which can be automated via a shell script, grep / sed, and gh to open many nearly identical PRs across a bunch of repos. I worry that is allowed underhhhh the spirit of this rule, but not the letter.

That said, I am 👍 on banning users who open PRs without a human in the loop. We should not be asking maintainers to review code the person opening the PR has not read yet.

I worry that is allowed under the spirit of this rule, but not the letter.

I suspect that unless you know the repo very well, you'll have checked that your PR does only do the intended changes. Also, you'll only do this for repos you've been involved in and/or you know the maintainer is fine with it.

But better wording is always welcome, e.g. we could add "is strictly forbidden ..., unless a core developer has agreed to this in advance."

We should not be asking maintainers to review code the person opening the PR has not read yet.

Just as a comment: My main point is this: If we want AI-generated solutions to our issues, it's more efficient if we (as core developers) control the AI directly. So any new contributors can only add value to the project if they bring something that AI cannot.

Pandas policy states they may ban for “irresponsible use” of AI, which is broader than proposed here
https://pandas.pydata.org/docs/dev/development/contributing.html#automated-contributions-policy

I have seen a couple of policies floating around. Will try to put them all here (or at least the ones I remember):

• Scientific Python https://blog.scientific-python.org/scientific-python/community-considerations-around-ai/
• Zulip https://github.com/zulip/zulip/blob/main/CONTRIBUTING.md#ai-use-policy-and-guidelines
• Current discussion on NumPy https://mail.python.org/archives/list/numpy-discussion@python.org/thread/LAR7P3KQWHWAIKYSHTS2MY7X4HUBVA3L/
• PyOpenSci (in the context of peer-reviewed packages) https://www.pyopensci.org/blog/generative-ai-peer-review-policy.html

There are many more links under the links above. The point is that, at a minimum, I think requiring that folks declare usage such as suggested in #30848 is necessary. I don't believe all folks will do it sincerely, but it is a bit of positive friction to prevent it.

I think the Zulip policy is currently the one I like best, personally. It is pretty clear about what is acceptable and what is not. I also agree with @timhoffm that the goal here is to have clear documentation to point people to when a PR is closed or we ask whether a contribution is AI-generated.

I think this is a hard problem and I'm not sure myself what the solution is, but I think whatever the consensus is, it needs to be documented clearly (keeping in mind we'll need to update the policy regularly since the field is changing so quickly!).

Do we know from ScikitLearn whether agents.md works?

So roughly paraphrasing @betatim here, they feel like they've seen slop go down but can't pin point which/how much of each thing they did (agents, aggressive closing, docs) contributed to the reduction in slop.

I think AGENTS.md in general works. Instructions in it are followed by AI tools. However, the bit about "add a sentence saying AI was used to all your output" doesn't seem to work. At least when I use AI tools I basically never see this. I assume this is because LLMs decide that the instruction is "not relevant"?

Discussed on today's dev call and merged urgently in response to the #31132 activities.

On a general note: This is not anti-AI. We have to take these measures to protect the current contribution model.

The responsibility for suitability and quality of accepted code lies with the maintainers. Review is a manual process bound by the scare resource of maintainer time. We cannot handle a volume of auto-generated PRs. Additionally, we require human oversight to improve the quality standard of PRs reaching us.

As with everything related to AI things will continue to change. But for now this is the only way to keep the project running.