AI Policy - #530

Yay, thanks!

1. Should we also state that maintainer reserves the right to close low-value PRs without further discussions and label them as "Invalid"?
2. Should we put this policy in https://github.com/astropy/.github for the whole org? And then individual repo can link there? Or am I misunderstanding what the .github is for?

@pllim it only makes sense to put in .github if it is a special file GitHub will do something with (like templates for instance)

Do we want to link to this new policy doc from

a. astropy.org website?
b. astropy core dev doc and readme?

Also the PR template?

Does anyone actually read the stuff in PR template? 😝

I do! ;)

Totally agree with @pllim and @Cadair's suggestions re: linking it, I was just thinking we get the policy done first before adding those links. But I'll add an issue to remind us to add the relevant link/ads.

Case study:

• [PERF] Replace np.column_stack with np.vstack().T matplotlib/matplotlib#31132

Additional policy examples and discussions: https://github.com/chaoss/wg-ai-alignment/tree/main/moderation#readme

I'm going to explicitly say here I don't think this goes far enough. I'm not completely sure I know how far I think we should go at this point but this isn't it.

Scikit-learn and llvms policys are a lot closer to where I think we need to be.

Full agree with Stuart on all of this! AI is not our friend when it's about library development. Do hack away with it for a website deployment, but here we talk about soaking up a lot of scarce maintainer resources and goodwill on onboarding new "people".

I'm going to explicitly say here I don't think this goes far enough. I'm not completely sure I know how far I think we should go at this point but this isn't it.

Scikit-learn and llvms policys are a lot closer to where I think we need to be.

I do really like sunpy and scikit-learn's policy paragraphs, because they are concise and part of the readme. Perhaps we could combine these points in this PR into a single paragraph in the README? and include something along the lines of "we will block you from contributing if you waste our maintainers' time"?

I'm watching this PR for ideas on how we'll need to write policies for spacetelescope repos, if we ever get hit with AI slop

I will say that the sunpy policy clearly isn't working so we are working on drafting more guidance: sunpy/sunpy#8505

Perhaps blocking users from collaborating with the organization should be a more readily-used option if they waste maintainer time with slop PRs and comments (as in, it's clear that they are not engaging with the maintainers themselves). In my (limited) experience with users who submit slop, they usually aren't sources of quality code contributions in the first place, even without using genAI, so indiscriminately blocking them doesn't really seem like a big loss for external collaboration

However, I'm not an active contributor to astropy and don't really have a feel for this community at large, so take my opinion with that in mind

Perhaps blocking users from collaborating with the organization should be a more readily-used option if they waste maintainer time with slop PRs and comments

The problem is that this effectively forces the maintainers in a kind of policing role (this is a point made by David Hogg in his arXiv white paper on astronomy more generally).

As a maintainer, if every PR from a new contributor has a substantial likelihood (>10%?) of being AI slop, I'll end up not looking at new-contributor PRs anymore (actually happening already...). I'd still prefer a "web of trust" system were we make people introduce themselves -- then, it just happens once, at first entry. Obviously, that poses an extra hurdle, but I think it can also be a way to more immediately make a connection, leading to more friendly review. (And, yes, there will still be bad actors, but surely fewer; lying is different from submitting something that an AI tells you is a good solution.)

The problem is that this effectively forces the maintainers in a kind of policing role

This is one of my two biggest concerns with a significantly stronger policy. In practice you cannot know if something was AI slop or just regular 'ol human slop, unless the author tells you. I know of plenty of cases where a junior human wrote something without any AI assistance, but then was told "this is obviously chatGPT-written" or whatever, and that leaves a permanent negative impression... So then how could we expect maintainers to spend even more time on these issues than is already being wasted?

@Cadair or others, I'm not sure what we do with "it needs to be stronger but not sure how". I have found scikit-learn's policy to be a good example except for one key issue: it says both "Please refrain from submitting issues or pull requests generated by fully-automated tools." and also "Review all code or documentation changes made by AI tools and make sure you understand all changes and can explain them on request, before submitting them under your name" - the former statement could be read as "no AI allowed" whereas the latter is "only use AI if you are willing to stand behind everything it wrote when you submit a PR". I think the latter is the intended meaning, and it was exactly that scikit-learn stance that was the original inspiration for this policy, but with some modifications for more straightforward wording. So can someone help me understand what more we could add to strengthen this without making it more confusing?

Thought that occurred to me after the above: would it help to add a "this is inspired by scikit-learn, see there for some similar but additional thoughts"?

(we should credit scikit-learn anyway given the inspiration, but I'm curious if folks think it also addresses some of the above cocnerns)

the former statement could be read as "no AI allowed"

To me the "fully-automated" in that statement is you ask one of the tools "solve this issue and open a PR", i.e. basically no human in the loop at all. The most extreme example of this being the recent mpl debacle.

I think we can have documentation with strict guidelines on what we deem to be acceptable and still only enforce them when we want to. This is the case for many of the technical guidelines we have as well.

For me the main thing this is lacking is some strong words on not pasting the output of a LLM into the comments / OP of a PR. This is the the most egregious thing imo.

I'm not sure if this was on the numpy thread or elsewhere, but someone nicely put it: for those parrotting low quality PRa we really don't need the contributor to play middle-person, the maintainer could prompt the AI themselves and do it better.
So, if the contributor doesn't understand the changes and be to the point in the comments (as opposed to the extremely long comments) then they just add noise which should be closed off as is, without further interaction.
And this is where sklearn is very strict, they in deed close PRs fast and ban repeated offenders.

(And they also run first time contributor sprints, low quality contributions from human first timers really should come through events and not just randomly lashed off on maintainers all the time expecting hand holding and personalised mentoring)

In practice you cannot know if something was AI slop or just regular 'ol human slop, unless the author tells you.

This is where the web of trust would seem the only option for a real reduction -- i.e., a need for a contributor to be on a whitelist before even submitting a PR. This adds a hurdle, but I think we can make this something that helps attract new contributors. But more importantly, it should help keep maintainers sane.

For me at least, a policy I can point to when closing PRs is actually relatively little help, though I think it is good to have one if only to explain why we have a whitelist.

So, more on topic of this PR, my suggestion would be to not spend too much more time on the exact policy -- as is, I'd tend to simply adopt that of scikit-learn -- and start thinking about how to implement the web of trust.

@Cadair

I think we can have documentation with strict guidelines on what we deem to be acceptable and still only enforce them when we want to. This is the case for many of the technical guidelines we have as well.

Ah I see your point here - totally agree this is similar to e.g. style guidance where sometimes exceptions are ok if they are for good reasons and its left to maintainers to make the judgement call.

For me the main thing this is lacking is some strong words on not pasting the output of a LLM into the comments / OP of a PR. This is the the most egregious thing imo.

I really like this idea! Hopefully @ceb8 can add something to this effect. And combined with the above point maybe this helps with some of the concerns @bsipocz raised? (it doesn't solve them all but at least starts us down a path?)

@mhvk

I think the idea of web-of-trust or similar is a good one, except I'm not as sure we can implement it usefully in a context like Astropy where there are quite different communities that have overlapping but different perspectives that are both needed (i.e. the engineering/developer/technical community and the science community). That is, even if we made a successful web-of-trust model for, say the astronomy research community, it might exclude technical experts who aren't known in the research community but are still valuable contributor/maintainers. It also has the potential to further exacerbate some of the existing issues where e.g. people in the Global South are underrepresented because they aren't plugged into the right communties. I'm sure there are solutions to this but they are not something that I think we can do right now. (which I guess is maybe your point of why we want you want to focus on that instead of this policy?)

I'm not sure if this was on the numpy thread or elsewhere, but someone nicely put it: for those parrotting low quality PRa we really don't need the contributor to play middle-person, the maintainer could prompt the AI themselves and do it better. So, if the contributor doesn't understand the changes and be to the point in the comments (as opposed to the extremely long comments) then they just add noise which should be closed off as is, without further interaction. And this is where sklearn is very strict, they in deed close PRs fast and ban repeated offenders.

(And they also run first time contributor sprints, low quality contributions from human first timers really should come through events and not just randomly lashed off on maintainers all the time expecting hand holding and personalised mentoring)

#530 (comment) might need even stronger language to align with what you're saying

@eteq - agreed that a web of trust needs some thought, which is why I suggested with starting with something like having a welcome message that asks new contributors to introduce themselves, to try to get going with a human connection asap.

To me, the details of the AI policy are less important, since I do not expect them to be read, i.e., the burden is still on the maintainer to be policing it. The only help is that at least there is something to point to.

Aside: the discussion in numpy is (somewhat) going in the direction of adopting the scipy policy (or perhaps a version of llvm); see https://scipy.github.io/devdocs/dev/conduct/ai_policy.html

Personally, I like the idea of a large part of the scientific python ecosystem having the same policy (and making things easier for ourselves by adopting instead of rewriting).

I think this policy is a great start — thanks @ceb8 and others for drafting and opening the PR!

From reading the discussion above, it seems like there is some tension in that some want a stricter policy and others want to keep it simpler (and more easily enforceable). I sympathize with both! But I do think having a policy like the one here could be a good starting point to get something in place, and then we can iterate on it (and other possible changes, like updates to the README or PR template) after some more discussion either in a bigger-picture issue or perhaps at the coordination meeting?

This comment is perhaps less directly relevant to the policy itself, but something that it got me thinking about. I'm happy to move this discussion elsewhere if we decide to continue the broader conversation in a different issue/PR/slack thread/whatever.

One thing I haven't seen mentioned here is the idea of asking for disclosure of LLM usage in contributions, and I'm curious what others think. This has come up in other contexts, like publishing and grant proposals, where it is often just a checkbox. I'm not sure how useful a checkbox would be here, though, since there is a large dynamic range in what "I used an LLM in this PR" means ("I asked Copilot to autocomplete a line" vs. "I generated all the code from a prompt").

One thing we could be after is to have a better record or description of how a PR came to be, and what the broader context of that PR is. As a thought experiment, I was considering what it would look like to make opening a PR closer to publishing a paper. In publishing, there are authors, acknowledgements, and citations. I think of these categories as Authors = "people who contributed and have autonomy to sign off on the content," Acknowledgements = "a place to recognize people, facilities, datasets, etc. that contributed at a more conceptual level", and Citations = "tools/software used, prior work built upon, relevant issues, etc."

Some PRs discuss subsets of these things, some mention all, but many don't. Capturing this information -- or at least requesting it via the PR template -- could be valuable in general (I've certainly worked on PRs where people contributed intellectually but don't appear in the discussion or commit authors), and especially so in the LLM era. For example, it could help us understand how LLMs were used in a given PR to have another metric to assess the level of human review and accountability behind the code changes. Someone could say: "I used a clanker to draft the initial implementation and then manually reviewed and revised it," which gives reviewers more useful context than a yes/no checkbox.

Thinking out loud here (it's confusing to know exactly what to do given that the landscape is changing and the way and extent to which developers are using LLMs is changing), but I guess a concrete suggestion would be to add some more structure to the PR template to ask for some of this information, and to ask for a description of how LLMs were used if they were used?

asking for disclosure of LLM usage

@neutrinoceros and I discussed this somewhere and I am sure similar other conversations had happened here and there as well. My worry is that the people who use "AI" the most are the ones too lazy to also disclose it. I see slops where the PR description itself was generated by AI to the point that I am not even sure if the human even bothered reading it first. While those are obvious, it also shows how human can easily ignore this request. As a result, we would end up penalizing the honest ones. So, I am hesitant on this, but curious what others think.

@pllim - I don't see a problem. Disclosing doesn't mean rejection, but non-disclosing would just make it easier to auto reject the slop.

I agree with Brigitta. The point of disclosure isn't to penalize honesty, but to reduce our need for suspicion. And yes, asking for it also makes it dishonest not to comply, which facilitates rejection in the cases we need it.

Quick update from the telecon: we (@astrofrog, @taldcroft and I) discussed adopting Scipy's policy and agreed it looked good as is, and that we should do same thing as they did, that is, copy the policy instead of linking to it, so we don't silently endorse any upstream change, and are free to amend it if need be.

The CoCo has decided to proceed with the current document rather than adopting that of Scipy, because we wanted to start with something simpler and then reassess once we see how it functions. To this end we will discuss the policy, and how we do or do not want to change it, including whether we want to adopt another project's policy, at the coordination meeting.

Thanks so much everyone who gave feedback!

Thanks, all!

Also, do you notice all the non-astropy repos linking to this? 👀

there's this in-development github action that purports to compare pull requests against a set of metrics (number of emojis, description length, number of code changes, etc.) and give reviewers a score of how much the PR smells like AI; that probably won't catch all of the AI slop PRs, but it might make it easier for / reduce the cognitive load on reviewers and maintainers when looking at PRs

https://github.com/peakoss/anti-slop