Add AI policy label actions - #722

All commit authors signed the Contributor License Agreement.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ead995a) to head (ab2f578).

@@            Coverage Diff             @@
##              main      #722    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           18        20     +2     
  Lines         2182      2366   +184     
  Branches       100       117    +17     
==========================================
+ Hits          2182      2366   +184     

Flags with carried forward coverage won't be shown. Click here to find out more.

It's plausible we might want to have a slightly different message for issues instead of pull requests, i.e., less of a focus on "contributors" for issues.

e.g. zanieb/bedevere@ai-policy-labels...ai-policy-issue-messages

What is the intention on how the comment vs closed label meant to be used? When do you comment and when do you close? Is there rules about it on the devguide?

In my experience, it depends on the quality of the content. I use "close" when it seems likely it's in bad faith or is particularly low effort, and "comment" when it seems likely that they will engage. I'll pull some numbers and gather some feedback from other people on my team. We started with just the "close" action in June then split it into a "comment" action in August following some feedback from @AlexWaygood

it seems like a majority of the times we've used this label over in Ruff/ty we've actually just wanted the comment linking to the AI policy (we usually want to give the author time to respond and amend their PR/issue), and maintainers have immediately reopened the PR/issue after the PR/issue is auto-closed by the bot.

It might make sense to see what usage looks like before adding it to the devguide? but I don't have strong feelings about that.

Yeah, I think it probably depends on whether you think the PR is salvageable or not? I'm not sure I've ever actually used the label to just insta-close the PR, but I can see a use for it if you can see that the PR comes from a completely unmonitored agent, or if it's something that you don't even want a PR for. Usually, in practice, I tend to want to give the contributor a chance to rectify or fix their error, so I usually just add the label that adds a comment without closing the PR.

I pulled some data. In ruff and uv, we've used "close" 104 times and "comment" 9 times. Of the 9 commented on, 6 were subsequently closed by maintainers, 1 was closed by its author, 1 merged, and 2 remain open. Of the 104 closed, 3 were reopened and 1 merged.

As another guiding use-case, you can see we also use the "close" action for cases where someone opens a bunch of pull requests https://github.com/astral-sh/ruff/issues?q=state%3Aclosed+label%3Abot%3Aai-policy-close

We'll obviously also need to add the labels to the CPython repo:

See python/core-workflow#611 for that. Not entirely sure the process on that; we all have GH permission to add them, but I'm not sure if there needs to be more formal approval (SC?).

I'll draft a devguide change

Could we start with just the comment label, and we can close it ourselves if we find the pr author did not follow through?

Eg we default to 'commenting' first.
If we also think it is worth closing after the initial comment, then we close it ourselves.

Given the data from our experience I'm a bit wary to drop a "close" action, but am certainly happy to do whatever people prefer here. We could always add a "close" action in the future if data in the CPython repository / feedback from maintainers indicates it is useful. I'm not sure what it costs us to add it now though (a bit of complexity in the devguide?).

I think having an automated way to post a comment and close would be useful, it feels less personal.

(And when dealing with unsalvageable LLM content, it's already not personal on their side, so let's make it easier on us.)

We'll obviously also need to add the labels to the CPython repo:

See python/core-workflow#611 for that. Not entirely sure the process on that; we all have GH permission to add them, but I'm not sure if there needs to be more formal approval (SC?).

I think the usual process is: open issue, wait a bit that there's consensus or no objections, create the labels. A merge here is good consensus to go ahead and create.

Created python/devguide#1933

more formal approval (SC?).

I don't think we've ever needed that before, things like type-refactor were added based on consensus of the core team.

@Mariatta @zanieb I added some suggestions to this PR and the devguide PR.

I agree with Zanie that having "comment" and "close" makes sense.

Mariatta, I hope my suggestions help new contributors by thoughtful use of "comment". For clear cases, "close" signals the need for respect of maintainers'/triagers' time.

Note as I was looking at Carol's feedback I realized I didn't like the repetition in the issue comments and have edited them in 5069951