Repository navigation
Add AI policy label actions #722
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zanieb
wants to merge
10
commits into
python:main
Choose a base branch
from
zanieb:ai-policy-labels
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
e79154d
Add AI policy label actions
zanieb 4f4c497
Tailor AI policy messages for issues
zanieb f379c2b
Apply Hugo's feedback on prose
zanieb 5069951
Remove repetition from AI policy messages for issues
zanieb 5772837
Give the AI policy reason before closing issues and PRs
zanieb ab2f578
Capitalize "AI"
zanieb 7c9885c
Adjust the section header
zanieb f1af168
Select AI policy actions by webhook event type
zanieb b1b40f6
Match issue and pull request webhook fields in policy tests
zanieb 4c08149
Normalize line endings on the AI policy labels
zanieb File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| """Respond to AI policy labels applied by triage and core teams.""" | ||
|
|
||
| import os | ||
|
|
||
| import gidgethub.routing | ||
|
|
||
| from . import util | ||
|
|
||
| router = gidgethub.routing.Router() | ||
|
|
||
| COMMENT_LABEL = "AI policy comment" | ||
| CLOSE_LABEL = "AI policy close" | ||
|
|
||
| _PR_MESSAGE_TEMPLATE = """\ | ||
| Thank you for your interest in contributing to Python. This is a community-driven \ | ||
| project, and we want it to be a welcoming place to learn and collaborate. | ||
|
|
||
| Contributors are responsible for reviewing and \ | ||
| understanding what they submit, checking its claims, and being able to explain \ | ||
| changes in their own words. This helps reviewers focus on the contribution and \ | ||
| leaves more time to support other contributors. | ||
|
|
||
| {action} | ||
|
|
||
| Please read our [guidelines for using AI tools]\ | ||
| (https://devguide.python.org/getting-started/ai-tools/index.html).""" | ||
|
|
||
| PR_COMMENT_MESSAGE = _PR_MESSAGE_TEMPLATE.format( | ||
| action=( | ||
| "This contribution appears to have been authored with AI assistance. " | ||
| "Please explain your contribution in your own words and describe how you " | ||
| "checked its correctness." | ||
| ) | ||
| ) | ||
|
|
||
| PR_CLOSE_MESSAGE = _PR_MESSAGE_TEMPLATE.format( | ||
| action=( | ||
| "We believe this contribution does not meet our guidelines for using AI " | ||
| "tools, so we're closing it. " | ||
| "If you think we've misunderstood your contribution, please let us know. " | ||
| "You're welcome to explain your approach or ask questions about the guidelines." | ||
| ) | ||
| ) | ||
|
|
||
| _ISSUE_MESSAGE_TEMPLATE = """\ | ||
| Thank you for taking the time to open this issue. Python is a community-driven \ | ||
| project, and we want it to be a welcoming place to learn and collaborate. | ||
|
|
||
| {action} | ||
|
willingc marked this conversation as resolved.
|
||
|
|
||
| Please read our [guidelines for using AI tools]\ | ||
| (https://devguide.python.org/getting-started/ai-tools/index.html).""" | ||
|
|
||
| ISSUE_COMMENT_MESSAGE = _ISSUE_MESSAGE_TEMPLATE.format( | ||
| action=( | ||
| "This issue appears to have been written with AI assistance. " | ||
| "Please explain your issue in your own words and describe how you checked " | ||
| "the information you provided. This helps us understand your issue and " | ||
| "leaves more time to address other issues." | ||
| ) | ||
| ) | ||
|
|
||
| ISSUE_CLOSE_MESSAGE = _ISSUE_MESSAGE_TEMPLATE.format( | ||
| action=( | ||
| "We expect issues to be explained in your own words. " | ||
| "We believe this issue does not meet our guidelines for using AI tools, " | ||
| "so we're closing it. " | ||
| "If you think we've misunderstood, you're welcome to clarify the issue " | ||
| "or ask questions about the guidelines." | ||
| ) | ||
| ) | ||
|
|
||
|
|
||
| @router.register("issues", action="labeled") | ||
|
zanieb marked this conversation as resolved.
|
||
| @router.register("pull_request", action="labeled") | ||
| async def label_added(event, gh, *args, **kwargs): | ||
| """Post a policy reminder and close the contribution when requested.""" | ||
| label = util.label_name(event.data) | ||
| if label not in {COMMENT_LABEL, CLOSE_LABEL}: | ||
| return | ||
| if event.data["sender"]["type"] == "Bot": | ||
| return | ||
|
|
||
| if "pull_request" in event.data: | ||
| target = event.data["pull_request"] | ||
| issue_url = target["issue_url"] | ||
| comment_message, close_message = PR_COMMENT_MESSAGE, PR_CLOSE_MESSAGE | ||
| close_data = {"state": "closed"} | ||
| else: | ||
| target = event.data["issue"] | ||
| issue_url = target["url"] | ||
| comment_message, close_message = ISSUE_COMMENT_MESSAGE, ISSUE_CLOSE_MESSAGE | ||
| close_data = {"state": "closed", "state_reason": "not_planned"} | ||
|
|
||
| if target["state"] != "open": | ||
| return | ||
|
|
||
| issue = await gh.getitem(issue_url) | ||
| if issue["state"] != "open" or label not in util.labels(issue): | ||
| return | ||
|
|
||
| marker = f"<!-- bedevere: {label} -->" | ||
| app_id = os.environ.get("GH_APP_ID") | ||
| async for comment in gh.getiter(issue["comments_url"]): | ||
| app = comment.get("performed_via_github_app") or {} | ||
| if str(app.get("id")) == app_id and marker in comment["body"]: | ||
| break | ||
| else: | ||
| message = close_message if label == CLOSE_LABEL else comment_message | ||
| await gh.post(issue["comments_url"], data={"body": f"{message}\n\n{marker}"}) | ||
|
|
||
| if label == CLOSE_LABEL: | ||
| # A maintainer may remove the label while the comment is being posted. | ||
| issue = await gh.getitem(issue_url) | ||
| if issue["state"] != "open" or label not in util.labels(issue): | ||
| return | ||
| await gh.patch(target["url"], data=close_data) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,211 @@ | ||
| from copy import deepcopy | ||
|
|
||
| import pytest | ||
| from gidgethub import sansio | ||
|
|
||
| from bedevere import __main__ as main | ||
| from bedevere import policy | ||
|
|
||
| APP_ID = 123 | ||
| ISSUE_URL = "https://api.github.com/repos/python/cpython/issues/42" | ||
| PR_URL = "https://api.github.com/repos/python/cpython/pulls/42" | ||
| COMMENTS_URL = f"{ISSUE_URL}/comments" | ||
|
|
||
|
|
||
| def make_event(kind, label, *, action="labeled", state="open", sender_type="User"): | ||
| target = { | ||
| "number": 42, | ||
| "state": state, | ||
| "labels": [{"name": label}], | ||
| "comments_url": COMMENTS_URL, | ||
| } | ||
| data = { | ||
| "action": action, | ||
| "label": {"name": label}, | ||
| "sender": {"login": "triager", "type": sender_type}, | ||
| } | ||
| if kind == "pull_request": | ||
| data["pull_request"] = {**target, "url": PR_URL, "issue_url": ISSUE_URL} | ||
| else: | ||
| data["issue"] = {**target, "url": ISSUE_URL} | ||
| return sansio.Event(data, event=kind, delivery_id="policy-label-delivery") | ||
|
|
||
|
|
||
| class FakeGH: | ||
| def __init__(self, label): | ||
| self.issue = { | ||
| "state": "open", | ||
| "labels": [{"name": label}], | ||
| "comments_url": COMMENTS_URL, | ||
| } | ||
| self.comments = [] | ||
| self.calls = [] | ||
| self.post_error = None | ||
| self.patch_error = None | ||
| self.remove_label_after_post = False | ||
|
|
||
| async def getitem(self, url): | ||
| self.calls.append(("get", url)) | ||
| return deepcopy(self.issue) | ||
|
|
||
| async def getiter(self, url): | ||
| self.calls.append(("getiter", url)) | ||
| for comment in self.comments: | ||
| yield comment | ||
|
|
||
| async def post(self, url, *, data): | ||
| self.calls.append(("post", url, data)) | ||
| if self.post_error: | ||
| raise self.post_error | ||
| self.comments.append( | ||
| {"body": data["body"], "performed_via_github_app": {"id": APP_ID}} | ||
| ) | ||
| if self.remove_label_after_post: | ||
| self.issue["labels"] = [] | ||
|
|
||
| async def patch(self, url, *, data): | ||
| self.calls.append(("patch", url, data)) | ||
| if self.patch_error: | ||
| raise self.patch_error | ||
| self.issue.update(data) | ||
|
|
||
|
|
||
| @pytest.fixture(autouse=True) | ||
| def app_id(monkeypatch): | ||
| monkeypatch.setenv("GH_APP_ID", str(APP_ID)) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize( | ||
| "kind, label, message", | ||
| [ | ||
| ("issues", policy.COMMENT_LABEL, policy.ISSUE_COMMENT_MESSAGE), | ||
| ("issues", policy.CLOSE_LABEL, policy.ISSUE_CLOSE_MESSAGE), | ||
| ("pull_request", policy.COMMENT_LABEL, policy.PR_COMMENT_MESSAGE), | ||
| ("pull_request", policy.CLOSE_LABEL, policy.PR_CLOSE_MESSAGE), | ||
| ], | ||
| ) | ||
| async def test_policy_action_through_main_router(kind, label, message): | ||
| event = make_event(kind, label) | ||
| gh = FakeGH(label) | ||
|
|
||
| await main.router.dispatch(event, gh) | ||
|
|
||
| assert gh.calls[0] == ("get", ISSUE_URL) | ||
| assert gh.calls[1] == ("getiter", COMMENTS_URL) | ||
| assert gh.comments[0]["body"] == f"{message}\n\n<!-- bedevere: {label} -->" | ||
| assert gh.calls[2] == ("post", COMMENTS_URL, {"body": gh.comments[0]["body"]}) | ||
| if label == policy.CLOSE_LABEL: | ||
| close_url = PR_URL if kind == "pull_request" else ISSUE_URL | ||
| data = {"state": "closed"} | ||
| if kind == "issues": | ||
| data["state_reason"] = "not_planned" | ||
| assert gh.calls[3:] == [("get", ISSUE_URL), ("patch", close_url, data)] | ||
| else: | ||
| assert len(gh.calls) == 3 | ||
| assert gh.issue["state"] == "open" | ||
|
|
||
|
|
||
| async def test_unrelated_label(): | ||
| gh = FakeGH("type-bug") | ||
| await policy.router.dispatch(make_event("issues", "type-bug"), gh) | ||
| assert not gh.calls | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("action", ["opened", "reopened", "synchronize", "unlabeled"]) | ||
| async def test_other_actions_do_not_trigger_closure(action): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| event = make_event("pull_request", policy.CLOSE_LABEL, action=action) | ||
| await policy.router.dispatch(event, gh) | ||
| assert not gh.calls | ||
|
|
||
|
|
||
| async def test_bot_applied_label(): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| event = make_event("issues", policy.CLOSE_LABEL, sender_type="Bot") | ||
| await policy.router.dispatch(event, gh) | ||
| assert not gh.calls | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("kind", ["issues", "pull_request"]) | ||
| async def test_already_closed_at_labeling(kind): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| event = make_event(kind, policy.CLOSE_LABEL, state="closed") | ||
| await policy.router.dispatch(event, gh) | ||
| assert not gh.calls | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("state", ["open", "closed"]) | ||
| async def test_removed_label_or_closed_target(state): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| gh.issue["state"] = state | ||
| if state == "open": | ||
| gh.issue["labels"] = [] | ||
| await policy.router.dispatch(make_event("issues", policy.CLOSE_LABEL), gh) | ||
| assert gh.calls == [("get", ISSUE_URL)] | ||
|
|
||
|
|
||
| async def test_removed_label_during_comment(): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| gh.remove_label_after_post = True | ||
| await policy.router.dispatch(make_event("issues", policy.CLOSE_LABEL), gh) | ||
| assert len(gh.comments) == 1 | ||
| assert gh.issue["state"] == "open" | ||
| assert not any(call[0] == "patch" for call in gh.calls) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("label", [policy.COMMENT_LABEL, policy.CLOSE_LABEL]) | ||
| async def test_repeated_delivery(label): | ||
| event = make_event("issues", label) | ||
| gh = FakeGH(label) | ||
| await policy.router.dispatch(event, gh) | ||
| await policy.router.dispatch(event, gh) | ||
| assert len(gh.comments) == 1 | ||
| assert sum(call[0] == "patch" for call in gh.calls) == (label == policy.CLOSE_LABEL) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("kind", ["issues", "pull_request"]) | ||
| async def test_retry_after_close_failed(kind): | ||
| event = make_event(kind, policy.CLOSE_LABEL) | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| gh.patch_error = RuntimeError("closure failed") | ||
| with pytest.raises(RuntimeError, match="closure failed"): | ||
| await policy.router.dispatch(event, gh) | ||
|
|
||
| gh.patch_error = None | ||
| await policy.router.dispatch(event, gh) | ||
| assert len(gh.comments) == 1 | ||
| assert gh.issue["state"] == "closed" | ||
|
|
||
|
|
||
| async def test_comment_failure_does_not_close(): | ||
| gh = FakeGH(policy.CLOSE_LABEL) | ||
| gh.post_error = RuntimeError("comment failed") | ||
| with pytest.raises(RuntimeError, match="comment failed"): | ||
| await policy.router.dispatch(make_event("issues", policy.CLOSE_LABEL), gh) | ||
| assert gh.issue["state"] == "open" | ||
| assert not any(call[0] == "patch" for call in gh.calls) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("app", [None, {"id": APP_ID + 1}]) | ||
| async def test_other_authors_cannot_suppress_message(app): | ||
| gh = FakeGH(policy.COMMENT_LABEL) | ||
| gh.comments.append( | ||
| { | ||
| "body": f"<!-- bedevere: {policy.COMMENT_LABEL} -->", | ||
| "performed_via_github_app": app, | ||
| } | ||
| ) | ||
| await policy.router.dispatch(make_event("issues", policy.COMMENT_LABEL), gh) | ||
| assert len(gh.comments) == 2 | ||
| assert policy.ISSUE_COMMENT_MESSAGE in gh.comments[-1]["body"] | ||
|
|
||
|
|
||
| async def test_comment_then_close(): | ||
| gh = FakeGH(policy.COMMENT_LABEL) | ||
| await policy.router.dispatch(make_event("issues", policy.COMMENT_LABEL), gh) | ||
| gh.issue["labels"].append({"name": policy.CLOSE_LABEL}) | ||
| await policy.router.dispatch(make_event("issues", policy.CLOSE_LABEL), gh) | ||
| assert len(gh.comments) == 2 | ||
| assert policy.ISSUE_COMMENT_MESSAGE in gh.comments[0]["body"] | ||
| assert policy.ISSUE_CLOSE_MESSAGE in gh.comments[1]["body"] | ||
| assert gh.issue["state"] == "closed" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.