Skip to content

feat: check the core repos against the standard every week - #118

Closed
lesnik512 wants to merge 1 commit into
mainfrom
feat/conformance-check
Closed

lesnik512 wants to merge 1 commit into
mainfrom
feat/conformance-check

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Summary

Every drift the manual audits found built up slowly, one repo at a time, and was only seen when someone went looking. This adds the check that keeps the now-clean baseline clean.

  • conformance/: a checker whose checks are named by requirement ID. It finds the core repos itself (a PyPI project that links back to github.com/modern-python/<repo>, so a PyPI name clash like chat-app is not core), clones each, reads its GitHub metadata and recent PR titles, and evaluates it.
  • .github/workflows/conformance.yml: runs it every Monday and on dispatch, writes the summary to the run page, and keeps one conformance-labelled issue per failing repo in this repo (find-or-comment, the same shape as links.yml), closing it once the repo passes. Issues live here because the workflow token cannot write to other repos.

How it reads the standard

  • Exemptions come from the standard's table, so a PR adding a row is still what grants one. the core as a whole, except [TS2] is understood, and a PV1 exemption drops the free-threaded entry from the PV2 and MD4 expectations too.
  • SHOULD and phase-ins report as warnings: RL6 (PR titles), CI6 floors-matrix drops (allowed only where no wheel exists, which a script cannot see), and TS6 bare pragmas until 2026-11-01, failures after.
  • Not checked automatically: JF3, PV3 (permissions), FL3 (glossary content), FL6 (an override is allowed and indistinguishable from drift), RM2 (wording of the opening). The summary says so.
  • RL5 compares release.yml across repos ignoring action version pins, so a Dependabot bump in one repo is not drift.

Current result

Run locally against the org: 24 core repos, 0 failing, 5 warnings (three documented floors drops, two old non-conventional PR titles).

Checklist

  • uv run pytest: 158 passed, 29 skipped (25 new tests: a fully conforming fixture repo, one failure per sampled requirement, exemptions, the TS6 date switch, RL5 consistency, scope, and parsing the real standard.md and profile)
  • mkdocs build --strict passes
  • End-to-end run against the org (above)
  • The issue-sync step has not run yet; the first scheduled or dispatched run exercises it

@lesnik512

Copy link
Copy Markdown
Member Author

Closing: drift so far came from the standard changing, and each change already ends with a sweep. A standing checker would add a maintenance tax on every standard change for little gain; audits stay on demand.

@lesnik512 lesnik512 closed this Sep 27, 2026
@lesnik512
lesnik512 deleted the feat/conformance-check branch September 27, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant