Skip to content

Database-backed plugin directory - #583

Merged
compscidr merged 22 commits into
mainfrom
feat/db-backed-plugin-directory
Sep 20, 2026
Merged

compscidr merged 22 commits into
mainfrom
feat/db-backed-plugin-directory

Conversation

@compscidr

Copy link
Copy Markdown
Collaborator

Summary

  • The directory plugin is now the registry: repos are submitted at /plugins/submit (plain HTML form, honeypot, 5/h/IP, one validation at a time, 90 s budget), validated in-process (wasm.LoadBytes in the Extism sandbox, no compilation-cache retention) and queued; admins approve / reject / add / rebuild / delist under Admin → Plugins → Directory; /plugins, /plugins/<name>, /plugins/<name>.json and /plugins/index.json are served from two new tables (directory_repos, directory_builds).
  • plugins/directory/registry ports the registry repo's validation/build (GitHub over net/http, no new deps): manifest rules, ValidateEntry, BuildRepo, LatestVersion. Scheduled refresh only re-downloads when a repo's latest tag changed; a failed rebuild keeps the previous document and records last_error.
  • Seven admin endpoints under /api/v1/directory/…; /api/v1/plugins/status gains directory_hosted.
  • The installer's Fetcher moves to plugin/installer (no behaviour change for other goblogs).
  • Settings: index_url removed; refresh_minutes default 360 (min 15); optional github_token (password input).
  • Docs: spec docs/superpowers/specs/2026-09-20-db-backed-plugin-directory-design.md (+ plan), docs/PLUGIN_CONTRACT.md moved in from the registry repo, README section rewritten.

Follow-ups (not in this PR)

Test plan

  • go test ./... green (new tests: registry ×4 files, store/limiter/service/directory/submit, admin directory API, wasm NoCache)
  • Local run against real GitHub: enabled the plugin, submitted goblog-plugin-hello and goblog-plugin-scholar via /plugins/submit (both queued), approved hello → /plugins, /plugins/hello, /plugins/hello.json, /plugins/index.json correct
  • Browser pass of Admin → Plugins → Directory (Add, Show README, Reject, Approve, Rebuild, Delist) — see spec §6 checklist
  • After release: verify goblog.live site_url, set github_token, refresh_minutes=360, seed hello + scholar

🤖 Generated with Claude Code

compscidr and others added 20 commits September 19, 2026 22:16
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
rebuildRepo now scopes its final write to the build's id and refuses it
(rather than falling back to Create) if Delist removed the row during the
in-flight network build, and checks nameTaken before writing a renamed
plugin's document so a collision during rebuild is ErrNameTaken with the
previous document kept, not a raw DB error.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
RejectDirectoryRepo ignored BindJSON's error, which had already
written a 400 on decode failure (including on an empty body, which
was meant to mean "no reason"); the handler then wrote 200 on top of
it. Switch to ShouldBindJSON, only treat a malformed non-empty body as
an error, and switch AddDirectoryRepo to ShouldBindJSON for
consistency.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…er GC

Drop the gorm text-type tag on Build.Doc/LastError (MySQL's 64 KiB text
cap can't hold a big README+changelog+release-notes doc). Add
wasm.Options.NoCache so validating a public submission doesn't pin its
compiled module in the process-wide wazero cache forever. Bound the
directory's IP limiter map with a periodic sweep of expired addresses
(unbounded over IPv6). Set the 429 Content-Type before flushing the
header so it isn't sniffed. Use Limit(1).Find instead of First for
Detail() so crawler hits on unknown plugin names stop spamming gorm
ERROR logs. Cap GitHub release pages at 10 and rendered README/CHANGELOG
HTML at 1 MiB so one pathological repo can't cost unbounded API calls or
storage. Fix a stale renderDetail comment.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
README: note that README/changelog/release-note HTML is rendered by
GitHub's markdown API and shown as-is, including in the admin's pending
card. PLUGIN_CONTRACT: "the tool" is now "submission" (there is no
separate tool). Spec: submitter_ip is kept for the admin (rate limiting
is in-memory, not DB-backed); the wire types live in
plugins/directory/registry and are aliased in plugins/directory, only
Fetcher moves; requireDirectory's 503 is about the plugin not being
wired/initialised, not about enabled=false (admin curation works while
disabled, for seeding); expand the seeding checklist (site_url,
github_token, refresh_minutes, a manual Directory-tab pass); rename Out
of scope to Out of scope / accepted risks and record the unbounded
pending-submission count, the 90s lock-hold risk from dropping the
Docker fence, and the CSRF follow-up's new submit-then-approve angle.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

There are concrete security/UX issues to address in the new directory admin API and submit parsing (notably Content-Type enforcement for CSRF hardening and case-insensitive GitHub URL parsing).

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 Medium severity

Open (3)
What changed in this PR

This PR converts the built-in directory plugin from a remote-index mirror into a database-backed plugin registry that validates GitHub repos in-process, stores curated entries/build artifacts in new DB tables, and serves /plugins pages + JSON from that data. It also adds admin APIs/UI for approvals and moves the installer’s directory fetcher into plugin/installer.

Changes:

  • Implement DB-backed directory registry with submission flow (/plugins/submit), validation/build pipeline, and scheduled refresh.
  • Add admin directory API endpoints and a new Admin → Plugins → Directory tab for curation.
  • Move installer Fetcher into plugin/installer, add wasm NoCache option for validation, and update docs/README.
File Description
themes/​default/​templates/​admin_settings.html Add password input rendering for plugin settings.
themes/​default/​templates/​admin_plugins.html Add “Directory” admin tab and JS to manage pending/approved/rejected repos.
README.md Rewrite plugin directory documentation to match DB-backed flow.
plugins/​directory/​templates/​submit.html New public submission form template with honeypot.
plugins/​directory/​templates/​listing.html Update listing copy and empty-directory messaging; link to submit page.
plugins/​directory/​submit.go Implement submit parsing, honeypot, rate-limit/busy handling, and rendering.
plugins/​directory/​submit_test.go Tests for repo parsing and submit handler behavior/status codes.
plugins/​directory/​store.go Add directory_repos / directory_builds models + index encoding helpers.
plugins/​directory/​store_test.go Persistence + encoding tests for repo/build storage and index generation.
plugins/​directory/​service.go Core registry service: submit/add/approve/reject/delist/rebuild/refresh/index cache.
plugins/​directory/​service_test.go Service behavior tests including races, rebuild semantics, refresh, and listing.
plugins/​directory/​render.go Add submit page renderer; adjust detail renderer signature/docs.
plugins/​directory/​registry/​validator.go Add WasmValidator + FakeValidator and sha helper.
plugins/​directory/​registry/​validator_test.go Validator tests (identity, garbage, cancelled context, fake validator).
plugins/​directory/​registry/​validate.go Port/implement end-to-end repo validation (ValidateEntry, LatestVersion).
plugins/​directory/​registry/​validate_test.go Validation tests covering tag rules, missing files/assets, mismatches, etc.
plugins/​directory/​registry/​source.go Implement GitHub REST client (Source) with paging, limits, markdown rendering.
plugins/​directory/​registry/​source_test.go HTTP fake tests for GitHubSource correctness and bounds.
plugins/​directory/​registry/​manifest.go Manifest schema + validation rules (name/license/runtime/hosts/min version).
plugins/​directory/​registry/​manifest_test.go Manifest validation tests (defaults, host rules, errors).
plugins/​directory/​registry/​build.go Build detail docs (README/changelog/releases), size caps, stars best-effort.
plugins/​directory/​registry/​build_test.go Build behavior tests, URL shaping, rendered-size limits, error propagation.
plugins/​directory/​limiter.go In-memory IP limiter with sweep to bound memory growth.
plugins/​directory/​limiter_test.go Limiter behavior and sweep pruning tests.
plugins/​directory/​index.go Replace local wire types with aliases to registry types.
plugins/​directory/​directory.go Make directory plugin the registry: migrations, service wiring, routes, refresh job.
plugins/​directory/​directory_test.go Integration tests for directory plugin pages, JSON, submit, and hosted behavior.
plugin/​wasm/​wasm.go Add NoCache option to skip shared compilation cache for validation loads.
plugin/​wasm/​wasm_test.go Test NoCache load path.
plugin/​installer/​installer.go Switch installer to use local Fetcher and add directory_hosted to status.
plugin/​installer/​installer_test.go Update tests for moved fetcher.
plugin/​installer/​fetcher.go Move fetcher implementation into installer package; keep types from directory pkg.
plugin/​installer/​fetcher_test.go Update package declaration for moved fetcher tests.
goblog.go Wire installer fetcher move; wire directory plugin into admin; add directory API routes.
docs/​superpowers/​specs/​2026-09-20-db-backed-plugin-directory-design.md Add design/spec doc for DB-backed registry.
docs/​PLUGIN_CONTRACT.md Add/port plugin publishing contract documentation.
admin/​plugins.go Include directory_hosted in /api/v1/plugins/status.
admin/​plugins_test.go Update harness for installer fetcher move and admin pointer usage.
admin/​directory.go New admin directory API handlers (/api/v1/directory/...).
admin/​directory_test.go Tests for directory API lifecycle, error mapping, and plugin status integration.
admin/​admin.go Add Directory *directory.Plugin field to Admin wiring.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread admin/directory.go
Comment thread plugins/directory/submit.go
Comment thread themes/default/templates/admin_settings.html
compscidr and others added 2 commits September 20, 2026 07:18
…-directory

# Conflicts:
#	plugins/directory/directory_test.go
…CSRF test for approve

ParseRepo now accepts any capitalisation of github.com and ErrBadRepo
mentions the bare owner/repo form. Password-typed plugin settings render
empty in every theme (the stored secret no longer reaches the page) and
an empty password is skipped on save, so blank means keep. The
Content-Type check from #571 already covers the directory endpoints;
csrf_test.go now pins the cross-site /approve case.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@compscidr
compscidr merged commit fbf6479 into main Sep 20, 2026
1 check passed
@compscidr
compscidr deleted the feat/db-backed-plugin-directory branch September 20, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants