agentctl

Turns GitLab milestones into merged merge requests. You say /swarm colmena 7; isolated pods on neutrino pick up groups of related issues, open MRs, and the ones that pass their project's own CI and violate no protected path merge themselves.

The unit of work is a group, not an issue. Issues that share a theme — or, more to the point, share files — go to one agent on one branch. Two agents independently editing the same service produce two merge requests that cannot both merge, and you find out after paying for both. Grouping makes that collision impossible rather than unlikely, and the shared cost of reading the codebase is paid once instead of three times.

A group is not atomic: the agent writes /out/addressed.json saying which issues it actually solved, and only those are closed. One impossible issue must not drag two solved ones back into the backlog.

Target repositories are pure content — agentctl never writes a file into them, never adds a CI job, never asks them to know it exists.

This is the part to understand before anything else. Everything else is detail.

telegram / CLI ──▶ agentctl  ── holds ALL credentials, `api` scope, cluster access
                    │
                    │ mints per-run token: 1 project, write_repository, short TTL
                    ▼
   ns: agents  ┌───────────────────────────────────────────────┐
               │ opencode + repo                               │
               │ egress: internet:443 only, RFC1918 BLOCKED    │
               │ no ServiceAccount token · no k8s API          │
               │ quota · 40min deadline · readOnlyRootFilesystem│
               └──────────────┬────────────────────────────────┘
                              │ pushes agent/issue-N   (cannot open the MR)
                              ▼
              agentctl opens the MR with its own api token
                              ▼
              the project's existing CI runs = the gate

Two rules produce most of the safety:

1. The pod never gets api scope. An api token can read the target project's CI/CD variables — your production database password. It gets write_repository and nothing else, so it can push a branch and that is the entire extent of its reach.
2. Egress blocks private address space, not the internet. neutrino runs production storefronts. The agent's dangerous neighbours are in-cluster and on the LAN, not on the public internet. Deleting the except block in k8s/01-networkpolicy.yaml defeats the whole model.

• Mark sensitive CI/CD variables protected. The agent controls the contents of the branch it pushes, including .gitlab-ci.yml; protected variables are only exposed on protected branches, and agent/* is not one. Without this, an agent-authored pipeline runs with your production credentials.
• Ensure agent/* is not a protected branch pattern.

Everything above compiles into one binary. internal/gitlab is a package, not a service: the process that opens merge requests is the same one that holds the api token, schedules Jobs, ingests logs and serves the UI. Splitting it would mean an internal API that itself needs authenticating, and both halves would still need the same token — the boundary that matters is agentctl vs. the agent pod, and that one is a container, not a port.

export AGENTCTL_DSN=postgres://…  GITLAB_TOKEN=…  AGENT_IMAGE=registry.gitlab.com/…/agent:latest
agentctl migrate                       # apply the schema
agentctl serve                         # control plane on :8080
agentctl run   grp/raffles 1           # one issue, now
agentctl swarm grp/neutrino "v1.2"     # a whole milestone
agentctl reap                          # orphaned pods and stray credentials
agentctl reconcile                     # grade attempts a human merged or closed

serve runs reap and reconcile on a timer as well. The reconciler is what keeps the ledger complete: the gate leaves an attempt pending whenever it refuses for a reason a human has to settle — a protected path, a conflict, no pipeline — and those merge requests are graded from what the human then did with them.

Clustering a milestone is one inference, and on a hundred-issue milestone that one inference decides how a hundred pods get spent. So the grouping is written down before anything spawns, and a large one waits for you:

agentctl swarm grp/neutrino "v1.2"   # over the threshold → prints a plan, spawns nothing
agentctl plan list                   # what is waiting
agentctl plan show 12                # the groups, their issues, and why
agentctl plan merge 12 1 3           # "these two should be one group"
agentctl plan split 12 9             # pull #9 out on its own
agentctl plan approve 12             # now it spawns
agentctl plan cancel 12              # or it never does

Thresholds are in settings: approval_max_groups (default 4) and approval_max_cost_usd (default 5.00). A milestone under both runs straight through — its grouping is still recorded, auto-approved, so "why did these two issues go to one agent" stays answerable a week later. Either at 0 switches that dimension off.

The estimate is the average cost of this project's past priced runs times the number of groups. A project with no history has no estimate, and unknown is not treated as cheap: the cost threshold simply cannot trip until the ledger has something real in it.

Approve/edit/cancel is a CLI command rather than an HTTP endpoint on purpose. The server has no authentication by design (Cloudflare Access sits in front, and it serves reads); "spawn twenty pods" is not a verb to hang off an unauthenticated mux in the one process that holds an api-scoped GitLab token. When the UI lands it can call the same scheduler methods.

Configuration is environment-only: a flag holding a GitLab token ends up in the process list and in your shell history.

make check is the gate. The pre-commit hook, CI and your terminal all run the same target, so there is one definition of "this passes" and no way for the three to drift.

make hooks    # install the pre-commit hook (once)
make check    # tidy · fmt · vet · lint · govulncheck · shellcheck · test -race
make help     # everything else

Every Go tool is pinned in go.mod under the tool directive and invoked via go tool, so nothing depends on what happens to be on your PATH.

The store tests need Postgres and skip silently without it — make check starts a throwaway container so a green run never means "the database tests didn't run".

Nothing has run yet. Step 1 below is the go/no-go.

A survey of every repo (2026-08-08) found that only raffles has CI that gates a merge request:

rules:
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Copying raffles' rules: block into neutrino's test jobs and dropping allow_failure: true is worth doing regardless of agentctl — right now its PHPUnit suite only runs after merge to default, so breakage in invoicing code is found post-merge.

neutrino is also five applications in one repository — supercartas, memetropolis, buzzly, cfdi, forwarder — and an agent handed 3680 files spends most of its context orienting before it writes anything. So a run is scoped to a subdirectory: projects.working_dir pins one for a whole project, and when it is empty the clusterer's predicted files pick one per group (Group.WorkingDir, capped at two levels — the point is to choose the app, not the package). The pod receives it as WORKING_DIR and the entrypoint starts the agent there.

It is a starting point, not a sandbox. The clone is whole, the commit still covers the whole tree, and a directory that does not exist falls back to the root — a wrong prediction costs orientation, never correctness.

The only question worth answering before writing a scheduler: does opencode-in-a-pod turn an issue into an MR you would actually merge? If it doesn't, none of the rest matters. Run it on raffles — the goal here is proving the plumbing end to end, not evaluating models.

# 1. the agent image is built by CI on master (see build-agent-image).
#    To build it by hand:
docker build -t registry.example.com/you/agentctl/agent:dev runtime/
docker push  registry.example.com/you/agentctl/agent:dev

# 2. cluster prerequisites (kubectl context: neutrino)
kubectl apply -f k8s/00-namespace.yaml
kubectl apply -f k8s/01-networkpolicy.yaml
kubectl apply -f k8s/02-limits.yaml
kubectl -n agents create secret generic openrouter --from-literal=api-key="$OPENROUTER_API_KEY"
# the cluster needs a pull secret for the private registry
kubectl -n agents create secret docker-registry gitlab-registry \
  --docker-server=registry.gitlab.com \
  --docker-username=<deploy-token-user> --docker-password=<deploy-token>

# 3. PROVE the isolation before running an agent (step 2, but do it now)
#    Edit bin/verify-isolation.sh first: add your actual production
#    Postgres host:port. A probe that only tests 10.43.0.1 proves nothing.
kubectl -n agents run isolation-probe --rm -it --restart=Never \
  --image=registry.example.com/you/agentctl/agent:latest \
  --labels='app.kubernetes.io/name=agentctl-agent' \
  --overrides='{"spec":{"serviceAccountName":"agent","automountServiceAccountToken":false}}' \
  --command -- bash -c "$(cat bin/verify-isolation.sh)"

# 4. run one issue
export AGENT_IMAGE=registry.example.com/you/agentctl/agent:latest
export AGENT_GIT_TOKEN=<project access token, write_repository scope ONLY>
./bin/run-issue.sh <group>/raffles 1

run-issue.sh does from your laptop exactly what agentctl will later do from the cluster: fetch the issue with your api token, mount it, mint a scoped secret, launch the Job, stream logs, clean up credentials on exit.

Judge the resulting MR the way you would judge a contractor's. If it needs rewriting, fix runtime/prompt.tmpl and run it again — the prompt is a bigger lever than the model at this stage.

The cornerstone, and the reason the schema looks the way it does. New models appear constantly, so the ladder cannot be a list you maintain by hand.

Escalation and exploration are different mechanisms. Within one issue, a failed rung escalates to a stronger model — and how it escalates depends on how it failed:

Resuming matters more than it looks. The second model gets the diff and the reason it failed, which is strictly more than any first attempt can have, and the work already paid for is not thrown away. It stays free to git reset --hard back to base if the approach is wrong at the root — the prompt says so explicitly, because a strong model handed a bad design will otherwise patch symptoms. Across issues, ε of runs explore a candidate, which is how new models earn their way onto the ladder. Exploration is forced off during time-boxed milestone burns: you want throughput then, not science.

The metric is cost per clean merge, not cost per token. attempts.outcome distinguishes ci_green_clean_merge from ci_green_human_edited, detected from git authorship on the branch before merge. Systems that only measure "CI passed" reliably conclude the cheap model is fine, because they never count the time you spent fixing its output. Verified on seed data:

model                       clean%   $/clean merge   human touches/merge
z-ai/glm-4.6                 20.0        $0.50              1.20
anthropic/claude-sonnet-5    70.0        $1.14              0.13

Cost alone says glm wins by 2.3×. The last column says it needs 9× the hand-holding. Both numbers have to be on screen or the ledger lies to you.

Rank on the run, not on the attempt. A rung that fails CI is resumed by the next one, so a clean merge can be two models' work — and model_effectiveness, keyed on the attempt, hands the finisher the whole merge while charging it only for the fix. run_effectiveness keys on the run: one row per run, who put commits on the branch, and what every rung of it cost. model_run_effectiveness rolls that up per finishing model and carries both numbers side by side, so the size of the flattery is one subtraction away.

Expect to distinguish good models from bad ones within a few dozen runs, and good from slightly-better approximately never. The coarse decisions are where the money is.

• Own k8s Jobs, not GitLab CI as executor. Once the job definition had to live outside the target repo, CI was just a remote control for creating a pod — an indirection layer bought for a borrowed log viewer.
• Milestone command only. No issue webhooks. Nothing runs uninvited; that is the single most effective runaway prevention available.
• Auto-merge on green + path policy. Deploy stays human-initiated.
• LLM review is advisory, never a gate. An LLM reviewing an LLM's output has correlated failure modes. Deterministic gates decide.
• The gate fails closed. No pipeline, an unreadable diff, an unrecognised merge status, a policy pattern that won't compile — every unknown blocks. A wrong block costs you one manual merge; a wrong merge puts unreviewed machine-written code in production. internal/gates is pure: no network, no database, no clock, so the decision is reproducible from its inputs.
• Any diff touching migrations or DDL never auto-merges and never auto-deploys. claude-ddl-checkpoint.sh first, by a human.
• Read-only mirror PVC + git clone --reference, not shared worktrees on a mutable .git — concurrent worktree add is not safe and one corrupted .git takes out every running agent.
• Go + templ + htmx + SSE + Postgres, single binary, behind Cloudflare Access. No app-level auth.
• stdout is the result channel. /out/result.json lives on an emptyDir that dies with the pod, so the entrypoint also prints the same JSON behind ##agentctl-result## as its last line and the driver greps for it. Only branch/commit/message come from there; the exit code comes from the container's terminated state, and the gate is CI. Nothing downstream trusts the sentinel.
• Go never does arithmetic on money. Every sum, ratio and per-merge figure is computed by Postgres against numeric(12,6); store.USD is a transport type, not an accumulator. The ledger is the product, and float drift in it is the one bug you could never spot by reading the dashboard.
• A slow log viewer gets dropped, never blocks ingestion. Durability is Postgres's job; the client reconnects with Last-Event-ID and replays. One wedged browser tab must not stall a run's database writes.
• Patch-back was designed and deferred — reserved for repos not under your control. Against your own projects it is ceremony.

• No smoke suites exist yet in any target project. Agent-driven deploy stays blocked until they do — an agent will happily report a green deploy of a broken app.
• raffles has a real gate but only 2 test files, so "CI green" there is a weak signal. Fine for a plumbing shakedown, useless for grading models — don't let early raffles results set the ladder.
• neutrino cannot be a target until its test jobs run on merge_request_event. Until then auto-merge-on-green would be merging on a pipeline of docker builds.
• bin/verify-isolation.sh ships with placeholder addresses. It is worthless until you put your real production Postgres in it.
• Per-run token minting is unproven. Verified 2026-08-08 against SimplicityLover/raffles: POST .../access_tokens returns 201 with scopes: ["write_repository"], access_level: 30; DELETE returns 204. The per-run credential design works on this plan. Two things that came out of the check: GitLab's expires_at is a date (one-day floor, so revocation is the real control), and a revoked token stays in the list with active: false — ListRunTokens filters those out or the reaper reports every cleaned-up credential as an outstanding leak.
• GitLab token expiry is date-granular. expires_at takes YYYY-MM-DD, so the "short TTL" in the design cannot be enforced by GitLab. The real bound is RevokeRunToken on the run's exit path plus Reap; treat expiry as the backstop for agentctl dying, not as the control.