Skip to content

feat(k8s): Kubernetes restart-safety + Contabo + agent-budgets feature set - #248

Open
ciprianiacobescu wants to merge 34 commits into
mainfrom
feat/kubernetes-readiness
Open

feat(k8s): Kubernetes restart-safety + Contabo + agent-budgets feature set#248
ciprianiacobescu wants to merge 34 commits into
mainfrom
feat/kubernetes-readiness

Conversation

@ciprianiacobescu

Copy link
Copy Markdown
Contributor

Summary

  • Agent budgets — budget enforcement in proxy (HTTP 429 budget_exceeded + Retry-After), budget-consumers admin UI page, live Grafana dashboard, OTel regression guard
  • Contabo Cloud API service template — form-encoded OAuth2 ROPC token exchange, x-request-id auto-injection, credential_hint.token_request_headers propagation
  • Tailscale service template — bearer-token, tailnet API test path
  • kubernetes-readiness (ADR-0030) — four fixes that make core services restart/replica-safe:
    • A1 broker: signing key loaded from MINTKEY_BROKER_SIGNING_KEY_FILE (seed file → ed25519.NewKeyFromSeed); dev falls back to ephemeral, production fails closed
    • A2 admin-api: OIDC login state moved from in-process dict to Postgres oidc_login_state table (Liquibase changeset 030, TTL-expiring, single-use pop())
    • A3 admin-ui / seed-job: Ed25519 keypair generated once by seed-job and mounted; signed_request.py rejects unsigned writes in production when no public key is present
    • A4 liquibase: DB role passwords parameterised via ${db_app_password} / ${db_subscriber_password} substitution vars instead of hard-coded literals

Test plan

  • go test ./apps/broker/internal/keyloader/... — 4 tests (file load, cross-validate, dev ephemeral, prod fail-closed)
  • pytest apps/admin-api/tests/unit/admin_api/test_oidc_state_store.py — 4 tests
  • pytest apps/admin-api/tests/unit/admin_api/test_signed_request_prod.py — 2 tests
  • pytest apps/admin-api/tests/unit/admin_api/test_015_migration_no_literals.py — 3 tests
  • pytest apps/seed-job/tests/test_admin_ui_keypair.py — 4 tests
  • docker compose up -d — all 17 services healthy (no regression)
  • .env unset → compose starts with dev defaults (A4 backward compat)

CiprianSpot added 30 commits June 23, 2026 12:25
P1 — Bootstrap sectioning + MCP Resource
- Add XML-tag section parser to bootstrap.py (parsed once at import)
- REST GET /v1/tools/bootstrap?section= returns compact index/section/full
- REST default = 'full' (backward compat, D-2); MCP tool default = 'index'
- Register mintkey://skill/agent-bootstrap as MCP Resource via
  resources/list + resources/read in the JSON-RPC dispatcher
- mintkey_bootstrap inputSchema gains 'section' enum param

P2 — Trim tools/list
- Remove 'title' key from all 10 TOOLS entries
- Shorten secret_put/get/list/delete descriptions to load-bearing text only

P3 — Pointer-ize _BUILTIN_CAPABILITIES
- Replace 474-char string with ~120-char pointer retaining all
  literals required by TestBuiltinCapabilitiesAdvertised

P6 — Cap error hints
- _upstream_to_tool_result: hint[:120] prevents unbounded context blowup

Also:
- admin_api_client.py: typed async httpx boundary for admin-api calls (INV-2)
- 4 new test files (test_bootstrap_sections, test_mcp_resources,
  test_tools_list_trim, test_error_hint_cap); 339 tests pass, 5 skipped
…outcomes

Remediations now have audit-trail state files committed:
- 2026-05-29 ci-egress-and-playwright: CLOSED (Playwright OIDC + offline egress coverage)
- 2026-05-31 apple-jwt: CLOSED (all 7 chunks merged, apple_jwt auth scheme on main)
- 2026-05-31 ghcr-compose-install: CLOSED (PR #131 — install.sh + docker-compose.ghcr.yml)
- 2026-05-31 vault-pg-migration: CLOSED (138 creds migrated, ADR-0021, d340c18)
- 2026-06-01 ssh-credential-flow: CLOSED (PR #139 — services.base_url canonical SSH)
- 2026-06-01 ssh-proxy-fix: CLOSED (C1–C9 all merged, SSH proxy fully functional)
- 2026-06-03 email-imap-addr-and-grant-notify: CLOSED (PR #193 — IMAP addr, pg_notify, Gmail userinfo)
- 2026-06-03 oauth2-authorized-ui-flag: CLOSED (PR #193 — oauth2_authorized derived from vault)
- 2026-06-04 pr-triage-resolve: CLOSED (18/22 PRs merged, uuid vuln fixed, 4 conflicted open)
- 2026-06-07 outlook-userinfo: CLOSED (PR #197 — Graph /me fetch, User.Read scope)

Also: apps/proxy-plugin/.gitignore to exclude compiled binary from tracking.
Call-count ceilings per permission grant with automatic proxy
circuit-breaking. When the ceiling is hit, the proxy returns 429
budget_exceeded without touching the upstream.

Schema:
- Liquibase changeset 028-budget-counters (composite PK, RLS, cascade FK)

Contracts:
- OpenAPI: Constraints.budget, BudgetStatus, GET/POST budget endpoints
- OpenAPI: 4 new audit event types (threshold/exceeded/config/reset)
- MCP tools: your_constraints.budget in describe_service output
- JSON Schema: budget audit event payload schemas

Admin API (Python):
- BudgetCounter SQLAlchemy model + Pydantic schemas
- Budget validation in grant CRUD (create/PATCH)
- GET /budget + POST /budget/reset endpoints
- Period boundary utility (hourly/daily/weekly/monthly, UTC-aligned)

Proxy Plugin (Go):
- Atomic budget.Check() with INSERT...ON CONFLICT upsert
- 429 + Retry-After on ErrBudgetExceeded
- Threshold audit emission with dedup tracker
- Change-channel subscriber for budget invalidation
- Prometheus metrics (used/ceiling/denied_total)

MCP Server:
- describe_service returns real-time budget status

Observability:
- Grafana dashboard (bar gauge + denied rate + 24h stat)
- Prometheus alert BudgetNearExhaustion (>90% for 1m)

Tests:
- 24 unit tests for period bounds (Python)
- 16 unit tests for budget check (Go)
- Integration tests for config validation + endpoints
- Acceptance tests (enforcement, rollover, reset, propagation)
- Architecture tests (RLS coverage, cascade deletion)

Documentation:
- ADR-0029 promoted from draft
- AGENTS.md + CLAUDE.md guardrails updated
- CHANGELOG entry

Refs: ADR-0029, P-011, FR-1..FR-10, NFR-1..NFR-6
… BFF routes, tests

Implements ADR-0029 agent budget enforcement on the admin-ui layer.

- BudgetStatusPanel: shows call ceiling, current count, period, threshold
  markers, and Edit/Reset/Remove actions on the permission-grant detail page
- BudgetForm: create+edit form for ceiling/period/thresholds (AdminJS action)
- BFF routes (src/routes/budget.ts): GET/POST proxy to admin-api budget endpoints
- budget-validate.ts / budget-format.ts: pure utility functions (period labels,
  input validation with error messages)
- admin-api permissions.py: budget removal via PATCH using model_fields_set to
  distinguish null (remove budget) from absent (keep budget)
- 69 tests across 5 test files; 3 pre-existing failures unrelated to this work
- Kiro spec budget-management-ui: all tasks complete
…roxy, long-lived-api-keys specs

All tasks were implemented in earlier sessions; Kiro updated the task files
to reflect [x] completion status.
…ix BFF paths

The BudgetStatusPanel was showing "Unable to load budget status" (502) because:
1. GET /v1/tenants/{tid}/permissions/{pid}/budget didn't exist — admin-api only
   had the agent-scoped path and a flat list endpoint, not a single-record or
   budget-status endpoint on the tenant router.
2. BFF budgetGetHandler was doing a two-hop via lookupPermission (which also
   called a non-existent GET /permissions/{pid} endpoint) before calling the
   budget endpoint.
3. budgetEditHandler/budgetRemoveHandler/budgetResetHandler were sending PATCH/POST
   to /v1/tenants/{tid}/permissions/{pid} (flat router, no PATCH handler) instead
   of the agent-scoped path.

Fixes:
- admin-api: add GET /{permission_id}/budget to tenant_permissions_router —
  reads constraints.budget from permission_grants + used/period from
  budget_counters; returns 404 when no budget configured.
- admin-api: add GET /{permission_id} to tenant_permissions_router — returns
  id/agent_id/service_id/action/constraints (used by BFF lookupPermission for
  write operations).
- admin-api: add POST /{permission_id}/budget/reset to agent-scoped router —
  resets current-period counter to 0, emits budget.reset audit event.
- BFF budgetGetHandler: call /permissions/{pid}/budget directly (remove two-hop).
- BFF edit/remove/reset handlers: fix PATCH/POST paths to include agents/{aid}.
New AdminJS custom page showing all budget-configured permission grants
ranked by consumption percentage with:
- Server-side aggregation endpoint (admin-api): GET /v1/tenants/{tid}/budget-consumers
  joins permission_grants, budget_counters, agents, services + audit_events
  for requests_last_30_min count
- BFF proxy route: GET /admin/api/budget-consumers
- React page: data table with 30s auto-polling, client-side filtering
  (threshold %, agent name, service name), red highlight on exhausted
  rows, inline "Unlock" button to reset exhausted budgets
- AdminJS sidebar nav item "Budget Consumers" (icon: Activity)
- Pure utilities: filterConsumers (logical AND), isExhausted
- Full test suite: pytest (admin-api), vitest + supertest (BFF),
  vitest + RTL (component), vitest (unit)

Spec: .kiro/specs/budget-consumers-dashboard
…clude all grants

Two bugs:
1. audit_events timestamp column is 'at', not 'created_at' — the query was
   500-ing on every call, causing the Budget Consumers page to never load.
2. WHERE filter excluded non-budgeted grants. Removed the two predicates so
   all permission grants appear (unlimited grants have null ceiling/used).

Also: bc.used is no longer COALESCE'd to 0 in SELECT (becomes null for
unlimited grants), consumption_percentage computation guards against None
ceiling/used, and ORDER BY uses COALESCE to sort unlimited grants last.
…geted rows

- BudgetConsumerRecord: widen ceiling/used/period/consumption_percentage to
  nullable (number | null) to reflect unlimited grants from the API
- isExhausted: returns false when ceiling or used is null (unlimited = never exhausted)
- filterConsumers: treats null consumption_percentage as 0 for threshold filter
- BudgetConsumersPage: renders "Unlimited" for null ceiling/period, "—" for null
  used/consumption_percentage, relabels threshold filter to "Min consumption %",
  updates empty-state copy to "No permission grants found"
… exhausted state

Implements optional spec tasks 3.3, 4.3, 4.4 from budget-consumers-dashboard spec.

- budget-consumers-bff.property.test.ts (Task 3.3):
  Property 3: BFF constructs /v1/tenants/{tid}/budget-consumers URL from session
  Property 4: BFF proxies HTTP status + body byte-identically

- budget-consumers-filter.property.test.ts (Tasks 4.3 + 4.4):
  Property A: filterConsumers applies all active filters as logical AND
  Property B: isExhausted(record) iff used >= ceiling; false when either is null

Tasks 4.5 (consumption % computation) and 4.6 (descending sort) are N/A:
the computation is server-side Python and the sort is pre-applied by the API.
Adds a pre-configured Tailscale catalog entry to service_templates.yaml
(category: networking, auth_type: bearer_token, test_path /api/v2/tailnet/-/devices).
Data-only change — no new auth scheme, no proxy route, no DB migration.
Includes OpenSpec proposal/design/tasks and registry test assertions.
The CREATE TABLE changeset lacked a preCondition guard, causing Liquibase
to fail when the table already existed (renamed from 028-budget-counters).
The RLS and index changesets had 028- IDs that didn't match the MARK_RAN
entries already inserted. All three are now idempotent.
…teError

OTel's _get_route_details has a try/except for Match.FULL but not for
Match.PARTIAL, crashing with AttributeError when a _IncludedRouter
(produced by FastAPI's include_router) matches partially. This caused
every POST /v1/tenants/{tid}/agent-secrets request to 500 before the
handler ran. Monkey-patch the function after instrumentation.
…rash

Move _safe_get_route_details to module level so it is importable for
direct unit testing. Add four unit tests in test_otel_patch.py covering
PARTIAL/FULL match on a router without .path, NONE match, and empty routes.
Add an ASGI-level regression test in test_agent_secrets.py that applies
configure_otel() to the test app and verifies POST returns non-500.
- New `contabo` HTTP service template using oauth2_password_grant auth.
  Credentials: client_id, client_secret, username, password, grant_type=password.
  Token endpoint: Contabo Keycloak ROPC at auth.contabo.com.
  test_path: GET /v1/users/client.

- Extend oauth2_password_grant exchanger to support form-encoded token bodies.
  When token_request_headers["Content-Type"] is application/x-www-form-urlencoded,
  the credential_fields are posted as url.Values instead of JSON. Backward-compatible:
  existing JSON-body templates are unaffected. Two new tests cover both paths.

- Auto-inject X-Request-Id UUID4 header on every proxied request when absent.
  Added to all three Director closures. Required by Contabo; harmless for all others.

- Add token_request_headers to CredentialHint model and propagate it through
  the from-template service-creation endpoint (services.py) so the Content-Type
  override reaches the vault credential at creation time.

- OpenSpec: openspec/changes/contabo-service-template/ (proposal + design + tasks).
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 1 package(s) with unknown licenses.
See the Details below.

License Issues

apps/admin-api/uv.lock

PackageVersionLicenseIssue Type
joserfc1.7.2NullUnknown License
Allowed Licenses: Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause, ISC, Unlicense, CC0-1.0, Python-2.0, PSF-2.0
Excluded from license check: pkg:golang/pgregory.net/rapid, pkg:pypi/hypothesis, pkg:golang/github.com/hashicorp/go-secure-stdlib/parseutil, pkg:golang/github.com/hashicorp/go-secure-stdlib/strutil, pkg:npm/@csstools/color-helpers, pkg:npm/@csstools/css-syntax-patches-for-csstree, pkg:npm/lru-cache, pkg:golang/golang.org/x/sync, pkg:golang/golang.org/x/net, pkg:golang/golang.org/x/time, pkg:golang/golang.org/x/text, pkg:golang/google.golang.org/protobuf, pkg:golang/github.com/stretchr/testify, pkg:golang/gopkg.in/yaml.v3, pkg:pypi/types-pyyaml, pkg:pypi/asyncssh, pkg:pypi/pytest-asyncio, pkg:golang/github.com/beorn7/perks, pkg:golang/github.com/cespare/xxhash/v2, pkg:golang/github.com/go-logr/logr, pkg:golang/github.com/go-logr/stdr, pkg:golang/github.com/golang-jwt/jwt/v5, pkg:golang/github.com/google/uuid, pkg:golang/github.com/pmezard/go-difflib, pkg:golang/github.com/prometheus/client_golang, pkg:golang/github.com/prometheus/client_model, pkg:golang/github.com/prometheus/common, pkg:golang/github.com/prometheus/procfs, pkg:golang/go.opentelemetry.io/auto/sdk, pkg:golang/go.opentelemetry.io/otel, pkg:golang/go.opentelemetry.io/otel/metric, pkg:golang/go.opentelemetry.io/otel/sdk, pkg:golang/go.opentelemetry.io/otel/trace, pkg:golang/google.golang.org/genproto/googleapis/rpc, pkg:golang/google.golang.org/grpc, pkg:npm/react-router-dom, pkg:npm/tinyglobby, pkg:npm/tinymce, pkg:npm/tldts, pkg:npm/tldts-core, pkg:npm/tsx, pkg:npm/caniuse-lite, pkg:pypi/uv

OpenSSF Scorecard

PackageVersionScoreDetails
pip/joserfc 1.7.2 UnknownUnknown
npm/fast-check 3.22.0 🟢 7.7
Details
CheckScoreReason
Security-Policy🟢 10security policy file detected
Dependency-Update-Tool🟢 10update tool detected
Maintained🟢 1030 commit(s) and 4 issue activity found in the last 90 days -- score normalized to 10
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Code-Review⚠️ 0Found 0/7 approved changesets -- score normalized to 0
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Pinned-Dependencies🟢 8dependency not pinned by hash detected -- score normalized to 8
CII-Best-Practices🟢 5badge detected: Passing
Vulnerabilities⚠️ 014 existing vulnerabilities detected
SAST🟢 10SAST tool is run on all commits
License🟢 10license file detected
Packaging🟢 10packaging workflow detected
Branch-Protection🟢 3branch protection is not maximal on development and all release branches
Signed-Releases🟢 84 out of the last 4 releases have a total of 4 signed artifacts.
Fuzzing🟢 10project is fuzzed
CI-Tests🟢 1030 out of 30 merged PRs checked by a CI test -- score normalized to 10
Contributors🟢 10project has 5 contributing companies or organizations
npm/pure-rand 6.1.0 🟢 6.6
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 0Found 0/12 approved changesets -- score normalized to 0
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies🟢 10all dependencies are pinned
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Security-Policy⚠️ 0security policy file not detected
License🟢 10license file detected
Fuzzing🟢 10project is fuzzed
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md
Packaging🟢 10packaging workflow detected
SAST🟢 8SAST tool is not run on all commits -- score normalized to 8

Scanned Files

  • apps/admin-api/uv.lock
  • apps/admin-ui/pnpm-lock.yaml

Comment thread apps/admin-api/tests/unit/admin_api/test_email_service_templates.py Fixed
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_bff.test.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_bff.test.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_consumers_bff.test.ts Fixed
CiprianSpot added 2 commits July 5, 2026 21:29
…RF suppressed (sameSite+signed-JWT); tighten URL assertion in email template test
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
…1.6.5→1.7.2; fix CodeQL suppression placement

- oidc_login_state is intentionally platform-scoped (login precedes tenant
  context) — add to PLATFORM_SCOPED_TABLES to satisfy ADR-0008 architecture test
- joserfc 1.6.5 has CVE-2026-49852 (HIGH, HMAC key empty/nil bypass); pin
  joserfc>=1.6.8, lock resolves to 1.7.2
- Move codeql[js/server-side-request-forgery] suppressions to the same line as
  fetch() in budget.ts (CodeQL requires inline suppression on the flagged line)
- Move codeql[js/missing-token-validation] suppression to session({ line in
  index.ts (was one line above — not picked up by CodeQL)
- Add codeql suppressions to test harness session() calls (CSRF + clear-text
  cookie false-positives in test-only code with secure:false)
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
Comment thread apps/admin-ui/src/routes/budget.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_bff.test.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_bff.test.ts Fixed
Comment thread apps/admin-ui/tests/test_budget_consumers_bff.test.ts Fixed
Comment on lines +32 to +35
const resp = await fetch( // codeql[js/request-forgery] ADMIN_API_URL is a fixed env var; permId is path-only, validated by admin-api
`${ADMIN_API_URL}/v1/tenants/${tenantId}/permissions/${permId}`,
{ headers: { Cookie: cookie } }
);
Comment on lines +64 to +67
const upstream = await fetch( // codeql[js/request-forgery] ADMIN_API_URL is a fixed env var; permId is path-only, validated by admin-api
`${ADMIN_API_URL}/v1/tenants/${tenantId}/permissions/${permId}/budget`,
{ headers: { Cookie: cookie } }
);
Comment on lines +66 to +71
session({ // codeql[js/missing-csrf-middleware] codeql[js/clear-text-cookie] test harness only
secret: "test-secret",
resave: false,
saveUninitialized: true,
cookie: { secure: false },
})
Comment on lines +66 to +71
session({ // codeql[js/missing-csrf-middleware] codeql[js/clear-text-cookie] test harness only
secret: "test-secret",
resave: false,
saveUninitialized: true,
cookie: { secure: false },
})
Comment on lines +67 to +72
session({ // codeql[js/missing-csrf-middleware] codeql[js/clear-text-cookie] test harness only
secret: "test-secret",
resave: false,
saveUninitialized: true,
cookie: { secure: false },
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants