Engineering

13 SystemsBare Metal K3sGitOps GitHub ↗

Deep dives into the systems on a bare-metal K3s cluster at home, deployed end-to-end with GitOps. Each covers why it exists and how it's built.

Motivation

The first version ran unattended Claude agents in sandboxed Kubernetes pods dispatched over NATS. When Claude's terms of service changed around unattended agent use, that dispatch model no longer held. Rather than retire the idea, I split the execution substrate apart from the model choice and rebuilt the substrate on Firecracker: every agent request gets its own hardware-isolated microVM instead of a shared pod, paused and snapshotted when idle so it costs nothing, and restored in tens of milliseconds when woken. The durable pieces from v1 carried over (on-cluster inference, the MCP gateway, the knowledge-graph surface); the sandbox underneath them got much stronger.

MicroVM per request
Every agent request runs in a fresh Firecracker microVM (kata-fc on a bare-metal node), not a shared container. For untrusted, agent-written code the isolation boundary is hardware-level, not a userspace kernel.
Snapshot / restore
An idle agent thread is paused and snapshotted (memory plus rootfs), releasing all compute, then restored on the next turn. Measured restore is 28ms cold, 6ms warm; trigger to first model call is ~140ms. A thin Postgres-reconcile controller owns the lifecycle, porting E2B's open-source snapshot architecture onto the Firecracker primitive.
Secret-swap egress
The guest is vsock-only and never holds a real credential. A TLS-terminating sidecar routes by SNI/Host and swaps a placeholder token for the real secret (a GitHub or model API key) at the network hop, so sandboxed code uses credentials it can never read or exfiltrate.
Postgres control plane
High-churn idle-agent state lives in Postgres, not etcd, keeping thousands of waiting threads off the cluster control plane. The same registry backs the list and resume catalog exposed over MCP.
Local inference
vLLM serves a Qwen3.6 MoE (35B-A3B, int4) on a single RTX 4090 for routine work; frontier models are reached over the swapped egress only where the task warrants it.
Motivation

fc-invoke proved the substrate but hardcoded one workload shape: a stateless invoke. EmberVM is the successor, built by forking the node daemon and putting a BEAM control plane in front of it. Semgrep scans already run on it, a public image renderer serves warm from it, and fc-invoke is frozen with the goose agent as its last tenant. The design goal: the control plane owns placement and policy but stays off every serving hit path.

Workload classes
task (one-shot, vsock-only guest with no NIC), session (a stateful sandbox, banked to disk when idle and relit on the next invoke), and serving (a warm HTTP endpoint on a tap NIC).
Serving data path
The control plane publishes endpoints over xDS to a per-node Envoy; the node's kernel DNATs each connection into the VM's tap network. Requests never touch the control plane or the Kubernetes apiserver.
Quotas and metering
Fail-closed: a principal with quota 0 is hard-stopped at submit. Metering rides the operation itself, bills success and failure alike, and is queryable at /v1/usage.
Public exposure
One public route, scoped at three layers: the HTTPRoute pins host and path, the node Envoy exact-matches the internal authority, and the guest shim keeps its control endpoints unreachable. 120 requests/min at Envoy plus a 3600 vCPU-second daily quota.
State
A Postgres op-log, not etcd: a 30-day journal with 7-day terminal-task retention. The milestone log, including post-ship defect records, is DECISIONS.md at the repo root.
03

Goosecracker

Agents
Motivation

I wanted a make-me-a-thing button: type a prompt in Discord, get back a working hosted mini-app. The hard part is doing it both safely and fast. The agent writes and runs untrusted code, so it has to be strongly isolated, but a fresh microVM per request must not feel slow. Goosecracker is the proof that per-request hardware isolation and sub-second cold start are not a trade-off.

Cold start
Dispatch to first model call is ~140ms: 10ms dispatch, a 83ms copy-on-write microVM cold start (35ms rootfs, 28ms Firecracker boot, ~20ms guest init), then 50ms to bring the agent up. A fresh VM per request, not a warm pool.
Isolation
Every request runs in its own Firecracker microVM with a hand-rolled init as PID 1. No shared container and no shared kernel between requests.
Secrets
The guest is vsock-only. A TLS-terminating proxy MITM-replaces a placeholder API key with the real one at the egress hop, so the sandbox can call an external model without ever holding the key.
Artifact
The agent builds a self-contained HTML artifact inside the VM and publishes it to object storage; it is served sandboxed at jomcgi.dev/artifact/<id> behind a strict CSP, with hot reload on iteration.
Shared substrate
Built on the same snapshot/restore substrate as the agent platform: the microVM, the egress sidecar, and the in-guest supervisor are reused, not one-offs.
Motivation

Notes are only useful if they come back at the right moment. The knowledge pipeline turns markdown into a queryable graph: an on-cluster LLM decomposes each note into atomic facts, critiques its own extraction, and stores embeddings for semantic recall.

Decomposition
An on-cluster Qwen model decomposes markdown into structured facts, with a self-critique pass before anything is committed to the graph.
Embeddings
voyage embeddings stored in Postgres pgvector with HNSW indexes. One database holds facts, edges, and vectors; no separate vector store to run.
MCP surface
Search, notes, tasks, and research-gap tools exposed over MCP, so any Claude session (local or scheduled) can read and write the graph.
Gap research
The graph files research gaps for itself. External questions are auto-researched by agents; judgment calls queue for human review.
This site
The notes view and Cmd+K search overlay on this site render the same live graph through the same API.
05

Loom

Data Pre-alpha
Motivation

Operating a governed data platform should scale sub-linearly with data and org size: a new dataset, domain, or transform should add no new system to run. Loom is a collaborative bet on that premise, with governance treated as a safety property (STPA hazard analysis, where unsafe means a governance violation, not a crash).

Status
Pre-alpha, built with a collaborator, to be open-sourced. The control-plane library exists; services are in progress.
Control plane
A single Postgres owns queue, catalog, ontology, ACL, and lineage as schema-separated concerns. The job queue is SKIP LOCKED plus LISTEN/NOTIFY, no broker.
Compute
Rust services embed Apache DataFusion. Tables are DuckLake on S3-compatible object storage, with the catalog in the same Postgres.
Wire protocol
Quack: DuckDB-compatible remote access, so existing DuckDB clients connect without a custom driver.
Governance
OpenLineage events plus STPA-derived constraints. The ontology and ACLs live next to the data they govern, not in a bolt-on service.
Motivation

HuggingFace models are huge and slow to download. I wanted to reference a model in a pod spec the same way you reference a container image, and have it just work. The operator caches models in an OCI registry and streams them to pods without touching disk.

PodMutator
An admission webhook intercepts pods with hf.co/ volume references, resolves the OCI ref synchronously (pod specs are immutable after admission), creates a ModelCache CR, and gates scheduling until the model is synced.
State machine
Built with Sextant: Pending, Resolving, Syncing, Ready, with a Failed state. Guards distinguish permanent errors from transient failures for automatic retry.
hf2oci
Streams HuggingFace models into OCI layers: HTTP response to tar to io.Pipe to registry push. Zero disk I/O. Safetensors and GGUF formats.
Smart naming
The HuggingFace baseModels API resolves derivative models to their base, so derivatives share OCI layers with the base repo for deduplication.
volumes:
  - name: model
    image:
      reference: hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF
07

Sextant

Operators
Motivation

Kubernetes operators are state machines, but we write them as imperative reconciliation loops. Every operator I wrote had the same bugs: invalid state transitions, forgotten error handling, missing metrics. Sextant defines the state machine declaratively and generates the boilerplate.

Compile-time safety
Each state is a Go struct and transitions return the next state's type. Going from Pending to Ready without passing through Creating is a compiler error.
Forced idempotency
Transition methods require request IDs in their signatures, which forces you to call the external API and record its ID before transitioning.
Guard conditions
Go expressions embedded in the YAML, evaluated at transition time. Invalid expressions fail at code-generation time, not in production.
CI drift guard
Generated code is committed, and CI regenerates from the spec and fails on any drift. The spec next to the operator is always the truth.
Generated metrics
Prometheus counters, histograms, and a state-duration gauge per machine, with automatic cleanup on resource deletion.
states:
  - name: Pending
    initial: true
  - name: Creating
    fields:
      requestID: string
  - name: Ready
    terminal: true

transitions:
  - from: Pending
    to: Creating
    action: StartCreation
    params:
      - requestID: string
08

Cloudflare Operator

Operators
Motivation

Every new service meant clicking through the Cloudflare dashboard: create a DNS record, create a Zero Trust application, update the tunnel config. I wanted to annotate a Deployment and have everything provisioned automatically.

Annotation-driven
cloudflare.ingress.hostname and cloudflare.zero-trust.policy annotations trigger reconciliation. No CRDs to manage for the common case.
State machine
Built with Sextant. Pending to CreatingDNS to CreatingZTApp to UpdatingConfig to Ready, each step idempotent.
Finalizers
Deleting the Deployment cleans up DNS records, Zero Trust apps, and tunnel routes. No orphaned Cloudflare resources.
Drift detection
Periodic reconciliation reverts manual dashboard edits. The operator is the source of truth.
metadata:
  annotations:
    cloudflare.ingress.hostname: myapp.jomcgi.dev
    cloudflare.zero-trust.policy: joe-only
09

Trips: Camera to Browser

Apps Live ↗
Motivation

I wanted an easy way to share trips with friends and family. A GoPro on the dash captures photos automatically, and my homelab turns them into a map they can follow day by day, or replay later. Works for anything with GPS-tagged photos.

Capture
A Python asyncio controller drives the camera over WiFi: GPS-triggered interval capture, a persistent SQLite download queue, and exponential backoff on connection drops.
Ingest
Each geotagged frame is POSTed to a private endpoint gated by Cloudflare Access at the gateway (no app-level key). The server extracts EXIF, derives a trip point, and upserts it. The write path ships only in the private image, so it is unreachable from the internet.
Store
Postgres is the source of truth for trip points; folding trips into the monolith retired the old NATS event store. Image bytes live content-addressed in SeaweedFS, so a re-POST of the same frame is idempotent.
Delivery
imgproxy resizes originals on the fly and Cloudflare caches at the edge. Content-addressed keys mean cache invalidation is never needed.
Display
A read-only, SSR tier renders the public pages from localhost in-pod and is CDN-cached: MapLibre vector tiles with terrain hillshade, day-by-day galleries, and an elevation-profile scrubber. No live socket, the public reads never touch the write path.
10

Ships: AIS Vessel Tracking

Apps Live ↗
Motivation

Living near the coast, I wanted to see what ships are passing by in real time. AIS data is publicly broadcast by vessels, but there is no simple way to visualize it locally. This pipeline streams AIS data through my cluster onto a map.

AIS ingest
A supervised background task inside the monolith holds a websocket to AISStream.io, filters to a Pacific Northwest bounding box, and batches position reports. NATS is gone: it writes straight to Postgres, and an AISStream hiccup can never crash the app.
Storage
Postgres is the single source of truth. ships.positions is range-partitioned by day, so retention drops whole partitions with zero vacuum churn; a stateless persister reads affected vessels back, dedups, and upserts a latest-positions serving table.
Ships API
SSR-only REST reached from localhost in-pod: a snapshot endpoint for the initial render and a per-vessel track on click. Both are CDN-cached with ETag 304s, so they run a few times a minute regardless of how many browsers are watching.
MapLibre UI
Vessels render as a GPU GeoJSON symbol layer, directional icons by heading, so panning stays smooth at scale. A separate WebGL layer maps distinct-vessel traffic density per ~500m cell.
11

Stars: Dark-Sky Windows

Apps Live ↗
Motivation

Finding a good stargazing night means combining how dark a site is with whether the sky will actually be clear once it gets dark. Stars pairs a light-pollution grid of road-accessible dark sites with rolling weather forecasts and surfaces the upcoming hours that are both dark and clear.

Site grid
A light-pollution grid of ~14k road-accessible dark sites is built offline and uploaded to SeaweedFS; a scheduled job wholesale-replaces the stars.sites table from it, so the scorer always works off a current site list.
Forecast scoring
An hourly job fetches MET Norway forecasts for every site and scores each future hour for darkness (sun below the threshold, astronomy via astral) and clear sky (cloud below the threshold).
Metric
The unit is clear-dark hours, not a single composite score. Qualifying hours land in Postgres (stars.site_hours); an hourly prune drops hours once their clock hour has elapsed.
Delivery
A wholly public, read-only domain folded into the monolith: a slim SSR payload lists sites with their upcoming windows, the per-site history graph loads lazily, and the page is edge-cached.
Motivation

I got tired of different build commands for every project. I wanted one system that works the same everywhere: laptop, CI, and agents working in the cluster. Everything is vendored, so there is nothing to install beyond Bazel itself.

format
One command runs every formatter, regenerates manifests and lock files in parallel, and finishes in seconds when nothing changed.
Custom rulesets
rules_helm (lint, template, package, OCI-push charts, plus an ArgoCD application macro), rules_semgrep, rules_wrangler for Cloudflare Pages, and apko image tooling. Each ships a Gazelle extension that writes the BUILD files.
GitOps manifests
Helm charts render through Bazel into the source tree, so PR diffs show exactly what changes in the cluster before it deploys.
Multi-platform images
apko Alpine images from YAML with pinned lock files. One target builds arm64 and amd64 and pushes a multi-platform index.
BuildBuddy RBE
All builds run remotely with a shared content-addressed cache. Unchanged code never rebuilds.
Motivation

Semgrep on managed CI took 2+ minutes per diff scan and rule-registry fetches made results non-deterministic. I needed scans that run in seconds, produce identical results from identical inputs, and only re-run when something changed. Bazel's content-addressed cache gives all three, but Semgrep had no Bazel integration.

No Python
Extracts the semgrep-core OCaml binary from PyPI wheels and vendors it as an OCI artifact on GHCR, bypassing the Python wrapper and its startup tax.
Digest-pinned
Engine binaries and Pro rule packs are pinned to sha256 digests; a daily job updates digests and opens a PR. Same inputs, same results.
Three rule types
semgrep_test for sources, semgrep_manifest_test for Helm-rendered YAML, semgrep_target_test for transitive deps via aspect. Gazelle generates all of them.
Supply chain
SCA lockfile scanning with Pro reachability, auto-detected from @pip and @npm dependency prefixes. Zero config.
Results
Cached diff scans in 30 seconds, down from 2+ minutes. Cold cache: 4 minutes for all tests, images, and scans.