Deep dives into the systems on a bare-metal K3s cluster at home, deployed end-to-end with GitOps. Each covers why it exists and how it's built.
The first version ran unattended Claude agents in sandboxed Kubernetes pods dispatched over NATS. When Claude's terms of service changed around unattended agent use, that dispatch model no longer held. Rather than retire the idea, I split the execution substrate apart from the model choice and rebuilt the substrate on Firecracker: every agent request gets its own hardware-isolated microVM instead of a shared pod, paused and snapshotted when idle so it costs nothing, and restored in tens of milliseconds when woken. The durable pieces from v1 carried over (on-cluster inference, the MCP gateway, the knowledge-graph surface); the sandbox underneath them got much stronger.
fc-invoke proved the substrate but hardcoded one workload shape: a stateless invoke. EmberVM is the successor, built by forking the node daemon and putting a BEAM control plane in front of it. Semgrep scans already run on it, a public image renderer serves warm from it, and fc-invoke is frozen with the goose agent as its last tenant. The design goal: the control plane owns placement and policy but stays off every serving hit path.
I wanted a make-me-a-thing button: type a prompt in Discord, get back a working hosted mini-app. The hard part is doing it both safely and fast. The agent writes and runs untrusted code, so it has to be strongly isolated, but a fresh microVM per request must not feel slow. Goosecracker is the proof that per-request hardware isolation and sub-second cold start are not a trade-off.
Notes are only useful if they come back at the right moment. The knowledge pipeline turns markdown into a queryable graph: an on-cluster LLM decomposes each note into atomic facts, critiques its own extraction, and stores embeddings for semantic recall.
Operating a governed data platform should scale sub-linearly with data and org size: a new dataset, domain, or transform should add no new system to run. Loom is a collaborative bet on that premise, with governance treated as a safety property (STPA hazard analysis, where unsafe means a governance violation, not a crash).
HuggingFace models are huge and slow to download. I wanted to reference a model in a pod spec the same way you reference a container image, and have it just work. The operator caches models in an OCI registry and streams them to pods without touching disk.
volumes:
- name: model
image:
reference: hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUFKubernetes operators are state machines, but we write them as imperative reconciliation loops. Every operator I wrote had the same bugs: invalid state transitions, forgotten error handling, missing metrics. Sextant defines the state machine declaratively and generates the boilerplate.
states:
- name: Pending
initial: true
- name: Creating
fields:
requestID: string
- name: Ready
terminal: true
transitions:
- from: Pending
to: Creating
action: StartCreation
params:
- requestID: stringEvery new service meant clicking through the Cloudflare dashboard: create a DNS record, create a Zero Trust application, update the tunnel config. I wanted to annotate a Deployment and have everything provisioned automatically.
metadata:
annotations:
cloudflare.ingress.hostname: myapp.jomcgi.dev
cloudflare.zero-trust.policy: joe-onlyI wanted an easy way to share trips with friends and family. A GoPro on the dash captures photos automatically, and my homelab turns them into a map they can follow day by day, or replay later. Works for anything with GPS-tagged photos.
Living near the coast, I wanted to see what ships are passing by in real time. AIS data is publicly broadcast by vessels, but there is no simple way to visualize it locally. This pipeline streams AIS data through my cluster onto a map.
Finding a good stargazing night means combining how dark a site is with whether the sky will actually be clear once it gets dark. Stars pairs a light-pollution grid of road-accessible dark sites with rolling weather forecasts and surfaces the upcoming hours that are both dark and clear.
I got tired of different build commands for every project. I wanted one system that works the same everywhere: laptop, CI, and agents working in the cluster. Everything is vendored, so there is nothing to install beyond Bazel itself.
Semgrep on managed CI took 2+ minutes per diff scan and rule-registry fetches made results non-deterministic. I needed scans that run in seconds, produce identical results from identical inputs, and only re-run when something changed. Bazel's content-addressed cache gives all three, but Semgrep had no Bazel integration.