Dailies

A camera assistant wears smart glasses through a shoot and walks off set with the paperwork already written.

Demo: 2 min 31 s on YouTube

Live: dailies-564641829203.us-east1.run.app, and it runs from any webcam, so it can be tried without the glasses.

Write-up: how I built it, and what the hardware forced

Built for Agentic Cinema, ClickHouse track.

Every shot in it is the deployed service running on real footage. The catch is at 0:49.

Every arrow in that diagram runs in the deployed service. Nothing in it is planned work.

A script supervisor stands just off set with a clipboard, and between every take writes down where the mug sat, whether the jacket was buttoned, which hand held the phone. That is what lets takes cut together. It is the one job on a shoot still done entirely on paper, and when a line gets missed nobody finds out until the edit.

By then it is not an edit problem. It is a pickup day. A published survey of 40 US production markets prices a fixed 6-person, 10-hour shoot day at $1,440 in the cheapest market, Santa Fe, and $3,020 in the most expensive, Boston. That page is on screen in the demo video at 0:23, at the moment the figure is spoken, because a number is worth what its source is.

Dailies watches a take for 1.6 cents, measured on real footage through the same Vertex AI path the deployment runs, priced at Google's published rates. A whole 40-take day costs about 63 cents, or 0.04% of the day it is guarding.

Neither number is mine. Google meters the tokens and sets the price; the industry sets the day rate. The workings are in COST.md, reproducible with python pipeline/measure.py. What is deliberately not claimed anywhere here is a count of reshoots prevented, because that would be my own code marking its own homework.

Every other tool in this space starts from the screenplay PDF. This one starts from the set.

The glasses capture the day. Gemini reads every take and extracts what is actually in frame: props, wardrobe, positions, what moved. That becomes structured records in ClickHouse. At wrap, instead of typing up a facing page, the crew asks:

which takes is the smartphone missing from?

and the agent writes its own SQL, runs it through the ClickHouse MCP server, and answers.

The artifact is the crew member's paperwork. Catching a continuity break is a consequence of having the records, not the identity of the product.

The question box is open on a public URL, and what it does with your sentence is let a model compose SQL and run it against a live cluster. That deserves an answer rather than a hope, so it has three independent layers and each was tested by trying to break it:

The third layer is the one that matters, because it holds even if the first two fail: the credential itself cannot write. dailies_ro holds GRANT SELECT ON default.* and readonly = 2, and it is handed only to the MCP subprocess. The service's own credential still writes, because takes, observations and every row in agent_runs have to be inserted by something. The flags are set explicitly rather than left to the library's defaults, since a safety property resting on someone else's current default is one dependency bump from not holding.

Check it from outside rather than believing this paragraph. /api/capabilities reports "sql_credential": "readonly", and it reads that from the same function that builds the subprocess environment, so the two cannot disagree.

This layer was missing for part of the build and the writeup described it anyway. The service ran with CLICKHOUSE_USER=default, the admin account, and the gap only surfaced when a DROP TABLE observations aimed at what was believed to be a readonly credential removed the table for real. It was rebuilt from out/ in a couple of minutes, which is the one genuinely good thing about the incident: the JSON state is the source of truth and ClickHouse is a projection of it, so nothing was lost. tests/test_sql_credential.py now fails if the admin credential can reach the MCP subprocess again.

Every model call this project makes also lands in ClickHouse, as a row in agent_runs: the operation, which model answered, latency, outcome, the entities it flagged, the SQL it chose, and whether a fallback fired. The rolling check writes the take it was checking against, so each verdict joins back to the shoot it belongs to.

So the agent's own behaviour is queryable through the same question box as the footage. "How fast is the rolling check, and what does it flag most?" is a SELECT over the same cluster, with no separate dashboard and no second tool. The four queries in pipeline/telemetry.py (live_latency_percentiles, divergence_rate, most_flagged_entities, fallback_usage) are the ones I found myself running by hand while tuning it, which is how they earned a place.

This is the shape ClickHouse's own AI team describes for agent observability, a trace kept inside the analytical store rather than beside it. Here it was not added for the writeup: once every read path was on ClickHouse, writing the run record there was simply the obvious place to put it.

/live is the same instrument in its operating mode. Pick a reference take, press Roll, and every few seconds it checks what the camera sees against what that reference recorded. The verdict is one word, large enough to read from across the room, because the operator is watching the scene and not this screen.

The report view answers what went wrong today. This answers what is wrong right now, which is the question the job actually asks. Finding the mug on the wrong side at wrap has documented a reshoot. Finding it twenty seconds into the take has prevented one.

A judge with no hardware can run all of this from a laptop webcam, and should. But the glasses are the instrument this is built for, not an interchangeable frame source, and the difference is the whole reason the rolling check works.

Continuity is an attention problem before it is a vision problem. A script supervisor does not scan the room uniformly; they look at the things that can betray a cut, and they look at them between takes, while walking, with a clipboard already in both hands. A tripod camera sees the set. Glasses see what the person responsible for continuity actually checked, which is a different and much better-aimed signal.

Two consequences the code had to be built around, both learned from real footage rather than assumed:

• Depth stops being a property of the object. From a head-mounted camera, a mug moving from midground to foreground usually means the wearer leaned in. Every per-object depth delta this project ever produced was a false positive, so depth now only survives as evidence that several objects moved together, which is a camera move and is reported once instead of as N continuity breaks. A tripod would never have forced that distinction.
• Framing is never stable, so position must be relative. The frame edges move constantly with the wearer's head, which is why position is judged against the other objects in shot rather than against the crop.

And the output has to be audio. The wearer is watching the scene, not a screen, with their hands full. These glasses have no display, so a verdict spoken into the ear is not a consolation prize for missing hardware; it is the only delivery that works on a set. That is also why the rolling check has to answer inside a take rather than at wrap.

Latency: 4.4s median, measured against the deployed service on seven consecutive checks, alongside the recall it is worth: 3 of 3 on the mirrored frame, 0 of 4 false alarms on the control. A take runs well over thirty seconds, so several checks land while the camera is still rolling.

It was 16.4s before a fix and 1.4s after a mistake, and the mistake is the more useful story:

The 9.3s fix was real and stands. The service's own timing said the delay was inside the model call, which pointed at Vertex having slowed down; it had not, and the identical request issued directly still answered in four seconds. make_client() ran per request, and on Vertex that performs credential discovery, which on Cloud Run is a metadata-server round trip for a service-account token. Nothing had got slower except how often the service was asking who it was.

The 1.4s was not a fix. Thinking was disabled on the strength of a control-and-mirror test that recorded 3 of 3 caught, and that test rebuilt the prompt by hand instead of importing the one the code sends. The two texts differ by a single closing sentence. Against the real prompt the same setting caught 0 of 3, and said HOLDS to a room where every object had swapped sides. It survived a day because the negative control passed the whole time, and a detector that never fires looks exactly like one that works.

build_prompt() now has the only copy of that text so a harness has to import it, and tests/test_live_config.py fails the build if thinking is disabled again. 4.4s that answers beats 1.4s that does not.

One caveat that matters for recording a demo: these are warm-instance figures. The first call after a cold start once took 182 seconds while the container built its first Vertex credential. Send one throwaway request before rolling.

Checks run sequentially rather than on a fixed timer; polling faster than the answer arrives just stacks requests until the queue collapses.

Two findings, both from measurement rather than reasoning, and the second only exists because the first was so expensive.

Flash-Lite is disqualified on recall, not speed. It benchmarked about ten times faster, 979ms against 5-15s. Then it was handed a horizontally mirrored frame, every object on the desk on the wrong side, and reported:

"Laptop, monitor, coffee mug, and microphone are in their expected positions."

The speed was not a tradeoff, it was an artefact of not looking. A detector with no recall is a green light wired to nothing, and on a set that is worse than no tool because someone will trust it.

Gemini 3.5 Flash beats 3.6 Flash here, three times faster at identical recall. Control frame (must stay silent) and mirrored frame (must flag), three runs each side:

The newer model is slower without being better at this task, and frame size barely moves the number, so the lever is the model rather than the pixels. Note 3.6 at 384px being slower than at 512px: that is server-side variance, and a good argument for measuring a configuration rather than reasoning about it.

Confirmed at twelve runs after the switch: 0 false alarms, 6 of 6 caught, 4.2s median. That 4.2s was the figure with thinking still on and a client built per request; both were fixed later and the check now runs at 4.4s median. The model comparison above is unaffected, since both models were measured under the same conditions as each other.

Flash-Lite benchmarked about ten times faster on this exact call, 979ms against 5-15s, and was chosen on that basis. Then it was handed a horizontally mirrored frame, every object on the desk on the wrong side, and reported:

"Laptop, monitor, coffee mug, and microphone are in their expected positions."

Flash, on the same frame, flagged the mug, microphone, mouse and mousepad at 0.95 each and named the swap. The speed was not a tradeoff, it was an artefact of not looking. A detector with no recall is not a fast detector, it is a green light wired to nothing, and on a set that is worse than no tool because someone will trust it.

Verified after the change: control frame silent on 4 of 4 runs, mirrored frame flagged on 4 of 4 with the correct objects.

Five real 3-minute takes of one desk, shot on Ray-Ban Meta glasses. Ground truth set by the operator: nothing was deliberately moved between takes.

A naive extract-and-diff reported 8 differences, 7 of them fake. Four passes, each added because of a defect measured on this footage rather than anticipated, bring that to 6 findings with no false alarms:

The last row adds findings rather than removing them, which is the point of it. Reconciling, presence and camera-shift all exist to delete false alarms; giving the model a controlled state vocabulary instead of free text let it report two things it had been seeing all along and had no way to say. Both are real: the mouse is held in take_001 and resting in the other three, and the phone is in both hands in take_004 and untouched in take_003.

What survives is real, and nothing was told to look for any of it:

[camera move] take_005: 3 objects shifted depth together
              foreground -> midground for computer mouse, desk, mousepad
              the whole frame moved, so this is framing, not continuity
[gone]        computer mouse   absent from take_004
[gone]        smartphone       absent from take_005
[off mark]    smartphone       right in takes 1-2, center in takes 3-4
[state]       computer mouse   held=right_hand in take_001, resting in 002/003/005
[state]       smartphone       held=both_hands in take_004, not held in take_003

The mouse is genuinely gone from take 4 because both hands were holding a phone. The phone is genuinely gone from take 5. And it genuinely moved from the desk to being held up between takes 2 and 3.

The two [state] lines are the same story told from the other side: the reason the mouse vanishes in take_004 is the reason the phone is in both hands in take_004. A facing page that records only presence and position cannot say that, which is exactly why the state vocabulary was added.

glasses ──► take.mp4 ──► ingest ──► frames + manifest
                                         │
                                         ▼
                                   extract  (Gemini, schema-constrained)
                                         │
                                         ▼
                                  reconcile  (one vocabulary across takes)
                                         │
                                         ▼
                                   presence  (re-examine what a pass missed)
                                         │
                                         ▼
                                   compare  ──► ClickHouse ──► ask (agent + MCP)

No query for any demo question is hardcoded. Asked which takes the phone is missing from, it ran four statements it chose itself. Asked to compare two takes it first wrote take_id IN (3, 4), got nothing back, and retried with ('take_003', 'take_004'). That self-correction only works because a failed query goes back to the model rather than killing the request.

Every statement it runs is returned with the answer. A continuity note nobody can check is not evidence.

https://dailies-564641829203.us-east1.run.app: no setup, no glasses, no key. Add ?theme=light for the light palette.

git clone https://github.com/JonathanSolvesProblems/dailies.git
cd dailies
python -m venv .venv && .venv/bin/pip install -r requirements.txt   # Windows: .venv\Scripts\pip
DAILIES_OUT=samples .venv/bin/python -m uvicorn web.app:app --port 8077

Open http://127.0.0.1:8077. That works with no credentials at all, against the real scene state in samples/. The question box appears only if a deployment can answer it.

To enable the agent, copy .env.example to .env and fill in a Gemini key and ClickHouse Cloud connection details, then:

.venv/bin/python pipeline/load_clickhouse.py --out samples
.venv/bin/python pipeline/ask.py "which takes is the smartphone missing from?" --scene scene_a

Needs ffmpeg and ffprobe on PATH.

python pipeline/ingest.py take_001.mp4 --out out/myscene --take-id take_001
python pipeline/extract.py out/myscene/take_001.json --scene-context "INT. KITCHEN - DAY"
python pipeline/reconcile.py out/myscene
python pipeline/presence.py out/myscene/reconciled
python pipeline/compare.py out/myscene/reconciled

• Gemini 3.6 Flash via google-genai, falling back a generation when overloaded. Flash rather than Pro because this is a high-volume vision call on every take of a shoot day. Reached through Vertex AI (location="global"), so the deployed service authenticates as its own Cloud Run service account and no API key exists in the deployment at all. The region is measured, not chosen: us-east1 does not serve 3.6 Flash and us-central1 404s on both 3.5 and 3.6, so only global serves every model this uses.
• ClickHouse Cloud via the official mcp-clickhouse MCP server. One denormalized observations table, ORDER BY (scene_id, entity, take_id), because every question this asks is a filter and group-by over that.
• FastAPI on Cloud Run.
• Ray-Ban Meta Gen 2 for capture. A DAT stream was measured holding 25 minutes and 45,002 frames with no cap, so the 3-minute limit belongs to the stock camera app rather than the platform.

The rules ask for runtime use, imported and called in code rather than named in a readme, so here is the shortest path to checking that. Every line below is on a request path, not in a script that has to be run by hand.

Or check the running service instead of the code:

curl https://dailies-564641829203.us-east1.run.app/api/health
{"status":"ok","backend":"clickhouse","scenes":1,"takes":5,"observations":38}

curl https://dailies-564641829203.us-east1.run.app/api/capabilities
{"ask":true,"model":"vertex:warden-agent-supervisor:global","store":"clickhouse","sql_credential":"readonly"}

backend is the store that answered, model is the live Vertex project and location, and sql_credential is which cluster user the SQL agent connects as. All three are read from the code that does the work rather than from configuration, so a deployment that has quietly lost one of them says so.

• The demo scene is one desk, five takes. The findings above are real and the ground truth is real, but this has not been run on a professional set.
• state is descriptive, not comparable. Across five takes of an identical scene the same mug was described "upright", "on napkin" and "placed on table". The diff runs only on normalized fields; a field that cannot be compared reliably is worse than no field, because a false continuity flag costs a crew real time chasing nothing.
• Depth is a weak axis from POV capture. It moves with the wearer's head, which is why camera-shift detection exists rather than reporting three breaks.
• The glasses can speak but cannot listen. The obvious next feature is the operator saying "mark take 3" instead of touching a phone, and it is not buildable on DAT 0.9.0. The public Stream interface exposes videoStream, capturePhoto, state, errorStream and start/stop, and no audio input. AudioFrame, AudioDecoder and MetaWearablesDATAudioEventListener do ship inside mwdat-camera, so the capability plainly exists, but reaching it would mean binding to internal classes that any SDK release can move. Output is unaffected: the glasses register as an ordinary Bluetooth audio device, so the spoken verdict needs no SDK support at all.
• Frames come from capturePhoto, not the video stream. getVideoStream() is available and would be the deeper use of the hardware, but it delivers encoded frames needing a MediaCodec pipeline, and a check every few seconds does not need 24fps. The photo path returns a Bitmap and is the honest fit for the duty cycle.
• The hosted question box takes several seconds. Cold start plus an MCP subprocess per request plus two round trips to Gemini.
• Frames are not distributed. samples/ carries the scene state, not the footage, which is private video of a home.

No source footage, no extracted frames. The repo is public and the footage is not.