Skip to content

feat(cli): add zigflow test one-shot workflow runner - #595

Open
arthurepc wants to merge 1 commit into
zigflow:mainfrom
arthurepc:feat/cli-zigflow-test-v1
Open

arthurepc wants to merge 1 commit into
zigflow:mainfrom
arthurepc:feat/cli-zigflow-test-v1

Conversation

@arthurepc

@arthurepc arthurepc commented Sep 21, 2026 •

Copy link
Copy Markdown

Summary

  • Add zigflow test to validate a workflow file, run it once against Temporal with JSON input, print the result, and exit with a non-zero code on failure (#304).
  • Introduce pkg/testrunner and cmd/run/session.go for a short-lived in-process worker on the fixed zigflow-test task queue (separate from document.taskQueue used by zigflow run).
  • Serialise concurrent test runs per Temporal address and namespace with a file lock so workers do not steal tasks for workflows they have not registered.
  • On failure, print structured output when available (for example OWS errors from raise) plus an error line and Temporal UI inspect link.
  • Document usage in the CLI and testing guides; add e2e coverage and manual concurrency fixtures under examples/test-concurrency.

Design decisions (discovered while implementing this feature, thus not present in #304)

These choices came out of implementation and e2e testing, not from the original issue text alone.

Fixed zigflow-test task queue (v1)

zigflow test always registers workers and starts workflows on zigflow-test, ignoring document.taskQueue in the YAML for this command. That keeps test runs off the same queue as long-lived zigflow run workers, so a dev server with zigflow run up does not need to be stopped to try a workflow once. v1 prioritises predictable local/CI behaviour over honouring the file’s queue name; overriding via registrationTaskQueue in runOptions is the internal hook if we extend this later.

Cross-process file lock on zigflow-test

Multiple zigflow test processes can connect to the same Temporal server and namespace. Each process registers only its own workflow type on the shared zigflow-test queue. Temporal does not guarantee that a poll goes to a worker that registered that workflow, so concurrent tests can fail with “workflow type not registered” style errors.

The lock is keyed by Temporal address + namespace (file under the user cache dir). It serialises who may poll zigflow-test at a time for that pair. Different namespaces or servers do not block each other. This is process-level coordination, not an in-memory mutex inside one binary.

Reuse zigflow run worker startup via StartSession

StartSession builds runOptions and calls the same prepareRegistrations, initTemporalClient, and startInitialWorkers path as zigflow run, without watch mode or signal handling. That avoids duplicating worker registration, codec, and external storage wiring. Test-specific behaviour is limited to flags on runOptions (below).

skipScheduleUpdates for test sessions

Schedule reconciliation (UpdateSchedules) is appropriate for long-lived zigflow run deployments. A one-shot test should not create or mutate Temporal schedules for the workflow under test. Test sessions set skipScheduleUpdates: true.

skipClientMetrics and Prometheus (Temporal client metrics only)

zigflow run enables Temporal SDK client metrics via temporal.WithPrometheusMetrics(..., nil). A nil registry uses Prometheus’s process-wide default registerer. Each call creates a new reporter that registers the same metric names again.

That is fine for one initTemporalClient per process (zigflow run). It breaks when several short-lived sessions run in the same process: central e2e (go test in one binary), multiple StartSession calls, or unit tests that invoke initTemporalClient without skipping metrics. The second registration collides on the default registry (helpers/Tally behaviour; see zigflow/helpers Prometheus tests).

Test sessions therefore set skipClientMetrics: true. Workers in Zigflow do not currently set worker.Options.MetricsHandler, so this is not a client-vs-worker double registration inside one session; it is repeated client setup in one process.

We did not adopt a process-wide shared PrometheusHandler or per-session prom.NewRegistry() in this PR: zigflow test does not expose a stable metrics scrape target, and sharing would need explicit refcount/Close() lifecycle. Skipping client metrics for test is the smallest correct fix. initTemporalClient unit tests also set skipClientMetrics: true so the cmd/run package’s go test does not hit the same collision.

zigflow run behaviour is unchanged (metrics still enabled once per process).

No “non-local Temporal” warning

Earlier feedback on #304: do not warn when the Temporal address does not look “local”. Cloud, in-cluster, and custom endpoints are all valid for zigflow test; guessing locality is misleading.

Failure output and e2e assertions

Failed runs print workflow output when the SDK returns it (useful for raise and OWS-shaped errors), plus the error and a Temporal UI history link. E2e assertions for the failure path were relaxed to match ApplicationError shape rather than assuming a specific string in the CLI error line alone.

Test plan

  • go test ./pkg/testrunner/... ./cmd/...
  • go test ./...
  • task e2e (central e2e including TestE2EZigflowTest and TestE2EZigflowTestFailed)
  • Manual: temporal server start-dev, go run . test examples/hello-world/workflow.yaml --input examples/hello-world/test-input.json
  • Manual: failure path with go run . test examples/raise/workflow.yaml
  • Manual: concurrent slow/fast fixtures under examples/test-concurrency/ to verify lock serialisation

@arthurepc
arthurepc force-pushed the feat/cli-zigflow-test-v1 branch 2 times, most recently from 1847e74 to d6805b7 Compare September 22, 2026 14:38
@arthurepc
arthurepc marked this pull request as ready for review September 22, 2026 14:50
@mrsimonemms mrsimonemms added the ai-review Trigger an AI code review for this PR label Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Codex PR Review

Automated review note: Zigflow uses AI-assisted PR review tooling to help identify potential bugs, regressions and edge cases.
These comments are intended as supplemental review feedback only and may occasionally be incorrect or overly cautious.
Please treat them as additional input for discussion rather than authoritative project direction.

  • High: Test runs can execute the wrong workflow definition. cmd/run/session.go#L35-L38 assigns every session the same queue, but the file lock only coordinates processes sharing a user cache directory. Concurrent CI containers or machines targeting the same Temporal namespace can consume each other’s tasks. Different definitions with the same workflow type can produce a false pass. Use a unique queue per session and test concurrent runs with differing definitions.

  • Medium: Windows lock acquisition can hang beyond the timeout. pkg/testrunner/lock_windows.go#L38-L42 calls LockFileEx without LOCKFILE_FAIL_IMMEDIATELY on a synchronous os.OpenFile handle. Under contention, the call blocks before the context polling loop can observe cancellation. Add LOCKFILE_FAIL_IMMEDIATELY and treat ERROR_LOCK_VIOLATION as contention. Verify the cancellation test on Windows.

Go checks could not run because the read-only filesystem prevented module-cache creation.

@arthurepc
arthurepc force-pushed the feat/cli-zigflow-test-v1 branch from b5e92f0 to 12e9027 Compare September 22, 2026 18:01
@arthurepc

arthurepc commented Sep 22, 2026 •

Copy link
Copy Markdown
Author

@mrsimonemms I’d like your opinion on the third bullet in Codex’s review.

The fixed zigflow-test queue is intentional and documented in this PR’s description. For v1 I see two realistic options, not a third “distributed lock” path for now:

  1. Keep the current design: one well-known test queue, serialised on each machine with the file lock (separate from zigflow run on document.taskQueue). That gives a clear queue boundary and avoids multiple processes polling the same queue without each registering the same workflow types. The known gap is that the lock is per host, so parallel jobs on different runners sharing the same Temporal namespace are not coordinated.

  2. Per-run task queue (as Codex suggests): each zigflow test uses its own queue name so concurrent runs do not steal each other’s tasks, at the cost of abandoning a single fixed test queue and allowing parallel test workers.

I am not keen to add a third-party dependency (Redis, etc.) purely for cluster-wide locking in the CLI. That feels out of scope for Zigflow as an engine/CLI, even if a distributed lock would work in theory. My bias is option 1 for v1, with docs/CI guidance where needed (e.g. namespace per job), unless you think per-run queues are clearly better. Does that align with how you expect zigflow test to be used?

@mrsimonemms mrsimonemms added ai-review Trigger an AI code review for this PR and removed ai-review Trigger an AI code review for this PR labels Sep 22, 2026
@arthurepc
arthurepc force-pushed the feat/cli-zigflow-test-v1 branch from 12f2451 to a339452 Compare September 22, 2026 21:04
@arthurepc

Copy link
Copy Markdown
Author

Actually, I’m now leaning towards the per-run task queue. The part I’m still uneasy about is how to bound concurrency so we don’t spawn one active worker per concurrent zigflow test invocation without an explicit limit.

@arthurepc
arthurepc force-pushed the feat/cli-zigflow-test-v1 branch from dee9f3e to b636110 Compare September 22, 2026 21:28
@mrsimonemms mrsimonemms added ai-review Trigger an AI code review for this PR and removed ai-review Trigger an AI code review for this PR labels Sep 22, 2026
@mrsimonemms

Copy link
Copy Markdown
Collaborator

Yeah, it's a difficult one. I wonder if this is one of those things where we should open a discussion on Slack and get a wider response from users? Also, I'm at an offsite in the US this week so I'm not going to be as responsive this week - I need to have a bit of a think about your questions.

One thing - can you tidy up the commit history a bit please? Zigflow uses rebase so these commits will be brought into main as-is.

@arthurepc

Copy link
Copy Markdown
Author

I like the idea of opening a thread on slack to gather opinions on that. I'll send a message there so we can discuss with users.

Regarding this branch's commit history, thank you for heads up! I'll do that.

Introduce zigflow test and pkg/testrunner for validate-run-print flows on
the zigflow-test queue, with per-machine file locking, failure output,
docs, and e2e coverage.

Test sessions use StartSession with skipScheduleUpdates and
skipClientMetrics, terminate timed-out executions on the server,
exclusive Windows file locks (overlapped lock file handle), and
RunBytes for in-memory definitions.

Signed-off-by: Arthur Costa <[email protected]>
@arthurepc
arthurepc force-pushed the feat/cli-zigflow-test-v1 branch 4 times, most recently from 3236a3b to 346a8c9 Compare September 23, 2026 14:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-review Trigger an AI code review for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants