FlakeWarden

Agentic flaky-test triage and self-healing reviewer for UiPath Test Cloud.

UiPath AgentHack 2026 — Track 3: UiPath Test Cloud

Flaky tests are the most corrosive failure mode in CI. When a red build might be a real regression or just noise, engineers either burn time triaging every failure or, worse, start ignoring red builds, and a genuine regression ships. Google's continuous-testing study reported that ~16% of their tests had some level of flakiness and that ~84% of pass→fail transitions came from flaky tests (Memon et al., Taming Google-Scale Continuous Testing, ICSE-SEIP 2017). As an illustrative model: at a 5% flake rate a 2,000-test suite produces ~100 spurious failures per full run, and at an assumed 15–45 minutes of triage each that is tens of engineer-hours per cycle (the per-failure minutes are an assumption, not a measured figure).

FlakeWarden looks at a failing test's execution history and the surrounding evidence and answers the only question that matters: is this a real defect, a flaky test, or an environment problem? It then routes each failure to the right action, with a human in charge of every change.

The design follows one principle: deterministic where it must be exact, generative where the context is messy.

• A deterministic flake-scorer (auditable statistics over run history) handles the clear cases and never guesses.
• A grounded Agent Builder classifier (RAG over stack traces, DOM diffs, commit messages, and runner logs) reasons over only the ambiguous failures.
• UiPath Maestro orchestrates the two plus a Repair Agent, and every fix or quarantine passes through a mandatory Action Center human-review gate.

Run against a labeled corpus of 150 failures (corpus/failures.jsonl, on the offline rule-based classifier so the numbers reproduce with no API key):

The architecture forces every error into the safe direction: the deterministic scorer only auto-resolves a defect on a positive selector fingerprint, a flaky-looking history with any regression hint is double-checked by the classifier, and the classifier tie-breaks toward real defect when evidence is split. On this corpus that yields a measured 0% safety false-positive rate (no real regression hidden) and a 0% auto-heal-of-a-defect rate, enforced as a hard gate by eval/negative_control.py. The 12% noise (a flaky test escalated as a defect) wastes a little triage but hides nothing. These numbers are measured on a synthetic-but-adversarial corpus, not a production study — see docs/limitations.md. Reproduce with python eval/harness.py; full report in eval/report.md.

flowchart LR
    TC[UiPath Test Cloud<br/>Test Manager results] -->|execution history| SC
    subgraph MAESTRO[UiPath Maestro process]
        SC[Deterministic<br/>flake-scorer<br/><i>auditable math</i>]
        SC -->|confident: flaky / defect| ACT
        SC -->|ambiguous failure| CL[Grounded Classifier<br/>Agent Builder + RAG]
        CL --> ACT[Governed action router]
        ACT -->|flaky| HEAL[Repair Agent<br/>drafts selector fix]
    end
    HEAL --> AC[Action Center<br/>human review gate]
    ACT -->|real defect| AC
    AC -->|approve| ORCH[Orchestrator<br/>quarantine / promote baseline / open defect]

See ARCHITECTURE.md for the full data flow and the deterministic-vs-generative boundary. For how this differs from detection tools (Datadog, Develocity, Trunk) and healing tools (Healenium, Tricentis, UiPath Autopilot), and the defensible uniqueness claim, see docs/prior-art.md.

What is actually deployed and verified on the platform (not mocked):

• Triage Classifier agent — built in UiPath Agent Builder (Studio Web) with a grounded context, a structured output schema, an evaluation set, and an AI-Trust-Layer model. Verified live across all three classes: real_defect (0.97), flaky (0.86, with a proposed fix), environment (0.97), each with correct, evidence-cited reasoning. Published (v1.0.0) and deployed as an Orchestrator process (Solution.1.agent.Agent).
• Maestro BPMN orchestration (flakewarden-maestro/) — Start → agent call (Orchestrator.StartAgentJob) → verdict extraction → exclusive gateway on the label → three routed branches (flaky → human-gated heal, real_defect → escalate, environment → re-run). Authored entirely through the uip CLI and passing uip maestro bpmn validate.
• Built with a coding agent end to end — the agent scaffolding and the entire Maestro orchestration were produced by Claude Code driving the UiPath uip CLI (UiPath for Coding Agents): uip login, uip skills, uip tools install, uip agent deploy, uip maestro bpmn registry/init/validate. See docs/coding-agents.md.

Documented next step (honest): wiring the deployed agent's Orchestrator job-argument envelope and a serverless robot to the agent folder so the BPMN runs the agent end-to-end unattended, plus an Action Center action app for the in-Maestro human gate. The agent itself runs correctly today (verified in Agent Builder); these are the deployment-plumbing steps between "agent runs" and "BPMN runs the agent unattended."

This solution combines coding agents with low-code Agent Builder agents:

• Built with a coding agent. The entire solution (the deterministic scorer, the classifier interface, the eval harness, the Maestro orchestration, packaging and deployment) was scaffolded and iterated using Claude Code driving the UiPath uip CLI (UiPath for Coding Agents). See docs/coding-agents.md.
• Low-code Agent Builder agents. The Triage Classifier and Repair Agent are defined as Agent Builder agents (agents/) with grounded sources, an output schema, guardrails, and an evaluation set with a release gate.

Runs fully offline with a deterministic rule-based classifier; set ANTHROPIC_API_KEY to route the ambiguous band through a real Claude model.

python --version                      # 3.10+

# 1. Generate the labeled evaluation corpus (deterministic, seeded)
python corpus/generate_corpus.py

# 2. Reproduce the measured accuracy + false-positive rate
python eval/harness.py --report eval/report.md

# 3. Prove the safety invariants hold (gates CI)
python eval/negative_control.py

# 4. End-to-end on a freshly-run seeded test suite
python seeded_suite/run_history.py --runs 14
python -m flakewarden.cli triage seeded_suite/history.jsonl

Step-by-step wiring into your UiPath Labs tenant (Maestro process, Agent Builder agents, Test Manager connection, Action Center, uip CLI packaging and deploy) is in SETUP.md.

flakewarden/      deterministic scorer, grounded classifier, orchestration, CLI
agents/           Agent Builder definitions + the grounded classifier prompt
maestro/          Maestro process definition + step-to-component mapping
corpus/           labeled evaluation corpus + its deterministic generator
seeded_suite/     a seeded flaky test suite + history exporter (demo data source)
eval/             accuracy/false-positive harness + negative-control gate
tests/            unit tests for the scorer and orchestration invariants
docs/             architecture, demo script, presentation outline, coding-agents note

See docs/limitations.md. In short: the corpus is synthetic-but-adversarial (a solo builder cannot ship a real enterprise's CI history); production requires connecting the live Test Manager results API and a prospective accuracy study against a gold-standard labeled set. The architecture, governance gates, and eval methodology are production-shaped; the data is the gap.

MIT.