Repository navigation
fix(aidd-dev): report missing runtime evidence in functional review #976
Description
Problem
aidd-dev:05-review performs functional review statically and explicitly does not execute the application or browser.
Its functional-review contract treats an acceptance criterion as satisfied when the diff provides evidence for it. This is sufficient for criteria that can be established from the changed files, but some criteria describe runtime behavior. For example, a criterion such as "submitting the form displays a success state" may look correctly implemented in the diff even though the affected workflow has never completed successfully.
When no prior runtime evidence is available, the current contract does not make that limitation explicit. The review can therefore appear to functionally verify the behavior even though it established only static support for it.
PR #950 strengthens runtime validation during implementation. Its maintainer confirmed that the corresponding review-side gap is intentionally separate and should be tracked in a dedicated issue.
Scope
- Strengthen the existing
05-reviewfunctional-review contract for acceptance criteria that depend on runtime behavior. - Distinguish runtime-dependent criteria from criteria that can be assessed from the diff alone.
- Consider sufficient runtime evidence when it already exists in the workflow's validation artifacts.
- Make missing or insufficient runtime evidence explicit in the existing review output.
- Preserve the static-review boundary and the existing workflow ownership.
Acceptance criteria
- Functional review remains static and does not execute the application, browser, or runtime tests.
- The review identifies acceptance criteria whose result depends on runtime behavior.
- Criteria that can be assessed from the diff alone continue to follow the existing static-review contract.
- Sufficient runtime evidence can be referenced when it is already available through the applicable validation workflow.
- When a runtime-dependent criterion has no sufficient evidence, the review explicitly reports that gap.
- The report identifies the affected criterion and validation limitation instead of silently treating static evidence as runtime verification.
- A statically complete diff cannot be presented as functionally verified when the required runtime evidence is absent.
- The review introduces neither a parallel execution workflow nor a new evidence store inside
05-review.
Prior art in this repo
plugins/aidd-dev/skills/05-review/SKILL.md#L10-L18assigns functional behavior review to05-reviewand defines it as read-only.plugins/aidd-dev/skills/05-review/actions/02-review-functional.md#L15-L19defines the static boundary, diff-based evidence check, verification summary, and functional findings.- #858 strengthens the same contract around snapshot inputs, phase scope, status tags, and verified-percentage calculation. It does not address runtime-evidence reporting.
- #928 requires
05-reviewto report missing or unverified contributor-workflow validation. It does not cover runtime-dependent behavior. - #955 validates feature runtime prerequisites during planning. It does not determine whether delivered behavior has runtime evidence during review.
- PR #950 requires the real affected workflow and observable evidence during implementation. The maintainer explicitly requested a dedicated issue for the remaining review boundary.
- #954 was closed as
not plannedafter maintainers redirected general repository-readiness concerns toward existing workflows. This issue follows that decision by strengthening05-reviewdirectly.
Out of scope
- Executing the application, browser, or runtime tests from
05-review. - Replacing the implementation-side validation contract in PR #950.
- Creating or expanding the independent Acceptance QA capability covered by #909, #918, and #919.
- Validating runtime prerequisites during planning, covered by #955.
- Discovering contributor-workflow commands, covered by #928.
- Addressing snapshot isolation and verified-percentage calculation, covered by #858.
- Defining a new universal runtime-evidence schema or storage mechanism.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Projects
- StatusShow more project fieldsIn review
Activity
fix(aidd-dev): a review never checks a runtime criterion from the code (
blafourcade commentedon Oct 11, 2026
Fixed by #995.
The review still only reads the code. A criterion that only running the app can prove (a banner shown, a timing, an outside service's answer) is no longer checked from the code. It is listed as unverified at runtime, with what a run would show, and it never blocks the verdict.
A test run recorded beside the plan on the exact reviewed commit decides it: passing checks it, failing leaves it unchecked. A run on another commit counts for nothing.
Running that test automatically before the PR is tracked in #996.