Skip to content

fix(aidd-dev): report missing runtime evidence in functional review #976

Closed

Description

@waewoo

Problem

aidd-dev:05-review performs functional review statically and explicitly does not execute the application or browser.

Its functional-review contract treats an acceptance criterion as satisfied when the diff provides evidence for it. This is sufficient for criteria that can be established from the changed files, but some criteria describe runtime behavior. For example, a criterion such as "submitting the form displays a success state" may look correctly implemented in the diff even though the affected workflow has never completed successfully.

When no prior runtime evidence is available, the current contract does not make that limitation explicit. The review can therefore appear to functionally verify the behavior even though it established only static support for it.

PR #950 strengthens runtime validation during implementation. Its maintainer confirmed that the corresponding review-side gap is intentionally separate and should be tracked in a dedicated issue.

Scope

  • Strengthen the existing 05-review functional-review contract for acceptance criteria that depend on runtime behavior.
  • Distinguish runtime-dependent criteria from criteria that can be assessed from the diff alone.
  • Consider sufficient runtime evidence when it already exists in the workflow's validation artifacts.
  • Make missing or insufficient runtime evidence explicit in the existing review output.
  • Preserve the static-review boundary and the existing workflow ownership.

Acceptance criteria

  • Functional review remains static and does not execute the application, browser, or runtime tests.
  • The review identifies acceptance criteria whose result depends on runtime behavior.
  • Criteria that can be assessed from the diff alone continue to follow the existing static-review contract.
  • Sufficient runtime evidence can be referenced when it is already available through the applicable validation workflow.
  • When a runtime-dependent criterion has no sufficient evidence, the review explicitly reports that gap.
  • The report identifies the affected criterion and validation limitation instead of silently treating static evidence as runtime verification.
  • A statically complete diff cannot be presented as functionally verified when the required runtime evidence is absent.
  • The review introduces neither a parallel execution workflow nor a new evidence store inside 05-review.

Prior art in this repo

  • plugins/aidd-dev/skills/05-review/SKILL.md#L10-L18 assigns functional behavior review to 05-review and defines it as read-only.
  • plugins/aidd-dev/skills/05-review/actions/02-review-functional.md#L15-L19 defines the static boundary, diff-based evidence check, verification summary, and functional findings.
  • #858 strengthens the same contract around snapshot inputs, phase scope, status tags, and verified-percentage calculation. It does not address runtime-evidence reporting.
  • #928 requires 05-review to report missing or unverified contributor-workflow validation. It does not cover runtime-dependent behavior.
  • #955 validates feature runtime prerequisites during planning. It does not determine whether delivered behavior has runtime evidence during review.
  • PR #950 requires the real affected workflow and observable evidence during implementation. The maintainer explicitly requested a dedicated issue for the remaining review boundary.
  • #954 was closed as not planned after maintainers redirected general repository-readiness concerns toward existing workflows. This issue follows that decision by strengthening 05-review directly.

Out of scope

  • Executing the application, browser, or runtime tests from 05-review.
  • Replacing the implementation-side validation contract in PR #950.
  • Creating or expanding the independent Acceptance QA capability covered by #909, #918, and #919.
  • Validating runtime prerequisites during planning, covered by #955.
  • Discovering contributor-workflow commands, covered by #928.
  • Addressing snapshot isolation and verified-percentage calculation, covered by #858.
  • Defining a new universal runtime-evidence schema or storage mechanism.

Activity

  1. blafourcade commented on Oct 11, 2026

    @blafourcade
    Contributor

    Fixed by #995.

    The review still only reads the code. A criterion that only running the app can prove (a banner shown, a timing, an outside service's answer) is no longer checked from the code. It is listed as unverified at runtime, with what a run would show, and it never blocks the verdict.

    A test run recorded beside the plan on the exact reviewed commit decides it: passing checks it, failing leaves it unchecked. A run on another commit counts for nothing.

    Running that test automatically before the PR is tracked in #996.

  2. moved this from Ideation to Done in AIDD Roadmapon Oct 11, 2026
  3. moved this from Done to In review in AIDD Roadmapon Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions