feat: add submission-level generated readouts by cedricvidal · Pull Request #1495 · microsoft/scope

Cedric Vidal (cedricvidal) · 2026-10-13T14:48:55Z

Summary

Adds the second layer of the submission report stack on top of #1494 for #1206.

introduces the dedicated submission-scoped pp-readout worker with exact-vector evidence loading, stale-attempt checks, schema-validated structured output, and Markdown derived from that structure;
stores every successful generation as a new template-less report plus reports/{reportId}/structured.json, while the exact submission-run pointer selects only the current result;
adds submission report filtering/history, standard report-by-ID and proxied structured-download access, generic Portal report UX, and CLI parity;
adds run- and submission-scoped report templates; submission templates trigger automatically and idempotently after the exact current attempt's pp-readout evidence completes, then generate through the normal report worker with exact-attempt evidence tools;
refines Submission Detail with attempt-scoped Reports/Runs/Processing tabs, browser-style selected-attempt report-type tabs connected to the report panel, stable attempt + report deep links with browser-history-safe navigation, a shared member-runs table, dynamic dependency progress, exact processing logs, and explicit Submission navigation from Runs views;
removes the carried-forward duplicate run-level Taxonomy worker/API/UI in favor of pp-observations evidence on Run Detail and synthesized reports on Submission Detail.

This PR supersedes #1053. It intentionally does not close or modify #1053.

Demos

The latest replacement five-run walkthrough was recorded headlessly from the
real local stack after the submission-template trigger and deep-link follow-up.
It shows creating a submission-scoped template through the Portal, the automatic
exact-attempt trigger, the generated template report as a sibling browser-style
tab, and direct navigation to that report tab. The placeholders below identify
the verified replacement binaries to upload; they are intentionally not claims
that the binaries have already been uploaded. Prior walkthrough evidence remains
listed in Testing as historical validation.

Before

N/A

After

Testing

Passed: pnpm exec vitest run packages/shared/src/submissions/submission-coordinator.test.ts packages/shared/src/submissions/submission-state.test.ts packages/shared/src/schemas/readout.test.ts packages/shared/src/scripts/register-handler.test.ts apps/api/src/routes/submissions.test.ts apps/api/src/routes/reports-structured.test.ts apps/api/src/openapi-snapshot.test.ts apps/api/src/openapi/openapi.test.ts apps/scheduler/src/handler-dispatcher.test.ts apps/scheduler/src/notify-routes.test.ts apps/workers/readout/src/readout-builder.test.ts apps/workers/readout/src/readout-queue-processor.test.ts apps/portal/src/components/SubmissionStatusCard.test.tsx apps/portal/src/components/ui/stories.play.test.tsx — 144 tests passed.
Passed: pnpm --filter shared build
Passed: pnpm --filter readout-handler build
Passed: pnpm --filter api build
Passed: pnpm --filter cli build
Passed: pnpm --filter static-prompt-evals build
Passed: pnpm --filter portal lint
Passed: pnpm --filter portal exec vite build
Passed: targeted submission-readout adapter, dataset-contract, coverage, and dataset validation commands; 223 approved cases and 16 quality targets validated.
Passed: pnpm eval:static-prompts:smoke selected the reviewed submission-readout case and completed the offline fake-model path. As designed, offline policy acceptance remained not-evaluated.
Infrastructure blocked: real-model pnpm --filter static-prompt-evals evaluate:quality -- --smoke selected the submission-readout case but could not generate or grade because no REST inference backend / Azure evaluator credentials are configured.
Passed: pnpm install --frozen-lockfile --offline --ignore-scripts
Passed: pnpm notice:check
Passed: pnpm headers:check
Passed — genuine local E2E: started MongoDB, Redis, Azurite, API, scheduler, Copilot coder, Judge, pp-atif/pp-observations, pp-readout, Portal, and CLI through Docker Compose; used a command-scoped gh auth token without printing or persisting it.
Passed — real inference and exact-vector flow: native one-request submission 442debb8-acd9-4aed-9c42-75e2190cf861 / request abc5a025-ed6e-4bb0-9309-2f483fa7a004 completed coding run dffb70c2-950e-4f14-9520-403745e09a52, pp-atif, a real Judge-backed pp-observations evaluation, submission reconciliation, and a real Copilot SDK readout for exact submission run 79b0e694-345c-50e0-b504-12b8ac672cb6.
Passed — persistence/retrieval: current report 867481ab-97d6-482e-b77d-70edfefa9a18 was persisted template-less with inline Markdown and a 3,985-byte application/json blob at reports/867481ab-97d6-482e-b77d-70edfefa9a18/structured.json; API report-by-ID and structured proxy downloads passed, CLI Markdown/structured downloads were semantically identical, and all three queues drained to zero.
Passed — retry/history: two failed readout attempts remained immutable and superseded after manual retry; attempt 3 selected only the latest report as current.
Passed: rebuilt the readout development image, verified its OS CA bundle, and created a real gpt-5.4 Copilot SDK session inside the container.
Passed: pnpm exec vitest run scripts/readout-container.test.ts apps/workers/readout/src/readout-queue-processor.test.ts packages/shared/src/schemas/readout.test.ts — 8 tests passed.
Passed: pnpm --filter readout-handler build
Passed: scripts/dev-compose.sh -f docker-compose.yml -f docker-compose.dev.yml --profile post-processor --profile readout config --quiet
Passed: Portal dev server returned the submission route and compiled the Submission Detail module with standard report-by-ID retrieval; browser-canvas rendering reached the local auth shell, while the authenticated data path was verified through the same live API.
Passed — generic report terminology: pnpm exec vitest run apps/portal/src/components/SubmissionStatusCard.test.tsx apps/portal/src/pages/SubmissionDetail.test.tsx apps/portal/src/pages/ReportPreviewPanel.test.tsx apps/portal/src/components/ui/stories.play.test.tsx — 40 tests passed; pnpm --filter portal lint, pnpm --filter portal exec tsc --noEmit, pnpm --filter portal exec vite build, and pnpm --filter portal build-storybook --quiet all passed.
Passed — five-run recorded E2E: native submission b1ff8749-43a4-4b68-a2be-ee33c213eb17 completed five exact request/run pairs: 12b5fdcf-fc2d-40a0-9438-8aae2b26c383 / d6f2f2ef-261d-477a-a976-364008f4cd9b, 748a47c5-1375-4593-ae38-34bd6a68637e / 52388e21-4859-4399-8071-5fc4af47282d, 7d917cac-9151-4b11-a4ef-6e08df1e64dd / 22004c8b-f3bc-43fa-91fe-4f07bd2ecf33, f1a92743-659d-4b2e-b39c-e6272c5277f2 / a3cfbdb4-0e62-415b-b742-dfb259a4b164, and c15b99dc-3077-4de3-98d7-8c9bd1ae4167 / c3108d5f-6d21-4a97-a478-ad4ec67b07c0. The real coder, pp-atif, Judge-backed pp-observations, scheduler reconciliation, and pp-readout worker all executed.
Passed — exact report result: current submission run 37fd2c77-ceb9-578c-87b9-a22cb6309a2a selects report 281f4a1c-4d28-4899-8642-e7c904bef08b; attempts 1ba94643-1698-5550-a2a8-797a178e74f4 and 862c080a-b2b4-53dd-85e2-f31ff565690e remain superseded. The submission report history API, standard report-by-ID response, and proxied Structured JSON returned schema scope.submission-readout version 1.0, five runs, and exact request/run provenance. CLI Markdown and Structured JSON retrieval matched the API.
Passed — live-found regression fix: the five-run synthesis exposed that task criteria were queried but not attached to each run's evidence. Commits b1811e10 and c1b81100 include task, configured observation, run-result, and turn-result criteria. pnpm exec vitest run apps/workers/readout/src/readout-queue-processor.test.ts — 4 tests passed; pnpm --filter readout-handler build passed.
Passed — report API convention correction: pnpm exec vitest run apps/api/src/routes/submissions.test.ts apps/api/src/openapi-snapshot.test.ts — 6 tests passed; pnpm exec vitest run apps/cli/src/commands/submission.test.ts apps/cli/src/commands/report.test.ts — 4 tests passed; pnpm exec vitest run apps/portal/src/pages/SubmissionDetail.test.tsx apps/portal/src/components/SubmissionStatusCard.test.tsx apps/portal/src/components/ui/stories.play.test.tsx — 39 tests passed.
Passed — report API convention builds: pnpm --filter api build, pnpm --filter cli build, pnpm --filter portal lint, pnpm --filter portal exec vite build, and pnpm --filter portal build-storybook --quiet.
Passed — refined five-run E2E: native submission 620c3b42-bdb5-4d86-9ebf-2f86cc7ce263 completed exact request/run pairs fa1c6726-84d4-4a6b-aa58-7266cb17c734 / 32e0e1fa-c59c-4b10-9ff0-568045f1036f, 9d6c8760-1fe9-4c9f-93d0-e886b3330ddc / c6d0c117-25b5-410c-b316-3ba6e2fff483, 28e3e2c3-23f0-4945-8812-1975b3af745b / 71badb76-0d7b-409f-9b16-f079e45bf352, c485e475-062f-4d83-9e7c-dd4409fc5e97 / a9ff6c5a-3441-4c3f-92ce-bf5cc6051c88, and c5dd6da2-af3b-4e22-a27f-797a21980b66 / cf0348b4-396e-4c69-8409-8e8fadd911c2. The real coder, pp-atif, Judge-backed pp-observations, scheduler reconciliation, and pp-readout worker executed for all five runs.
Passed — refined report history: initial submission run 8e303801-c02c-5223-a715-208284b4d77a produced historical report 27507d7b-e201-4c46-bca2-f0b72fe36342; a successful retry created current submission run f6c2e939-c171-5020-b212-68dda35b44f9 and current report 0a9c29fc-2da9-41c1-9133-b46e2d67ec95. The first attempt/report remained immutable and historical while the embedded current pointer selected only the retry.
Passed — refined walkthrough artifacts: recorded with pinned full Chromium in headless mode at 1440×900 and DPR 1, assembled two edit passes, and retained /tmp/scope-five-run-refined-demo/scope-five-run-submission-report-walkthrough.mp4 (5,915,206 bytes, 136.6 seconds, H.264/yuv420p, 30 fps, +faststart). Browser playback reported ready state 4 at 1440×900 and advanced normally; representative frames across every clip boundary were inspected with no white loading gaps. Twenty-two annotated PNGs were captured under /tmp/scope-five-run-refined-demo/. Binaries remain outside the repository, are available in the session Library, and have not been uploaded.
Passed — refined submission UX and report-type correction (65aa8818): pnpm exec vitest run apps/portal/src/pages/SubmissionDetail.test.tsx apps/portal/src/components/SubmissionLink.test.tsx apps/portal/src/components/SubmissionStatusCard.test.tsx apps/portal/src/components/ui/stories.play.test.tsx — 50 tests passed. Coverage includes explicit Submission navigation, shared member-run table navigation, attempt switching, arbitrary dependency labels, Reports/Runs/Processing tabs, selected-attempt report scoping, one and multiple horizontal report-type tabs, deterministic newest-per-type duplicate selection, and current-attempt progress when only historical reports exist.
Passed — submission templates, browser tabs, and deep links (337c8a0a): pnpm exec vitest run packages/shared/src/report-templates/prompt.test.ts apps/api/src/routes/reports-submission-trigger.test.ts apps/api/src/routes/report-templates.test.ts apps/api/src/openapi/openapi.test.ts apps/api/src/openapi-snapshot.test.ts apps/scheduler/src/submission-dispatcher.test.ts apps/workers/report-generator/src/submission-tools.test.ts apps/workers/report-generator/src/report-queue-processor.test.ts apps/portal/src/pages/CreateReportTemplate.test.tsx apps/portal/src/pages/SubmissionDetail.test.tsx apps/portal/src/components/ui/stories.play.test.tsx — 269 tests passed across 11 files. Coverage includes Portal creation of a submission-scoped template, exact-current-attempt and idempotent triggering, scheduler claim rollback, exact-attempt evidence tools, direct report deep links, stale duplicate canonicalization, browser back/forward navigation, and connected browser-tab styling.
Passed — submission-template production validation (337c8a0a): Portal lint, TypeScript check, direct Vite production build, and Storybook build passed; shared, API, scheduler, report-generator, and static-prompt-evaluation builds passed. Submission adapter tests passed (5 passed, 12 skipped), the dataset adapter contract passed (2 tests), and dataset validation approved 224 cases across 17 quality targets.
Passed — latest real submission-template E2E: created template five-run-delivery-summary through the Portal as a submission-attempt template, then submitted native five-run submission a642e500-6d3d-4db3-8ad4-0a4128d8d832, exact attempt 6eede106-db0e-59bb-bd3f-74876a477f4a. All five coding runs succeeded: ed25ddab-fb7b-4ada-ba26-8b829cbda723 / a3203a94-839e-4dd3-8baf-623314f3b437, 383fc3d1-2315-4408-afa7-8c2e6b79de9b / d8aa5fbf-9e81-47ca-b089-2e086eac4b4d, 42f6e540-59b5-4517-819f-90a9b383e47d / fda0f7a0-0d64-4a67-aeb0-c1e2e522b0e7, 84c8cbf6-7f54-42e4-933c-c4aff97c8a64 / 6263ab65-c157-4192-b623-0b11f196dfc8, and 2b2b7ef6-1d39-4c4a-9140-fb60a7586730 / c0e9900c-e885-4221-b506-5769681e6333. The real coder, task Judge, pp-atif, Judge-backed pp-observations, scheduler reconciliation, and pp-readout completed.
Passed — real automatic template trigger: current pp-readout report 1fbad7c2-1aa3-4114-8d1c-b9e3a4c5fd26 completed first. The scheduler then claimed the exact current attempt, called the submission trigger, and created template report dc667846-9d43-45b0-a572-da010908ce54; report-generator moved it from generating to completed using exact-attempt structured evidence. Submission Detail showed the two results as connected sibling report-type tabs. The stable direct link http://127.0.0.1:5106/submissions/a642e500-6d3d-4db3-8ad4-0a4128d8d832?tab=reports&attempt=6eede106-db0e-59bb-bd3f-74876a477f4a&report=dc667846-9d43-45b0-a572-da010908ce54 selected the generated template tab directly.
Passed — latest replacement walkthrough artifacts: recorded with pinned full Chromium in headless mode at 1440×900 and DPR 1, with attention cues and captioned beats for Portal template creation, automatic trigger semantics, live five-run membership/readiness, processing logs, connected browser-style report tabs, generated template content, and the exact report deep link. /tmp/scope-submission-template-deeplink-demo/scope-submission-template-deeplink-walkthrough.mp4 is 4,829,247 bytes, 91.17 seconds, H.264/yuv420p, 30 fps, and +faststart. Headless Chromium playback reported ready state 4 at 1440×900 and advanced normally; six boundary frames and the contact sheet were inspected with no blank transition frames. Eleven revised annotated PNGs are in the same folder. The folder is in the session Library; binaries remain outside the repository and have not been uploaded.
Passed — historical report-type E2E: native submission edadd5b5-72e7-4634-ba2b-5ae5ef963fde completed exact attempt 7b8d619c-9f0b-52be-87be-387b62ea702e with five successful request/run pairs: a3c2cc20-52ca-4145-b55d-dd94297ae92d / 73a49193-9b95-4ca4-95ea-3de56b3ae18b, 506963dd-65f6-4553-b94c-a2084765777c / c77b64e4-d2cf-473b-aa6e-4d1fd515286d, bae59020-5a37-4a2c-8e3e-d77495db0f02 / a8a38bc9-72c5-4a68-abb7-0609b1324707, 952e08f5-09fe-46b5-b5fa-bd13c3ecdc70 / 54d4aa28-1f52-4e66-9727-e924add8b15c, and 96f8072b-594d-4576-a855-28f0ec4f9548 / 11eca64c-26ce-4ba6-b657-4133b0647187. The real coder, pp-atif, Judge-backed pp-observations, scheduler reconciliation, and pp-readout produced current report ad85ab86-2e17-439f-9197-6be736e0729f.
Passed — historical two-type demonstration: before the real submission-template trigger existed, one safe completed template-style demo report 483dbb72-4541-436c-b4a5-9af68bffeb49 with templateId: executive-summary was seeded for that exact attempt without changing the embedded current pointer. The live API and Portal showed Executive Summary and reporter-derived Submission Readout as sibling horizontal report-type tabs; no report-version/history sidebar was present.
Passed — historical report-type walkthrough artifacts: retained /tmp/scope-five-run-report-types-demo/scope-five-run-report-type-tabs-walkthrough.mp4 (6,009,918 bytes, 131.53 seconds, H.264/yuv420p, 30 fps, +faststart) and its nineteen annotated PNGs as prior evidence. The latest real-trigger walkthrough above supersedes it.
Passed — exact submission-attempt logs: pnpm exec vitest run packages/shared/src/logging/log-publisher.test.ts packages/shared/src/queue/base-queue-processor.test.ts apps/api/src/routes/submissions.test.ts apps/api/src/openapi-snapshot.test.ts — 28 tests passed. Live Redis events and persisted logs use the exact submissionRunId; historical replay closes without following a newer current attempt.
Passed — refined UX builds: pnpm --filter portal lint, pnpm --filter portal exec vite build, pnpm --filter portal build-storybook, pnpm --filter shared build, and pnpm --filter api build.
Known base-only failure: pnpm --filter static-prompt-evals test:ts has the PR A criterion-authoring assertion drift: production returns suggestedSubject: "run" but the inherited expectation omits it. The submission-readout prompt tests pass.
Known base-only failure: pnpm --filter portal build fails in inherited CriteriaList.tsx because CriteriaBadge is referenced without an import. Portal tsc --noEmit and direct Vite production bundling pass for this layer.

Documentation and compatibility

Updated the post-processing, application design, architecture overview, environment-variable, database-migration, and static prompt-evaluation documentation. Regenerated the API and website OpenAPI artifacts.

Existing run reports remain request-scoped and compatible. Submission reports use
submissionId plus exact submissionRunId, with no report-kind discriminator.
Report templates now carry a backward-compatible subject (run by default);
submission templates use the automatic trigger and generate through the existing
report worker after the exact current attempt's readout evidence is complete. The
readout worker adds its own queue/profile and aligned existing dependency versions;
the lockfile and notices were validated.

The draft-only submission API is aligned with existing run report conventions:
GET /api/v1/submissions/:id/reports lists current and historical submission
reports, while the embedded current attempt's reportId is resolved through
GET /api/v1/reports/:reportId. No singular submission report resource or
compatibility alias is retained.

The readout image installs the operating-system CA bundle required by the bundled Copilot CLI, honors the configured npm registry with a pinned pnpm fallback, and uses the development image target for source-based Compose registration jobs.

Submission attempts now expose exact-attempt SSE processing logs at GET /api/v1/submissions/:id/runs/:submissionRunId/logs. Submission Detail scopes its immutable vector, dependencies, report types, and logs to the selected attempt. Reports are grouped into connected browser-style tabs by templateId or template-less reporter.id; pathological same-type duplicates resolve to the newest report without version-history UI. Stable tab=reports&attempt=<submissionRunId>&report=<reportId> links select a specific attempt and report type, stale duplicate IDs canonicalize to the newest report of that type, user navigation participates in browser back/forward history, historical attempts remain read-only, and only the embedded current report pointer is marked current.

Checklist

If Portal features changed, keep CLI capabilities in sync.
If Portal components changed, update their Storybook stories.
If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. — N/A; no new database migration is required.
If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed.
Video showing the behavior before the suggested change — N/A for this draft.
Video showing the behavior after the suggested change — upload placeholder included above.

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Add the pp-readout submission handler, exact-vector evidence aggregation, structured artifacts, report APIs, and stale-publication safeguards. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Render current and historical submission readouts, support structured downloads, and add CLI access while removing the duplicated run taxonomy surface. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Describe submission-scoped readouts, refresh API artifacts, and register the production prompt contract and reviewed smoke case. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Install the system CA bundle required by the bundled Copilot CLI, forward configured npm registries, and run all source-based handler registration jobs from development images. Log SDK cleanup failures and add focused container configuration regression coverage. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Use the plural submission reports collection and resolve the current report through the standard report-by-ID resource. Align Portal and CLI clients, filters, Structured JSON retrieval, OpenAPI, tests, and architecture documentation. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Replace report-version history with attempt-scoped horizontal report-type tabs and deterministic newest-per-type selection. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Add submission-scoped templates, exact-attempt automatic triggering, browser-style report tabs, and stable report deep links. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

 // Licensed under the MIT License.

-import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, McpServerDocument, CreateMcpServerRequest, UpdateMcpServerRequest, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ResourceDocument, ResourceRevisionDocument, ResourceBindingSpec, CreateResourceBody, CreateResourceRevisionBody, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, RunFacetsResponse, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType, RunSortField, RunSortDir, CriterionKind, TaxonomyElementId, CriterionSubject, Submission } from "@/types";
+import type { Run, RunState, CriteriaDocument, CriteriaGraphData, GeneratePromptResponse, AnalysisResponse, PromptFeatureDocument, Report, BulkReportStatus, BulkReportSummary, ReportTemplate, ReportTrigger, ReportTemplateSystemPrompt, KeyDocument, KeyValidationResult, CreateKeyRequest, UpdateKeyRequest, CodingAgent, AgentVersion, BulkResubmitOverrides, Insight, InsightWithReference, TaskPrompt, TaskPromptFeatureExtractionResult, Model, FeatureFlag, SkillDocument, SkillSearchResult, SkillDiscoveryResult, SkillRevisionDocument, CodebaseDocument, CodebaseRevisionDocument, CodebaseSourceType, ResourceDocument, ResourceRevisionDocument, ResourceBindingSpec, CreateResourceBody, CreateResourceRevisionBody, ExtensionDocument, ExtensionSearchResult, ExtensionVersionInfo, MdpResponse, AccountDocument, CreateAccountRequest, UpdateAccountRequest, ProfileWithVersion, ProfileVersionDocument, ProfileDocument, RunGroup, RunFacetsResponse, CursorPaginatedResponse, IterationOp, GateConfig, GateId, PromptType, RunSortField, RunSortDir, CriterionKind, TaxonomyElementId, CriterionSubject, Submission, SubmissionRunAttempt } from "@/types";


[P1] Restore the MCP API type imports

This removes McpServerDocument, CreateMcpServerRequest, and UpdateMcpServerRequest, but the MCP methods later in this file still use them. The Portal build now fails with TS2304 at those methods, with cascading implicit-any errors in the MCP pages. Please retain these imports alongside the new submission types. These are introduced errors, separate from the pre-existing CriteriaBadge build error.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+                      {submissionHandlerLabel(blocking.handlerId)}
+                    </span>
+                    <Badge variant="destructive">
+                      {blocking.reason.replaceAll("-", " ")}


[P1] Use an ES2020-compatible replacement here

The Portal's tsconfig.app.json sets lib to ES2020, which does not declare String.prototype.replaceAll. This new expression fails the Portal build with TS2551 even though the component tests pass. Use an equivalent such as blocking.reason.replace(/-/g, " ") rather than introducing an ES2021 API under the existing configuration.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+      scheduler:
+        condition: service_started
+    restart: "no"
+    entrypoint: ["npx", "tsx", "packages/shared/src/scripts/register-handler.ts"]


[P1] Make the registration entrypoint match the production image

Without docker-compose.dev.yml, this service builds the final production stage of the readout Dockerfile. That stage copies only compiled dist files and sets the working directory to /app/apps/workers/readout, so this relative TypeScript-source entrypoint does not exist. The relative HANDLER_YAML_PATH is also rooted incorrectly there. Registration fails, and readout-handler cannot start because it depends on this job completing successfully.

Set the registration job's dev build target in the base Compose file, or invoke the compiled registration script with a working directory and YAML path that match the production image. The current dev-only override does not fix the documented base Compose startup path.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+    if (!claimed) {
+      await log("info", "Discarding stale or duplicate readout message", {
+        submissionId,
+        submissionRunId,
+      });
+      await this.safeDeleteMessage(message.messageId, heartbeat.stop());
+      return;


[P1] Recover abandoned processing claims before deleting redeliveries

If a worker exits after claiming queued -> processing, its queue message becomes visible again while the submission handler remains processing. claimSubmissionHandler only accepts queued, so the replacement worker gets null and deletes this legitimate redelivery as a duplicate. Reconciliation only redispatches a ready attempt, and the existing stuck-run reaper handles request runs rather than submission handlers, leaving this attempt permanently processing. The Portal also disables retry while it is processing.

I reproduced this using the actual claim helper and worker handler. Please add ownership/lease or heartbeat-based recovery for abandoned processing claims, while preserving the stale/superseded-message discard behavior.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+        await ctx.reportCollection.insertOne(reportDoc);
+        await ctx.reportQueueClient.sendMessage(


[P1] Recover reports persisted before a failed queue send

The insert succeeds before sendMessage, but every subsequent trigger skips the report once findOne sees it. If enqueueing fails, the scheduler's retry therefore returns success without resending the job, leaving the report pending forever. I reproduced a first trigger returning 500 followed by 201 with triggered: 0; sendMessage was called only once and the persisted report remained pending.

Make persistence plus dispatch recoverable, for example with an outbox or an idempotent pending-job recovery path. An existing report must not be treated as successfully dispatched solely because the document exists.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+    refetchInterval:
+      submissionQuery.data?.readiness === "done" ||
+      submissionQuery.data?.readiness === "failed" ||
+      submissionQuery.data?.readiness === "blocked"
+        ? false


[P2] Keep refreshing sibling reports after the readout is done

Submission readiness becomes done when pp-readout completes, but submission-template reports are created and generated afterward. Disabling report polling at that point either misses the new report types entirely or freezes them at pending/generating until the user reloads. I reproduced the actual component fetching reports only once and never displaying a template report that completed afterward.

Refresh through the template-creation phase and until pending/generating reports reach a terminal state, or subscribe to report lifecycle changes. Readout readiness alone is not a completion signal for this reports collection.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+        exactRuns.flatMap(({ request, run }) => [
+          ...request.scenario.criteria,
+          ...(request.observations ?? []),
+          ...(run?.observationResults ?? []).map((result) => result.criterionId),
+          ...(run?.turns ?? []).flatMap((turn) =>
+            (turn.observationResults ?? []).map((result) => result.criterionId)


[P2] Include configured and evaluated gate criteria in the evidence

This gathers scenario.criteria and observation IDs, but omits request.gates[].criteria and turn.criteriaResults. Gate configuration can differ from scenario.criteria, and evaluation results also contain resolved DAG ancestors. The full run passed to the model contains these results, while their criterion definitions are absent from entry.criteria; composeSubmissionReadout then rejects a legitimate finding that cites one of them as an unknown criterion.

I reproduced an existing build-gate result being omitted by the loader and rejected by the composer. Include configured gate IDs and all evaluated criterion-result IDs in both this global collection and the per-run runCriterionIds collection below.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+          const effectiveSubject = subject ?? existing.subject ?? "run";
+          const effectiveTrigger = trigger ?? existing.trigger;
+          if (
+            effectiveSubject === "submission" &&
+            effectiveTrigger !== undefined &&
+            effectiveTrigger.type !== "always"
+          ) {


[P2] Validate the effective trigger when only the subject changes

This validation is inside if (trigger !== undefined), so a subject-only update bypasses it. Updating a criteria-triggered run template with { "subject": "submission" } returns 200 and preserves its criteria trigger, even though trigger-submission silently skips every submission template whose trigger is not always. The saved template consequently never runs automatically.

Validate the resulting subject/trigger combination whenever either field changes, including the existing trigger when the update omits it. Reject or explicitly normalize an unsupported combination instead of persisting it.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

  .object({
    id: z.string(),
    name: z.string(),
+    subject: z.enum(["run", "submission"]).default("run"),


[P2] Wire submission-template subjects through the CLI as well

The Portal can now create and edit submission templates, but scope report-template create and update have no subject option, and mapYamlReportTemplate drops the YAML subject field. Importing a template with subject: submission therefore silently creates a run template because this schema defaults the omitted field to run. This makes the new capability unavailable to CLI/CI users and violates the repository's CLI/Portal parity contract.

Add subject support to template create/update, preserve it in the YAML mapper, and expose it in template list/get output. Cover the import path so a submission template cannot silently change subjects.

Wassim Chegham (manekinekko) · 2026-10-14T15:02:09Z

+      return {
+        request,
+        rawResponse: completion.content,
+        output: completion.content.trim(),


Non-blocking: validate the readout synthesis in the prompt-evaluation adapter

This returns arbitrary nonempty model text as a successful readout output. The inherited run-report output_schema check only tests output presence, not JSON or synthesis-schema validity, so malformed output can pass the advertised schema check. I confirmed that this adapter accepts not JSON without reporting an error, although production generation would reject it.

Reuse the production synthesis parsing/schema and composition checks here, and add exact-vector/provenance contract cases, so the evaluation catches the failures that matter to the runtime.

Cedric Vidal (cedricvidal) added 4 commits October 13, 2026 16:29

feat(taxonomy): restack readout handler on DAG foundation

8f951ab

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

docs(readout): document submission report generation

bd136be

Describe submission-scoped readouts, refresh API artifacts, and register the production prompt contract and reviewed smoke case. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

github-actions Bot added the area: cicd Build, test, release, and deployment pipelines. label Oct 13, 2026

Cedric Vidal (cedricvidal) added 10 commits October 13, 2026 21:08

fix(portal): use generic report terminology

0f8f70c

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

fix(portal): clarify pending report state

532530a

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

fix(readout): include task criteria in evidence

b1811e1

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

fix(readout): attach all criteria to run evidence

c1b8110

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

feat: stream exact submission attempt logs

229a439

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

feat: refine submission report experience

c8049d5

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

docs: describe attempt-scoped submission reports

deaf47d

Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

fix: scope submission reports by type

65aa881

Replace report-version history with attempt-scoped horizontal report-type tabs and deterministic newest-per-type selection. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

feat: trigger submission report templates

337c8a0

Add submission-scoped templates, exact-attempt automatic triggering, browser-style report tabs, and stable report deep links. Co-authored-by: Copilot App <[email protected]> Copilot-Session: 1cbbd5eb-61c6-4e6a-bb38-c377be5b2163

github-actions Bot added the area: scheduler label Oct 13, 2026

Wassim Chegham (manekinekko) reviewed Oct 14, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat: add submission-level generated readouts - #1495

feat: add submission-level generated readouts#1495
Cedric Vidal (cedricvidal) wants to merge 15 commits into
cedricvidal-pr-a-submission-foundationfrom
cedricvidal-pr-b-submission-readout

Cedric Vidal (cedricvidal) commented Oct 13, 2026 •

edited

Loading

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

Wassim Chegham (manekinekko) Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		await ctx.reportCollection.insertOne(reportDoc);
		await ctx.reportQueueClient.sendMessage(

Conversation

Cedric Vidal (cedricvidal) commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demos

Before

After

Testing

Documentation and compatibility

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Cedric Vidal (cedricvidal) commented Oct 13, 2026 •

edited

Loading