Skip to content

Latest commit

 

History

History
186 lines (144 loc) · 6.86 KB

File metadata and controls

186 lines (144 loc) · 6.86 KB
title Pull request comments
description The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.

@e2e-dev/github posts test results as a pull request comment, including failures, source links, and links to workflow artifacts. Reruns update the same comment, and a --last-failed rerun updates it as one run. The reporter also writes the results to the job summary.

Setup

```bash npm npm install --save-dev @e2e-dev/github ```
pnpm add -D @e2e-dev/github
bun add -d @e2e-dev/github
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { github } from '@e2e-dev/github';

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  reporters: ['list', github()],
} satisfies E2EConfig;

Give the workflow permission to write pull request comments and pass GITHUB_TOKEN to the test step:

permissions:
  contents: read
  pull-requests: write

jobs:
  e2e:
    steps:
      # ...checkout, install, browsers
      - name: Run e2e
        run: npx e2e run --reporter list,junit
        env:
          GITHUB_TOKEN: ${{ github.token }}
      - name: Upload artifacts
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        with:
          name: e2e-artifacts
          path: .e2e/artifacts

Upload artifacts even when a retry passes. Each failure's evidence links open the run's Artifacts section; inside the download, every test has a directory named after it and the agent that ran it (web/tests_billing.e2e.ts__upgrades/default/attempt-0/), so keep path: .e2e/artifacts and the download lines up with the report. A project below the checkout root, such as packages/e2e-tests, gets source links under that path. The checkout root is the nearest directory holding .git inside GITHUB_WORKSPACE, so actions/checkout with a path input works too.

The comment

### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31

**🔴 billing › upgrades to Pro**  
[tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)

**ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls

> expected heading "Your cart" to be visible
> The Pay button opened a page that still shows the Free plan.

- Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
- Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
- Screen: `/checkout/plan`

Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)

<details>
<summary>⚠️ 1 flaky test passed on a retry</summary>

**⚠️ invite a member by email**  
[tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)

**STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s

> the invitation email took too long

Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)
</details>

<details>
<summary>All 6 tests in 5 files</summary>

| | Test | Agent | Time |
| --- | --- | --- | --- |
| 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
| 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
| ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
| ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
...
</details>

<sub>e2e 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>

The comment shows test counts and agent usage, followed by each failure's error, source line, and evidence links. Flaky tests and the full test list are collapsed; the list is one table with a row per file above its tests, and its agent column appears only when the run used the agent. Run-level errors appear above the test failures. Large reports are shortened to fit GitHub's comment limit. Text from the app under test that GitHub would turn into a mention, an issue link, or a URL is shown as code, so a screen reading @someone or https://... notifies no one and links nowhere.

The built-in markdown reporter writes the same summary to .e2e/summary.md.

Rerunning the failed tests

A step that gives failed tests a second chance keeps one comment for both passes:

      - name: Run e2e
        run: npx e2e run || npx e2e run --last-failed
        env:
          GITHUB_TOKEN: ${{ github.token }}

The comment covers the whole suite, not only the tests the rerun ran again. A test that failed and then passed shows as flaky, a test that failed twice stays failed, one that was interrupted and then passed shows as passed, and the rest keep their first-pass results. The comment matches the job's status: it stays red when another filter kept a failed test out of the rerun, and shows ⏹️ when it kept out a test the first pass never finished.

Matrix jobs

The reporter identifies a comment by project, workflow, and job. Matrix jobs share those identifiers, so give each matrix entry a distinct key:

reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],

Set MATRIX_BROWSER: ${{ matrix.browser }} in the step's env. The key also appears in the headline, such as e2e chromium: 77 passed. Keep it stable across runs so reruns update the existing comment.

When nothing is posted

The reporter never changes the run's status or exit code. When it cannot post, it says why in one row under the summary:

       Report  .e2e/report.json

       GitHub  not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
Situation What happens
Not on GitHub Actions Nothing is written; the row says so.
A push, schedule, or workflow_dispatch run The job summary is written; there is no pull request to comment on.
No GITHUB_TOKEN in the step's env The job summary is written; the row shows the line to add. GH_TOKEN is read too.
A pull_request event from a fork Its token is read-only, so posting fails and the reporter explains why.
An issue_comment event On a pull request's thread, the comment is posted without source links, since the checked-out commit is the default branch.
Missing pull-requests: write The same failure, with the permission named.

The token is read when the run finishes and is excluded from the report. GitHub Enterprise Server uses GITHUB_SERVER_URL and GITHUB_API_URL.