Skip to content

chore: version packages - #715

Merged
okwasniewski merged 1 commit into
mainfrom
changeset-release/main
Oct 2, 2026
Merged

okwasniewski merged 1 commit into
mainfrom
changeset-release/main

Conversation

@github-actions

@github-actions github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.

Releases

@e2e-dev/[email protected]

Minor Changes

  • #730 1554360 Thanks @szdziedzic! - easSimulators() authenticates with the eas-cli login when EXPO_TOKEN is not set: it reads the session eas login keeps in ~/.expo/state.json (under USERPROFILE on Windows) and sends it in the expo-session header, as eas-cli does, so a machine signed in to eas-cli needs no token. EXPO_TOKEN still wins when both are there. A session is stopped as the account that started it, even if the login changes during the run. Only the production login is read, since the sessions API is api.expo.dev. With neither a token nor a login, the lease fails with EXPO_TOKEN is not set and eas-cli is not logged in.

  • #731 88cef2c Thanks @szdziedzic! - easSimulators() takes projectId from the app config when the option is absent: extra.eas.projectId, the id eas init writes, read from app.config.json or app.json beside e2e.config.ts, or from a dynamic app.config.ts (or .js, .mjs, .cjs, .mts, .cts) as the project's own expo config --type public evaluates it in the run's environment, without .env files, as eas-cli runs it. It reads the config once per run, before the first session starts; a config that links no project fails the lease naming the file. projectId is optional now and still wins when it is passed. Reading it needs the @e2e-dev/mobile that passes projectRoot to device providers.

@e2e-dev/[email protected]

Minor Changes

  • #731 88cef2c Thanks @szdziedzic! - A DeviceProvider's acquire request carries projectRoot, the directory the config's relative paths resolve against, so a provider reads project files there instead of process.cwd().

  • #740 bb904d4 Thanks @NathanWalker! - device.fold(pose) puts a foldable iOS simulator (iPhone Duo) in a hinge pose: 'closed' lights the outer display, 'half-open' (a 130° book) and 'open' the inner one. It runs agent-device's fold command, which sends the simulator's hinge event and reads the angle back from CoreDevice before it resolves, so the next observation reads the app on the other display. Like setOrientation, it counts as an action for the transition budget. A device without a hinge is UNSUPPORTED_CAPABILITY, and Android is refused before any device command. The pose type is exported as FoldPose.

[email protected]

Patch Changes

  • #710 6b25a3a Thanks @okwasniewski! - A failure where no model answered (MODEL_PROVIDER_FAILED, MODEL_UNAVAILABLE) no longer evicts replay cache entries. It says nothing about the app, so a read-write run without model access keeps every recording it did not confirm, including a step that replayed whole before an agent.assert that could not reach its model and a replay that handed off to the model mid-step, and still saves the steps verified before the failure.

  • #736 c9c3e14 Thanks @programmrz! - Agent steps on DeepSeek's V4 models no longer fail on their first turn with MODEL_PROVIDER_FAILED. In thinking mode, DeepSeek's default, the API refuses a forced tool choice with Thinking mode does not support this tool_choice; that refusal now reads as one, so the step retries with auto and the tool-calls-only rule, as it already did for Anthropic, and thinking can stay on.

  • #735 28b09f3 Thanks @okwasniewski! - e2e init adds zod next to ai when it sets up a model gateway. zod is a peer of ai and every provider package; npm and pnpm install peers on their own, Yarn does not, so a Yarn project failed to load the generated config with CONFIG_LOAD_FAILED. A Yarn project scaffolded with a model gateway before this fix needs yarn add -D zod.

  • #708 4ee772c Thanks @okwasniewski! - The package ships the documentation pages: coding agents can read them offline from node_modules/e2e/docs.

  • #742 4f17480 Thanks @okwasniewski! - expect(value).toBeDefined() now passes on null, as it does in Playwright, Jest, and Vitest; only undefined fails it. It used to reject null too, so expect(null).not.toBeDefined() passed where Playwright fails. expect.poll(read).toBeDefined() follows: a read that returns null now passes on the first try instead of waiting. For the old non-nullish check, use toEqual(expect.anything()).

  • #739 8d38206 Thanks @NathanWalker! - The agent loop handles providers that downgrade a forced tool choice instead of refusing it, as the AI SDK's Anthropic provider does for Claude models that reject forced tool use (Claude Sonnet 5.5 through @ai-sdk/anthropic). Such a request succeeds with auto, so the HTTP 400 path never ran: the loop kept asking for a forced choice, the SDK logged an AI SDK Warning ... toolChoice line on every turn, steps ran without the tool-calls-only instruction, and a turn the model answered in prose failed the step with MODEL_PROVIDER_FAILED (the SDK's ToolChoiceViolationError). Now a toolChoice call warning switches the step to auto and gives its remaining turns the instruction, a prose answer under a forced choice is retried with auto like a refusal, and later steps on that model start in auto.

@github-actions
github-actions Bot force-pushed the changeset-release/main branch 17 times, most recently from 1ec0f7b to 2cd541e Compare October 2, 2026 08:30
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

🟢 e2e web: 90 passed

All 90 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 412ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 291ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 121ms
🟢 tests/async-states.e2e.ts · 2 passed 6.9s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.2s
🟢 async states › a wrong code is rejected after the toast has gone 4.7s
🟢 tests/canvas-only.e2e.ts · 2 passed 194ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 103ms
🟢 canvas only › a shape out of order paints the error and resets progress 91ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 305ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 171ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 134ms
🟢 tests/cart-totals.e2e.ts · 2 passed 313ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 187ms
🟢 cart totals › a quantity never drops below one 126ms
🟢 tests/checkout-review.e2e.ts · 1 passed 149ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 149ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 466ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 237ms
🟢 context menu trap › the decoy menu items refuse and keep the file 229ms
🟢 tests/control-inventory.e2e.ts · 7 passed 3.0s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 210ms
🟢 control inventory › a long press and a double click are told apart from a tap 967ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 186ms
🟢 control inventory › setInputFiles attaches files named from the project root 110ms
🟢 control inventory › the details view is left through the browser history 133ms
🟢 control inventory › scrollIntoView brings the footnote into view 221ms
🟢 control inventory › every exercise done shows the success message 1.2s
🟢 tests/css-content-ui.e2e.ts · 2 passed 282ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 161ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 121ms
🟢 tests/date-picker.e2e.ts · 2 passed 527ms
🟢 date picker › navigates three months ahead and books the required date 297ms
🟢 date picker › confirming without the exact date shows the error 230ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 577ms
🟢 div soup › a wrong code is rejected 262ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 315ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 579ms
🟢 drag and drop › an out-of-order item resets the dropzone 286ms
🟢 drag and drop › dropping Banana then Cherry passes 293ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 193ms
🟢 filter deep link › filtering writes the query to the URL 100ms
🟢 filter deep link › opening a shared link restores the filter 93ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 442ms
🟢 gift card purchase › refuses to buy without an amount 124ms
🟢 gift card purchase › refuses a recipient without an at sign 147ms
🟢 gift card purchase › sends the gift card 171ms
🟢 tests/hover-menu.e2e.ts · 2 passed 378ms
🟢 hover menu › a decoy leaf is refused 162ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 216ms
🟢 tests/iframe-form.e2e.ts · 2 passed 566ms
🟢 iframe form › a wrong code is refused by the host page 265ms
🟢 iframe form › the code read from one frame applies through the other 301ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 383ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 93ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 123ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 167ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 3.9s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 3.9s
🟢 tests/kanban-board.e2e.ts · 3 passed 611ms
🟢 kanban board › pointer drags move both cards into the goal layout 329ms
🟢 kanban board › a swipe along a path drags a card as well 133ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 149ms
🟢 tests/login-form.e2e.ts · 4 passed 567ms
🟢 login form › rejects a malformed email 132ms
🟢 login form › rejects a short password 134ms
🟢 login form › rejects an unknown account 131ms
🟢 login form › signs in and logs out 170ms
🟢 tests/lying-labels.e2e.ts · 4 passed 549ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 130ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 79ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 170ms
🟢 lying labels › visible text drives the payment through 170ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 380ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 132ms
🟢 native dialogs › dismissing the confirm cancels the order 115ms
🟢 native dialogs › a wrong gift code is rejected 133ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.5s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 238ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 275ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.0s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.4s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.6s
🟢 tests/order-history.e2e.ts · 2 passed 1.5s
🟢 order history › overview and settings render their panels 235ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.3s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.3s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 444ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 111ms
🟢 otp auto-advance › a wrong code clears every box and starts over 762ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 690ms
🟢 continues only after the banner and the overlay are gone 690ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 397ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 197ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 200ms
🟢 tests/price-sorting.e2e.ts · 3 passed 305ms
🟢 price sorting › opens in the featured order with sale prices struck through 89ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 103ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 113ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.4s
🟢 buys a deal once the promo chrome is out of the way 1.4s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 391ms
🟢 rich text editor › bolding exactly the required word passes the check 192ms
🟢 rich text editor › bold that spills into the previous word fails the check 199ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 468ms
🟢 saved addresses › adds an address and clears the form 161ms
🟢 saved addresses › ignores an add with an empty field 145ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 162ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 484ms
🟢 shadow DOM form › rejects a wrong access code 235ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 249ms
🟢 tests/smoke.e2e.ts · 1 passed 390ms
🟢 home lists the scenarios and opens one 390ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 161ms
🟢 accepts the terms once the button clears the sticky footer 161ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 1m 5s
🟢 scrolls the windowed table to the Golden Row and claims it 1m 5s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 231ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

e2e 0.15.2 · 1m 50s · web · run artifacts

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

🟢 e2e web agent: 32 passed

33 agent steps · 25 replayed from cache · 20 model calls · 76.2k tokens (84% cached) · $0.0048

All 32 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 6 passed 6 steps 2.5s
🟢 control inventory › check sets the checkbox and the radio 1 step 510ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 860ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 345ms
🟢 control inventory › upload attaches the named project files 1 step 195ms
🟢 control inventory › back returns from the details view to the inventory 1 step 229ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 337ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 333ms
🟢 act signs in with the benchmark account 1 step 333ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 11.2s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 5.0s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 1.9s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 4.3s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 12 calls 2m 36s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.4s
🟢 act completes lying-labels 1 step · 3 calls 4.6s
🟢 act completes iframe-form 1 step · 3 calls 5.8s
🟢 act completes infinite-scroll 1 step 13.5s
🟢 act completes virtualized-table 1 step 1m 41s
🟢 act completes overlay-trap 1 step 814ms
🟢 act completes sticky-chrome 1 step 6.7s
🟢 act completes async-states 1 step · 3 calls 6.8s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 429ms
🟢 act completes drag-and-drop 1 step 426ms
🟢 act completes kanban-board 1 step 655ms
🟢 act completes context-menu-trap 1 step 507ms
🟢 act completes stale-dom 1 step 413ms
🟢 act completes otp-auto-advance 1 step 646ms
🟢 act completes date-picker 1 step 765ms
🟢 act completes rich-text-editor 1 step 1.3s
🟢 act completes playbook-cleanup 1 step 567ms
🟢 act completes promo-storefront 1 step 1.4s
🟢 act completes gift-card-purchase 1 step 2.4s
🟢 act filters the list and the URL carries the filter 1 step 187ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 253ms
🟢 act adds an address with a run-unique label 1 step 253ms

e2e 0.15.2 · 2m 50s · web · run artifacts

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

🟢 e2e android agent: 20 passed

24 agent steps · 108 model calls · 590.8k tokens (82% cached) · $0.0398

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 108 calls 8m 41s
🟢 act completes Login Form 1 step · 3 calls 27.3s
🟢 act completes Infinite Scroll List 1 step · 3 calls 23.6s
🟢 act completes Modal Flow 1 step · 4 calls 10.1s
🟢 act completes Bottom Tabs 1 step · 3 calls 9.7s
🟢 act completes Text Input Variations 1 step · 3 calls 10.0s
🟢 act completes Broken Accessibility 2 steps · 6 calls 26.1s
🟢 act completes Vision Only 2 steps · 6 calls 30.2s
🟢 act completes Gestures 1 step · 8 calls 28.6s
🟢 act completes Async States 1 step · 6 calls 20.5s
🟢 act completes Debounced Search 1 step · 4 calls 9.5s
🟢 act completes Huge Virtualized List 1 step · 3 calls 1m 10s
🟢 act completes Flattened Registration Form 2 steps · 12 calls 1m 2s
🟢 act completes Flattened Login 2 steps · 6 calls 27.8s
🟢 act completes Sticky Chrome Target 1 step · 4 calls 20.3s
🟢 act completes WebView Accessibility 1 step · 5 calls 14.9s
🟢 act completes Permission Prompt 1 step · 3 calls 8.5s
🟢 act completes Product Catalog 1 step · 5 calls 13.3s
🟢 act completes Error Recovery 1 step · 6 calls 12.3s
🟢 act completes Choice Controls 1 step · 7 calls 39.0s
🟢 act completes Sequential Onboarding 1 step · 11 calls 56.8s

e2e 0.15.2 · 8m 42s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

🟢 e2e android: 1 flaky, 53 passed, 1 skipped

⚠️ 1 flaky test passed on a retry

⚠️ sequential onboarding echoes the exact values
tests/device.e2e.ts:52

ENGINE_FAILURE at step 7 of 7: locator.fill getByTestId("first-name-input"), after 4.7s

perform fill failed: Android fill verification failed Hint: Retry with --debug and inspect diagnostics log for details.

  • Screen: dev.e2e.benchmark

Evidence: screenshot, log

All 55 tests in 7 files
Test Time
⚠️ tests/device.e2e.ts · 1 flaky, 3 passed 40.6s
🟢 permission prompt › allowing the system dialog enables the microphone 6.7s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 14.4s
⚠️ sequential onboarding echoes the exact values (1 failed attempt first) 13.1s
🟢 onboarding rejects a malformed email before advancing 6.5s
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 2m 54s
🟢 control inventory › a label query resolves a labeled text field 22.4s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 12.7s
🟢 control inventory › a placeholder query and attribute reads find the field 5.2s
🟢 control inventory › a secure field takes a fill and denies value reads 6.6s
🟢 control inventory › press sends a character and Enter through the soft keyboard 8.4s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 10.1s
🟢 control inventory › toBeFocused follows the focus 9.7s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.3s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.0s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 11.4s
🟢 control inventory › a tab role answers getByRole 3.4s
🟢 control inventory › a radio group reports the checked option 9.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.9s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 6.4s
🟢 control inventory › a header answers its accessible name 3.7s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 5.3s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 4.8s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 6.7s
🟢 control inventory › a double tap lands inside the 300 ms window 7.8s
🟢 control inventory › an ambiguous locator fails at once 4.9s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 7.9s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.2s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 8 passed 1m 5s
🟢 device fixture › setOrientation rotates the window the app measures 5.0s
🟢 device fixture › home leaves the app, openApp brings it back where it was 5.6s
🟢 device fixture › foregroundApp sees the home screen after home() 3.9s
🟢 device fixture › closeApp then openApp relaunches on the home list 6.4s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.2s
🟢 device fixture › device.back pops the scenario 4.7s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 15.2s
🟢 device fixture › setLocation and clearLocation change what the app reads 19.9s
🟢 tests/flows-lists.e2e.ts · 7 passed 2m 16s
🟢 infinite scroll reaches the target item 23.4s
🟢 product catalog adds exactly two of the right variant 9.8s
🟢 gestures: long-press, then swipe left 7.8s
🟢 gestures: double-tap completes the flow 9.3s
🟢 sticky chrome: accept below the fold, then continue 11.0s
🟢 async states: load, pull to refresh, claim through the toast 10.9s
🟢 huge virtualized list reaches row 512 1m 3s
🟢 tests/flows.e2e.ts · 7 passed 50.4s
🟢 text inputs unlock the submit button 6.3s
🟢 filled text inputs report their values 5.6s
🟢 modal flow confirms through the native alert 8.0s
🟢 bottom tabs act inside the last tab 6.0s
🟢 error recovery retries, then confirms the delete 8.7s
🟢 choice controls place the exact order 7.8s
🟢 debounced search selects the target once results arrive 8.1s
🟢 tests/login-form.e2e.ts · 3 passed 29.5s
🟢 login form › rejects a malformed email 8.6s
🟢 login form › rejects an unknown account 8.6s
🟢 login form › signs in and logs out 12.3s
🟢 tests/smoke.e2e.ts · 1 passed 5.5s
🟢 home lists the scenarios and opens one 5.5s

e2e 0.15.2 · 8m 36s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

🟢 e2e ios: 1 flaky, 51 passed, 1 skipped

⚠️ 1 flaky test passed on a retry

⚠️ control inventory › toBeFocused follows the focus
tests/controls.e2e.ts:101

ASSERTION_FAILED at step 5 of 5: expect.toBeFocused getByRole("textbox", name: "Autofocus field"), after 5.1s

  • Expected: focused
  • Observed: default states (1 match)
  • Failed the same way on 2 attempts: ASSERTION_FAILED at step 5. The retry that passed is the exception.
  • Screen: dev.e2e.benchmark / Control Inventory

Evidence: screenshot, log

All 53 tests in 7 files
Test Time
⚠️ tests/controls.e2e.ts · 1 flaky, 22 passed, 1 skipped 3m 52s
🟢 control inventory › a label query resolves a labeled text field 13.0s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 20.6s
🟢 control inventory › a placeholder query and attribute reads find the field 6.1s
🟢 control inventory › a secure field takes a fill and denies value reads 11.6s
🟢 control inventory › press sends a character and Enter through the soft keyboard 10.2s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 13.0s
⚠️ control inventory › toBeFocused follows the focus (2 failed attempts first) 13.7s
🟢 control inventory › a switch checks, unchecks, and reports its state 13.3s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 10.3s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 16.2s
🟢 control inventory › a radio group reports the checked option 11.0s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 9.6s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 9.7s
🟢 control inventory › a header answers its accessible name 5.4s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 8.7s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 9.6s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 9.9s
🟢 control inventory › pointer verbs land at the point asked for 15.9s
🟢 control inventory › an ambiguous locator fails at once 7.4s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 15.7s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 350ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.2s
🟢 tests/device-fixture.e2e.ts · 9 passed 1m 43s
🟢 device fixture › setAppearance flips the color scheme the app reads 16.6s
🟢 device fixture › setOrientation rotates the window the app measures 7.8s
🟢 device fixture › the clipboard reads back what was written 7.5s
🟢 device fixture › home leaves the app, openApp brings it back where it was 7.2s
🟢 device fixture › closeApp then openApp relaunches on the home list 10.3s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.5s
🟢 device fixture › device.back pops the scenario 7.4s
🟢 device fixture › setLocation and clearLocation change what the app reads 15.0s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 24.9s
🟢 tests/device.e2e.ts · 2 passed 1m 2s
🟢 sequential onboarding echoes the exact values 46.2s
🟢 onboarding rejects a malformed email before advancing 15.6s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 49s
🟢 infinite scroll reaches the target item 29.8s
🟢 product catalog adds exactly two of the right variant 12.3s
🟢 gestures: long-press, then swipe left 10.9s
🟢 sticky chrome: accept below the fold, then continue 16.1s
🟢 async states: load, pull to refresh, claim through the toast 13.3s
🟢 huge virtualized list reaches row 512 2m 12s
🟢 web view coupon form applies the code shown on the page 14.5s
🟢 tests/flows.e2e.ts · 7 passed 1m 21s
🟢 text inputs unlock the submit button 20.6s
🟢 filled text inputs report their values 10.3s
🟢 modal flow confirms through the native alert 10.5s
🟢 bottom tabs act inside the last tab 8.3s
🟢 error recovery retries, then confirms the delete 11.0s
🟢 choice controls place the exact order 9.2s
🟢 debounced search selects the target once results arrive 11.0s
🟢 tests/login-form.e2e.ts · 3 passed 46.7s
🟢 login form › rejects a malformed email 12.4s
🟢 login form › rejects an unknown account 12.7s
🟢 login form › signs in and logs out 21.6s
🟢 tests/smoke.e2e.ts · 1 passed 8.4s
🟢 home lists the scenarios and opens one 8.4s

e2e 0.15.2 · 6m 53s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 125 model calls · 48.1k tokens (84% cached) · $0.0032

⚠️ 1 flaky test passed on a retry

⚠️ act completes Gestures
tests-agent/scenarios.e2e.ts:257

ASSERTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: long-press the first target, swipe the card left, then double-tap the last target", after 1m 10s and 13 model calls

agent.act failed: Long-pressed the first target and swiped the card left; both advanced to the next instruction. Double-tapped the final target twice, but the screen remains on Step 3 of 3 with no completion state.

  • Failed on 2 attempts: ACTION_FAILED at step 4, then ASSERTION_FAILED at step 4.
  • Turn 10: screenshot({}) → [screenshot] The screen changed substantially since revision b11. Current screen (revision b13, path dev.e2e.benchmark / Benchmark Examples, 11 nodes):
  • Turn 11: double_tap({"target":"n3948"}) → [double_tap] Double-tapped #n3948.
  • Turn 12: double_tap({"target":"n3978"}) → [double_tap] Double-tapped #n3978.
  • Screen: dev.e2e.benchmark / Benchmark Examples

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 125 calls 13m 34s
🟢 act completes Login Form 1 step · 8 calls 1m 15s
🟢 act completes Infinite Scroll List 1 step · 3 calls 1m 6s
🟢 act completes Modal Flow 1 step · 4 calls 20.1s
🟢 act completes Bottom Tabs 1 step · 3 calls 10.9s
🟢 act completes Text Input Variations 1 step · 5 calls 29.2s
🟢 act completes Broken Accessibility 2 steps · 7 calls 47.1s
🟢 act completes Vision Only 2 steps · 6 calls 43.8s
⚠️ act completes Gestures (2 failed attempts first) 1 step · 9 calls 42.0s
🟢 act completes Async States 1 step · 7 calls 31.8s
🟢 act completes Debounced Search 1 step · 5 calls 30.6s
🟢 act completes Flattened Registration Form 2 steps · 14 calls 1m 44s
🟢 act completes Flattened Login 2 steps · 9 calls 1m 12s
🟢 act completes Sticky Chrome Target 1 step · 8 calls 53.7s
🟢 act completes WebView Accessibility 1 step · 3 calls 16.6s
🟢 act completes Permission Prompt 1 step · 3 calls 17.7s
🟢 act completes Photo Picker 1 step · 4 calls 17.4s
🟢 act completes Product Catalog 1 step · 5 calls 15.9s
🟢 act completes Error Recovery 1 step · 6 calls 16.4s
🟢 act completes Choice Controls 1 step · 7 calls 34.8s
🟢 act completes Stripe PaymentSheet 1 step · 5 calls 44.5s
🟢 act completes Apple Pay 1 step · 4 calls 24.4s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

e2e 0.15.2 · 15m 36s · ios-simulator · run artifacts

@github-actions
github-actions Bot force-pushed the changeset-release/main branch from 2cd541e to 67ee185 Compare October 2, 2026 10:09
@okwasniewski
okwasniewski merged commit 17f18c5 into main Oct 2, 2026
4 checks passed
@okwasniewski
okwasniewski deleted the changeset-release/main branch October 2, 2026 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant