chore: version packages - #715
Conversation
1ec0f7b to
2cd541e
Compare
🟢 e2e web: 90 passedAll 90 tests in 41 files
e2e 0.15.2 · 1m 50s · web · run artifacts |
🟢 e2e web agent: 32 passed33 agent steps · 25 replayed from cache · 20 model calls · 76.2k tokens (84% cached) · $0.0048 All 32 tests in 5 files
e2e 0.15.2 · 2m 50s · web · run artifacts |
🟢 e2e android agent: 20 passed24 agent steps · 108 model calls · 590.8k tokens (82% cached) · $0.0398 All 20 tests in 1 file
e2e 0.15.2 · 8m 42s · android-emulator · run artifacts |
🟢 e2e android: 1 flaky, 53 passed, 1 skipped
|
| Test | Time | |
|---|---|---|
| tests/device.e2e.ts · 1 flaky, 3 passed | 40.6s | |
| 🟢 | permission prompt › allowing the system dialog enables the microphone | 6.7s |
| 🟢 | permission prompt › a denied prompt recovers once the permission is granted outside | 14.4s |
| sequential onboarding echoes the exact values (1 failed attempt first) | 13.1s | |
| 🟢 | onboarding rejects a malformed email before advancing | 6.5s |
| ⏭️ | tests/controls.e2e.ts · 24 passed, 1 skipped | 2m 54s |
| 🟢 | control inventory › a label query resolves a labeled text field | 22.4s |
| 🟢 | control inventory › a labeled field answers name, display value, value reads, fill, and clear | 12.7s |
| 🟢 | control inventory › a placeholder query and attribute reads find the field | 5.2s |
| 🟢 | control inventory › a secure field takes a fill and denies value reads | 6.6s |
| 🟢 | control inventory › press sends a character and Enter through the soft keyboard | 8.4s |
| 🟢 | control inventory › a field autofocuses on mount and focus() moves the focus | 10.1s |
| 🟢 | control inventory › toBeFocused follows the focus | 9.7s |
| 🟢 | control inventory › a switch checks, unchecks, and reports its state | 10.3s |
| 🟢 | control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap | 7.0s |
| 🟢 | control inventory › checkbox and radio roles check, uncheck, and report their state | 11.4s |
| 🟢 | control inventory › a tab role answers getByRole | 3.4s |
| 🟢 | control inventory › a radio group reports the checked option | 9.0s |
| 🟢 | control inventory › buttons report disabled, enabled, and a state that arrives later | 7.9s |
| 🟢 | control inventory › a display: none element is absent rather than hidden, its visible twin attached | 6.4s |
| 🟢 | control inventory › a header answers its accessible name | 3.7s |
| 🟢 | control inventory › a tab row answers count, all, texts, filters, positions, and switching | 5.3s |
| 🟢 | control inventory › a list answers counts, list-form text, reads, and boxes in order | 4.8s |
| ⏭️ | control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…) | |
| 🟢 | control inventory › a long press lands as one | 6.7s |
| 🟢 | control inventory › a double tap lands inside the 300 ms window | 7.8s |
| 🟢 | control inventory › an ambiguous locator fails at once | 4.9s |
| 🟢 | control inventory › screenshot, restart, and clearState reopen the app on its home list | 7.9s |
| 🟢 | refusals › verbs a device never declares are refused before the engine | 2ms |
| 🟢 | refusals › keys the soft keyboard cannot send are refused by the engine | 1.2s |
| 🟢 | refusals › a missing locator fails as not found once its timeout passes | 1.1s |
| 🟢 | tests/device-fixture.e2e.ts · 8 passed | 1m 5s |
| 🟢 | device fixture › setOrientation rotates the window the app measures | 5.0s |
| 🟢 | device fixture › home leaves the app, openApp brings it back where it was | 5.6s |
| 🟢 | device fixture › foregroundApp sees the home screen after home() | 3.9s |
| 🟢 | device fixture › closeApp then openApp relaunches on the home list | 6.4s |
| 🟢 | device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN | 4.2s |
| 🟢 | device fixture › device.back pops the scenario | 4.7s |
| 🟢 | device fixture › setNetwork and setAirplaneMode change what the app sees | 15.2s |
| 🟢 | device fixture › setLocation and clearLocation change what the app reads | 19.9s |
| 🟢 | tests/flows-lists.e2e.ts · 7 passed | 2m 16s |
| 🟢 | infinite scroll reaches the target item | 23.4s |
| 🟢 | product catalog adds exactly two of the right variant | 9.8s |
| 🟢 | gestures: long-press, then swipe left | 7.8s |
| 🟢 | gestures: double-tap completes the flow | 9.3s |
| 🟢 | sticky chrome: accept below the fold, then continue | 11.0s |
| 🟢 | async states: load, pull to refresh, claim through the toast | 10.9s |
| 🟢 | huge virtualized list reaches row 512 | 1m 3s |
| 🟢 | tests/flows.e2e.ts · 7 passed | 50.4s |
| 🟢 | text inputs unlock the submit button | 6.3s |
| 🟢 | filled text inputs report their values | 5.6s |
| 🟢 | modal flow confirms through the native alert | 8.0s |
| 🟢 | bottom tabs act inside the last tab | 6.0s |
| 🟢 | error recovery retries, then confirms the delete | 8.7s |
| 🟢 | choice controls place the exact order | 7.8s |
| 🟢 | debounced search selects the target once results arrive | 8.1s |
| 🟢 | tests/login-form.e2e.ts · 3 passed | 29.5s |
| 🟢 | login form › rejects a malformed email | 8.6s |
| 🟢 | login form › rejects an unknown account | 8.6s |
| 🟢 | login form › signs in and logs out | 12.3s |
| 🟢 | tests/smoke.e2e.ts · 1 passed | 5.5s |
| 🟢 | home lists the scenarios and opens one | 5.5s |
e2e 0.15.2 · 8m 36s · android-emulator · run artifacts
🟢 e2e ios: 1 flaky, 51 passed, 1 skipped
|
| Test | Time | |
|---|---|---|
| tests/controls.e2e.ts · 1 flaky, 22 passed, 1 skipped | 3m 52s | |
| 🟢 | control inventory › a label query resolves a labeled text field | 13.0s |
| 🟢 | control inventory › a labeled field answers name, display value, value reads, fill, and clear | 20.6s |
| 🟢 | control inventory › a placeholder query and attribute reads find the field | 6.1s |
| 🟢 | control inventory › a secure field takes a fill and denies value reads | 11.6s |
| 🟢 | control inventory › press sends a character and Enter through the soft keyboard | 10.2s |
| 🟢 | control inventory › a field autofocuses on mount and focus() moves the focus | 13.0s |
| control inventory › toBeFocused follows the focus (2 failed attempts first) | 13.7s | |
| 🟢 | control inventory › a switch checks, unchecks, and reports its state | 13.3s |
| 🟢 | control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap | 10.3s |
| 🟢 | control inventory › checkbox and radio roles check, uncheck, and report their state | 16.2s |
| 🟢 | control inventory › a radio group reports the checked option | 11.0s |
| 🟢 | control inventory › buttons report disabled, enabled, and a state that arrives later | 9.6s |
| 🟢 | control inventory › a display: none element is absent rather than hidden, its visible twin attached | 9.7s |
| 🟢 | control inventory › a header answers its accessible name | 5.4s |
| 🟢 | control inventory › a tab row answers count, all, texts, filters, positions, and switching | 8.7s |
| 🟢 | control inventory › a list answers counts, list-form text, reads, and boxes in order | 9.6s |
| ⏭️ | control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…) | |
| 🟢 | control inventory › a long press lands as one | 9.9s |
| 🟢 | control inventory › pointer verbs land at the point asked for | 15.9s |
| 🟢 | control inventory › an ambiguous locator fails at once | 7.4s |
| 🟢 | control inventory › screenshot, restart, and clearState reopen the app on its home list | 15.7s |
| 🟢 | refusals › verbs a device never declares are refused before the engine | 2ms |
| 🟢 | refusals › keys the soft keyboard cannot send are refused by the engine | 350ms |
| 🟢 | refusals › a missing locator fails as not found once its timeout passes | 1.2s |
| 🟢 | tests/device-fixture.e2e.ts · 9 passed | 1m 43s |
| 🟢 | device fixture › setAppearance flips the color scheme the app reads | 16.6s |
| 🟢 | device fixture › setOrientation rotates the window the app measures | 7.8s |
| 🟢 | device fixture › the clipboard reads back what was written | 7.5s |
| 🟢 | device fixture › home leaves the app, openApp brings it back where it was | 7.2s |
| 🟢 | device fixture › closeApp then openApp relaunches on the home list | 10.3s |
| 🟢 | device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN | 6.5s |
| 🟢 | device fixture › device.back pops the scenario | 7.4s |
| 🟢 | device fixture › setLocation and clearLocation change what the app reads | 15.0s |
| 🟢 | device fixture › enrollBiometrics and setBiometrics answer a biometric prompt | 24.9s |
| 🟢 | tests/device.e2e.ts · 2 passed | 1m 2s |
| 🟢 | sequential onboarding echoes the exact values | 46.2s |
| 🟢 | onboarding rejects a malformed email before advancing | 15.6s |
| 🟢 | tests/flows-lists.e2e.ts · 7 passed | 3m 49s |
| 🟢 | infinite scroll reaches the target item | 29.8s |
| 🟢 | product catalog adds exactly two of the right variant | 12.3s |
| 🟢 | gestures: long-press, then swipe left | 10.9s |
| 🟢 | sticky chrome: accept below the fold, then continue | 16.1s |
| 🟢 | async states: load, pull to refresh, claim through the toast | 13.3s |
| 🟢 | huge virtualized list reaches row 512 | 2m 12s |
| 🟢 | web view coupon form applies the code shown on the page | 14.5s |
| 🟢 | tests/flows.e2e.ts · 7 passed | 1m 21s |
| 🟢 | text inputs unlock the submit button | 20.6s |
| 🟢 | filled text inputs report their values | 10.3s |
| 🟢 | modal flow confirms through the native alert | 10.5s |
| 🟢 | bottom tabs act inside the last tab | 8.3s |
| 🟢 | error recovery retries, then confirms the delete | 11.0s |
| 🟢 | choice controls place the exact order | 9.2s |
| 🟢 | debounced search selects the target once results arrive | 11.0s |
| 🟢 | tests/login-form.e2e.ts · 3 passed | 46.7s |
| 🟢 | login form › rejects a malformed email | 12.4s |
| 🟢 | login form › rejects an unknown account | 12.7s |
| 🟢 | login form › signs in and logs out | 21.6s |
| 🟢 | tests/smoke.e2e.ts · 1 passed | 8.4s |
| 🟢 | home lists the scenarios and opens one | 8.4s |
e2e 0.15.2 · 6m 53s · ios-simulator · run artifacts
🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped25 agent steps · 125 model calls · 48.1k tokens (84% cached) · $0.0032
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped | 25 steps · 125 calls | 13m 34s | |
| 🟢 | act completes Login Form | 1 step · 8 calls | 1m 15s |
| 🟢 | act completes Infinite Scroll List | 1 step · 3 calls | 1m 6s |
| 🟢 | act completes Modal Flow | 1 step · 4 calls | 20.1s |
| 🟢 | act completes Bottom Tabs | 1 step · 3 calls | 10.9s |
| 🟢 | act completes Text Input Variations | 1 step · 5 calls | 29.2s |
| 🟢 | act completes Broken Accessibility | 2 steps · 7 calls | 47.1s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 43.8s |
| act completes Gestures (2 failed attempts first) | 1 step · 9 calls | 42.0s | |
| 🟢 | act completes Async States | 1 step · 7 calls | 31.8s |
| 🟢 | act completes Debounced Search | 1 step · 5 calls | 30.6s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 14 calls | 1m 44s |
| 🟢 | act completes Flattened Login | 2 steps · 9 calls | 1m 12s |
| 🟢 | act completes Sticky Chrome Target | 1 step · 8 calls | 53.7s |
| 🟢 | act completes WebView Accessibility | 1 step · 3 calls | 16.6s |
| 🟢 | act completes Permission Prompt | 1 step · 3 calls | 17.7s |
| 🟢 | act completes Photo Picker | 1 step · 4 calls | 17.4s |
| 🟢 | act completes Product Catalog | 1 step · 5 calls | 15.9s |
| 🟢 | act completes Error Recovery | 1 step · 6 calls | 16.4s |
| 🟢 | act completes Choice Controls | 1 step · 7 calls | 34.8s |
| 🟢 | act completes Stripe PaymentSheet | 1 step · 5 calls | 44.5s |
| 🟢 | act completes Apple Pay | 1 step · 4 calls | 24.4s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
e2e 0.15.2 · 15m 36s · ios-simulator · run artifacts
2cd541e to
67ee185
Compare
This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.
Releases
@e2e-dev/[email protected]
Minor Changes
#730
1554360Thanks @szdziedzic! -easSimulators()authenticates with the eas-cli login whenEXPO_TOKENis not set: it reads the sessioneas loginkeeps in~/.expo/state.json(underUSERPROFILEon Windows) and sends it in theexpo-sessionheader, as eas-cli does, so a machine signed in to eas-cli needs no token.EXPO_TOKENstill wins when both are there. A session is stopped as the account that started it, even if the login changes during the run. Only the production login is read, since the sessions API isapi.expo.dev. With neither a token nor a login, the lease fails withEXPO_TOKEN is not set and eas-cli is not logged in.#731
88cef2cThanks @szdziedzic! -easSimulators()takesprojectIdfrom the app config when the option is absent:extra.eas.projectId, the ideas initwrites, read fromapp.config.jsonorapp.jsonbesidee2e.config.ts, or from a dynamicapp.config.ts(or.js,.mjs,.cjs,.mts,.cts) as the project's ownexpo config --type publicevaluates it in the run's environment, without.envfiles, as eas-cli runs it. It reads the config once per run, before the first session starts; a config that links no project fails the lease naming the file.projectIdis optional now and still wins when it is passed. Reading it needs the@e2e-dev/mobilethat passesprojectRootto device providers.@e2e-dev/[email protected]
Minor Changes
#731
88cef2cThanks @szdziedzic! - ADeviceProvider'sacquirerequest carriesprojectRoot, the directory the config's relative paths resolve against, so a provider reads project files there instead ofprocess.cwd().#740
bb904d4Thanks @NathanWalker! -device.fold(pose)puts a foldable iOS simulator (iPhone Duo) in a hinge pose:'closed'lights the outer display,'half-open'(a 130° book) and'open'the inner one. It runs agent-device'sfoldcommand, which sends the simulator's hinge event and reads the angle back from CoreDevice before it resolves, so the next observation reads the app on the other display. LikesetOrientation, it counts as an action for thetransitionbudget. A device without a hinge isUNSUPPORTED_CAPABILITY, and Android is refused before any device command. The pose type is exported asFoldPose.[email protected]
Patch Changes
#710
6b25a3aThanks @okwasniewski! - A failure where no model answered (MODEL_PROVIDER_FAILED,MODEL_UNAVAILABLE) no longer evicts replay cache entries. It says nothing about the app, so a read-write run without model access keeps every recording it did not confirm, including a step that replayed whole before anagent.assertthat could not reach its model and a replay that handed off to the model mid-step, and still saves the steps verified before the failure.#736
c9c3e14Thanks @programmrz! - Agent steps on DeepSeek's V4 models no longer fail on their first turn withMODEL_PROVIDER_FAILED. In thinking mode, DeepSeek's default, the API refuses a forced tool choice withThinking mode does not support this tool_choice; that refusal now reads as one, so the step retries withautoand the tool-calls-only rule, as it already did for Anthropic, and thinking can stay on.#735
28b09f3Thanks @okwasniewski! -e2e initaddszodnext toaiwhen it sets up a model gateway.zodis a peer ofaiand every provider package; npm and pnpm install peers on their own, Yarn does not, so a Yarn project failed to load the generated config withCONFIG_LOAD_FAILED. A Yarn project scaffolded with a model gateway before this fix needsyarn add -D zod.#708
4ee772cThanks @okwasniewski! - The package ships the documentation pages: coding agents can read them offline fromnode_modules/e2e/docs.#742
4f17480Thanks @okwasniewski! -expect(value).toBeDefined()now passes onnull, as it does in Playwright, Jest, and Vitest; onlyundefinedfails it. It used to rejectnulltoo, soexpect(null).not.toBeDefined()passed where Playwright fails.expect.poll(read).toBeDefined()follows: a read that returnsnullnow passes on the first try instead of waiting. For the old non-nullish check, usetoEqual(expect.anything()).#739
8d38206Thanks @NathanWalker! - The agent loop handles providers that downgrade a forced tool choice instead of refusing it, as the AI SDK's Anthropic provider does for Claude models that reject forced tool use (Claude Sonnet 5.5 through@ai-sdk/anthropic). Such a request succeeds withauto, so the HTTP 400 path never ran: the loop kept asking for a forced choice, the SDK logged anAI SDK Warning ... toolChoiceline on every turn, steps ran without the tool-calls-only instruction, and a turn the model answered in prose failed the step withMODEL_PROVIDER_FAILED(the SDK'sToolChoiceViolationError). Now atoolChoicecall warning switches the step toautoand gives its remaining turns the instruction, a prose answer under a forced choice is retried withautolike a refusal, and later steps on that model start inauto.