# The Map of Browser Agent Testing

> A command-first map for coding agents comparing local web and native UI testing tools with independent verification.

Repository: https://github.com/sarthakagrawal927/agent-testing
Live map: https://browser-agents.sarthakagrawal.dev
Tool catalogue: https://browser-agents.sarthakagrawal.dev/tools
Experiment record: https://browser-agents.sarthakagrawal.dev/experiments
Machine-readable tools: https://browser-agents.sarthakagrawal.dev/tools.json
Machine-readable experiments: https://browser-agents.sarthakagrawal.dev/experiments.json
Exact observed pins: https://browser-agents.sarthakagrawal.dev/versions.json
Access: The source repository is public and can be cloned without authorization.

Last experiment: 2026-09-20
Catalogue reviewed: 2026-09-20
Status: completed experiment; retained without an active expansion roadmap.
Coverage: 79 tools in seven categories. Eleven tools were repeatedly benchmarked, ten received bounded screens, four have dated setup blockers, and the remaining source-reviewed entries have explicit scope or fit boundaries.
Quantitative matrix: 26 measured web/iOS journey arms with 118 verified completions across 129 attempts, plus seven readiness, diagnostic and local-model probes. Five seeded defect types. Estimated accepted paid API spend across the Jev evidence: $0.00479976.

## Start

gh repo clone sarthakagrawal927/agent-testing
cd agent-testing
npm test

node bin/agent-testing.mjs validate --manifest fixtures/good.manifest.json
node bin/agent-testing.mjs run --manifest fixtures/good.manifest.json --out artifacts/quickstart
node bin/agent-testing.mjs summarize --receipt artifacts/quickstart/receipt.json

## Add a product

Keep this repository beside the product. Add adapters/<your-product>/ with a strict manifest and bounded reset, readiness, workflow, and verification commands. Use a seeded non-production backend and a clean isolated product worktree.

The workflow and verifier must be separate. A successful driver exit is not proof of a correct displayed value, backend state, reload or relaunch result, or visual state.

## Current screening guidance

- Repeat web changes: saved Playwright workflows.
- Compare deterministic web drivers: Puppeteer was the fastest added screen at 2.081 s median, but did not meet the 2x adoption gate; Selenium, WebdriverIO, Nightwatch and Taiko also completed 5/5 clean runs and detected stale search.
- Explore supported web behavior: Jev plus an independent verifier.
- Replay a previously resolved semantic web action: Stagehand cache remains promising but unqualified as a replacement.
- Repeat iOS changes: saved Maestro flows.
- Targeted iOS inspection/action: AXe or XcodeBuildMCP, without assuming standalone reliability.

No overall replacement has qualified. Adopt a candidate only after it is at least 2x faster, or uses at least 50% less model without slower execution, with no missed seeded defects and no observed reliability regression.

The complete numeric matrix is in experiments.json under journey_comparisons and probe_comparisons. Every journey row includes verified_passes and attempts; null timing values mean the retained report did not establish that number.

## Exact observed versions

- Playwright 1.62.1
- agent-browser 0.38.1
- @playwright/cli 0.1.21
- Puppeteer Core 24.43.1
- Selenium WebDriver 4.49.0 with ChromeDriver 151.0.5
- WebdriverIO 9.31.9
- Nightwatch 3.16.0
- Taiko 1.5.0
- TestCafe 3.7.6 (bounded setup blocker)
- Cypress 16.1.0 package (application binary absent)
- Stagehand 3.4.0
- Browser Use 0.13.10
- chrome-devtools-mcp 1.9.0
- Jev 0.1.0 at commit 1231850a0bf1a0c0341fe408ef1668dbbfdfac46
- AXe 1.8.0
- Maestro 2.6.1 with Java 21
- XcodeBuildMCP 2.7.0
- ios-simulator-mcp 2.1.0
- Mobile MCP 1.0.4 with temporary device agent 0.0.26
- fb-idb 1.1.7 in an isolated experiment environment
- Chromium 151.0.7922.34
- Codex 0.155.1 using gpt-6-astra at medium reasoning
- Bonsai model revision 6ed5e12bf84b7a63069882c91dd9e9218647d17b
- Prism llama.cpp 0.2.0-dev build 10709 at commit 9a9394a895b96003ca842a6041cb28ac49a108f7

See versions.json for hashes and observed environment pins. Source-reviewed entries state an applicability boundary instead of inventing a runtime result.

Detailed guide: https://github.com/sarthakagrawal927/agent-testing/blob/main/docs/GETTING_STARTED.md
Full report: https://github.com/sarthakagrawal927/agent-testing/blob/main/adapters/vaultwealth/runtime/REPORT.md