Completed browser-agent experiment

The Map of
Browser Agent Testing

A short route from “the driver finished” to “the product is actually correct.”

Current call Keep Playwright and Maestro as regression baselines.

Use adaptive tools for exploration, then verify their outcome independently. No overall replacement has qualified yet.

Browse all 54 tools · Read the eight experiment records

Source: the runner, adapters, experiment records, and replay instructions are public on GitHub.

Clone

Give the agent the runner

Keep the shared runner beside your application. Do not copy it into the product.

gh repo clone sarthakagrawal927/agent-testing
cd agent-testing
npm test

Prove it

Run the built-in smoke benchmark

The fixture demonstrates warm and cold runs, bounded phases, sanitized receipts, and a separate correctness oracle.

node bin/agent-testing.mjs validate \
  --manifest fixtures/good.manifest.json

node bin/agent-testing.mjs run \
  --manifest fixtures/good.manifest.json \
  --out artifacts/quickstart

node bin/agent-testing.mjs summarize \
  --receipt artifacts/quickstart/receipt.json

Adapt

Add one real product journey

Create adapters/<your-product>/ with a strict manifest and bounded commands for reset, readiness, workflow, and verification.

adapters/your-product/
├── README.md
├── search.playwright.manifest.json
├── run-search.mjs
└── verify-search.mjs

The one rule A zero exit code is not a correct product state.

Verify displayed values, persisted backend state, reload or relaunch behavior, and selected screenshots outside the driver.

Compare

Use the tool that fits the job

Job	Use now	Boundary
Repeat web changes	Playwright	Saved workflows remain the qualified baseline.
Explore web behavior	Jev + verifier	Search worked 5/5; password login is unsupported.
Replay a semantic action	Stagehand cache	Promising 4/4 screening result; not a 2× win.
Repeat iOS changes	Maestro	AXe was faster in login screening but not reliable enough overall.

Playwright search2.768 s5/5 verified

Stagehand cached2.201 s4/4 preliminary

Jev search11.101 s5/5 + stale fault found

These are Vaultwealth screening results, not universal rankings. Sample sizes are preliminary and prepared workflows are not equivalent to unfamiliar exploration tasks.

Adopt

Raise the bar only after screening

Start with five warm runs. Promote promising candidates to 20 warm and three cold runs per journey.

Adoption gate

At least 2× faster, or 50% less model usage without slower execution—plus no missed seeded defects and no observed reliability regression.