Completed experiment · experiment record

What was actually run.

Vaultwealth web and iOS Simulator, seeded local fixtures, one Mac, bounded runs, and an independent correctness oracle.

Last experiment
20 September 2026
Current decision
Keep Playwright + Maestro

Bottom line

Keep Playwright and Maestro as regression baselines. Use adaptive tools for exploration with an independent verifier.

Promising results remain screening evidence. None completed the planned 20 warm and three cold qualification set across every journey.

Coverage in numbers

Tools mapped79
Tools executed21
Measured journey arms26
Verified journey attempts118/129
Seeded defect types5
Accepted paid API spend$0.00480

Executed means benchmarked or bounded-screened locally. The other 54 catalogue entries retain source-backed fit boundaries; they are not assigned synthetic scores.

Measured journey matrix 26 arms

Median and observed p95 include the timing basis shown under each mode. At these sample sizes, observed p95 is usually the maximum. A dash means the report did not establish that number.

PlatformJourneyModeMedianObserved p95VerifiedModel / costSeeded fault
Web Login Playwright savedverified passes 1.178 s 1.400 s 5/5 0 calls · $0 clipped control detected
Web Login agent-browser directverified passes 2.714 s 3.846 s 5/5 0 calls · $0 clipped control detected
Web Login agent-browser batchverified passes 1.323 s 1.734 s 5/5 0 calls · $0 clipped control detected
Web Edit + reload Playwright savedverified passes 1.898 s 1.950 s 5/5 0 calls · $0 3/3 write faults detected
Web Edit + reload agent-browser directverified passes 1.762 s 1.795 s 5/5 0 calls · $0 3/3 write faults detected
Web Edit + reload agent-browser batchno verified completion — — 0/5 0 calls · $0 not eligible after clean failure
Web Search Playwright savedverified passes 2.768 s 2.796 s 5/5 0 calls · $0 stale result detected
Web Search agent-browser directverified passes 2.826 s 2.833 s 5/5 0 calls · $0 stale result detected
Web Search agent-browser batchverified passes 2.649 s 2.836 s 5/5 0 calls · $0 stale result detected
Web Search Puppeteer Coreverified passes 2.081 s 2.146 s 5/5 0 calls · $0 stale result detected
Web Search WebdriverIOverified passes 2.958 s 3.087 s 5/5 0 calls · $0 stale result detected
Web Search Selenium WebDriververified passes 3.633 s 3.651 s 5/5 0 calls · $0 stale result detected
Web Search Nightwatchverified passes 5.425 s 5.700 s 5/5 0 calls · $0 stale result detected
Web Search Taikoverified passes 23.850 s 23.921 s 5/5 0 calls · $0 stale result detected
Web Search Stagehand cachedverified replays 2.201 s 2.233 s 4/4 0 replay calls · $0 stale result detected
Web Search Stagehand uncachedverified passes 26.951 s 31.401 s 5/5 3 local calls · $0 stale result detected
Web Search Jev + Bonsaiverified passes 11.101 s 11.391 s 5/5 7 paid selector + 3 local calls · ~$0.00063/run stale result detected
Web Search Browser Use + Bonsaisuccessful verified passes 90.730 s 93.623 s 4/5 8 local calls on successes · $0 stale result detected
Web Login Playwright CLIverified passes; p95 unavailable 3.235 s — 5/5 0 calls · $0 no fault sample
Web Edit + reload Playwright CLIverified passes; p95 unavailable 3.076 s — 5/5 0 calls · $0 no fault sample
Web Adaptive login Codex stepwiseverified passes 155.971 s 225.686 s 5/5 918,042 median input tokens clipped control detected separately
Web Adaptive login Codex batchedall attempts; 2/5 functional 123.684 s 139.657 s 0/5 398,585 median input tokens clipped calibration noticed; unqualified
iOS Login Maestro saved flowverified warm passes 32.964 s 34.775 s 5/5 0 calls · $0 clipped control detected
iOS Login AXe batchverified warm passes 12.612 s 12.892 s 5/5 0 calls · $0 clipped control detected
iOS Search Maestro saved flowverified warm passes 34.673 s 41.146 s 5/5 0 calls · $0 no fault sample
iOS Search AXe batch, one-char typingverified warm passes 15.279 s 15.317 s 5/5 0 calls · $0 no fault sample

Other measured probes 7

These numbers are useful, but they are not comparable end-to-end application journeys.

ProbeMedianObserved p95PassesNumeric detailBoundary
Bonsai text helper · budget-only 4.907 s 8.943 s 25/25 9.252 GB sampled RSS · 4,760 prompt / 2,525 output tokens Synthetic text-helper requests
Bonsai text helper · no-thinking 1.082 s 1.193 s 25/25 7.305 GB sampled RSS · 3,860 prompt / 215 output tokens Synthetic text-helper requests
Chrome DevTools MCP diagnostic bundle 0.826 s — 5/5 Lighthouse snapshot 1.243 s Diagnostic collection, not a journey
XcodeBuildMCP cash-field action 3.834 s — 5/5 1.343 s semantic observation median · ~35 s setup No save/relaunch qualification
AXe app-process relaunch readiness — — 0/3 standalone Launch 1.52–1.64 s · Maestro-assisted tree 6.12–6.23 s Startup diagnostic, not cold journey
ios-simulator-mcp readiness — — passed 17 tools · 6,940-character accessibility description Readiness only
Mobile MCP readiness — — passed 32 tools · 27 elements listed Readiness only; temporary agent removed

Clone and replay

gh repo clone sarthakagrawal927/agent-testing
cd agent-testing
npm test

# Validate the real-product adapter
npm run validate:vaultwealth

# Read exact setup and replay commands
open adapters/vaultwealth/runtime/README.md

The source repository is public. Product adapters must still use local or disposable seeded targets only.

Experiment 01 · complete screening

Saved web workflows

Arms: Playwright · agent-browser direct · agent-browser batch

Playwright remained the baseline. agent-browser batching improved its own login time, but no candidate met the 2x adoption threshold.

  • Login: Playwright 1.178 s; direct 2.714 s; batch 1.323 s medians; all 5/5.
  • Edit: Playwright 1.898 s and direct 1.762 s medians; batch failed 0/5.
  • Search: Playwright 2.768 s; direct 2.826 s; batch 2.649 s medians; all 5/5.

Evidence: adapters/vaultwealth/runtime/REPORT.md#web-saved-workflows-no-ai

Experiment 02 · complete screening

Seeded web defects

Arms: Playwright oracle · agent-browser direct · agent-browser batch where viable

The viable Playwright and direct arms detected all five seeded fault types with no clean-run false failure in the selected trials.

  • Detected save failure, wrong persisted value, duplicate write, stale search and clipped control.
  • The visual oracle detected 53.14% changed control pixels against a 2% limit.
  • This validates fixed oracles; it is not a blinded AI discovery rate.

Evidence: adapters/vaultwealth/runtime/REPORT.md#seeded-web-defects

Experiment 03 · complete screening

Coding-agent stepwise versus batched interaction

Arms: Stepwise Codex · Batched Codex

Stepwise passed 5/5. Batched qualified 0/5 and reduced total reported input mostly through cache, not uncached reasoning.

  • Stepwise verified median 155.971 s; 5/5.
  • Batched median attempt 123.684 s; only 2/5 functional completions and 0/5 within the recovery gate.
  • Total input fell 56.6%, while uncached input fell 8.6%.

Evidence: adapters/vaultwealth/runtime/REPORT.md#agent-interaction-strategies

Experiment 04 · complete screening

Jev specialist with local Bonsai text helper

Arms: Jev + TypeSafe selection + Bonsai text · Fixed Playwright/backend oracle

Authenticated search passed 5/5 and found the hidden stale-search fault. Login is unsupported because Jev excludes password inputs.

  • Clean search median 11.101 s; observed p95 11.391 s.
  • Seven TypeSafe calls plus three local Bonsai calls per run.
  • Estimated TypeSafe spend $0.00313047 across five clean runs; about $0.00063 per run.

Evidence: adapters/vaultwealth/runtime/REPORT.md#optional-jev-specialist-with-local-bonsai

Experiment 05 · screening incomplete

iOS saved flows: Maestro versus AXe

Arms: Maestro saved flow · AXe batch

AXe was much faster on warm login and search, but cold accessibility readiness and inconsistent text entry prevented replacement.

  • Login: Maestro 32.964 s versus AXe 12.612 s medians; both 5/5.
  • Search: Maestro 34.673 s versus AXe 15.279 s medians; both 5/5 with journey-specific AXe typing configuration.
  • Three app-process relaunch probes returned an empty AXe tree until a Maestro hierarchy request.

Evidence: adapters/vaultwealth/runtime/REPORT.md#native-saved-workflows-and-startup-diagnostic

Experiment 06 · complete synthetic screening

Local Bonsai 27B text-helper compatibility

Arms: Reasoning budget only · Explicit no-thinking template

Both profiles passed 25/25 exact responses. Explicit no-thinking was 4.54x faster and used less sampled memory.

  • Budget-only median 4.907 s; observed p95 8.943 s; sampled peak RSS 9.252 GB.
  • No-thinking median 1.082 s; observed p95 1.193 s; sampled peak RSS 7.305 GB.
  • This was a text-helper probe, not a browser-agent or defect-detection result.

Evidence: adapters/vaultwealth/runtime/BONSAI.md

Experiment 07 · complete screening

Round-two local browser tools

Arms: Stagehand uncached · Stagehand cached · Browser Use · Playwright CLI · Chrome DevTools MCP

Stagehand cache was the most promising replay result, but missed the 2x gate. Browser Use was slow and 4/5. Chrome DevTools MCP was useful for diagnostics only.

  • Stagehand uncached 26.951 s median; 5/5. Cached 2.201 s median; 4/4 after 25.544 s authoring.
  • Browser Use 90.730 s median among successes; 4/5.
  • Chrome DevTools MCP diagnostic bundle 0.826 s median; Lighthouse snapshot 1.243 s.

Evidence: adapters/vaultwealth/runtime/ROUND2.md

Experiment 08 · bounded probe

Additional native candidates

Arms: XcodeBuildMCP · XCUITest feasibility · Detox feasibility

XcodeBuildMCP replaced the formatted cash field in 5/5 probes but no save/relaunch journey was qualified. XCUITest and Detox were setup-blocked by checkout structure.

  • XcodeBuildMCP action median 3.834 s; semantic observation median 1.343 s.
  • Maestro/debug setup still took about 35 s.
  • No claim is made that XCUITest or Detox failed as tools.

Evidence: adapters/vaultwealth/runtime/ROUND2.md#other-round-two-findings

Experiment 09 · complete screening

Expanded saved-web driver screen

Arms: Puppeteer Core · Selenium WebDriver · WebdriverIO · Nightwatch · Taiko

All five candidates completed five independently verified clean searches and detected the stale-search fault. Puppeteer was fastest, but did not meet the 2x adoption gate against Playwright.

  • Workflow medians: Puppeteer 2.081 s; WebdriverIO 2.958 s; Selenium 3.633 s; Nightwatch 5.425 s; Taiko 23.850 s.
  • Every candidate passed 5/5 clean runs and detected the stale-search mismatch through the backend/UI oracle.
  • TestCafe 3.7.6 hit the bounded 60 s setup timeout; Cypress 16.1.0 had no isolated application binary. Neither is counted as a tool failure.

Evidence: adapters/vaultwealth/runtime/EXPANDED_WEB.md

Experiment 10 · complete bounded screen

Expanded native agent-tool readiness

Arms: ios-simulator-mcp · Mobile MCP · fb-idb

Both MCP servers started and found the dedicated simulator. ios-simulator-mcp returned the accessibility tree through isolated idb; Mobile MCP listed 27 elements and its temporary device agent was removed afterward.

  • ios-simulator-mcp 2.1.0 exposed 17 tools and returned a 6,940-character accessibility description.
  • Mobile MCP 1.0.4 exposed 32 tools and listed 27 elements with device agent 0.0.26.
  • These are readiness probes, not saved-flow latency or defect-detection results.

Evidence: adapters/vaultwealth/runtime/EXPANDED_NATIVE.md

Exact observed versions

Apple M5 Pro, 48 GiB RAM, Darwin 27.0.0. Dedicated Chromium at 1280x800. iPhone 16 Pro, iOS 26.5.

Tool or runtimeExact observed pinRecorded from
Playwright1.62.1experiment package lock
agent-browser0.38.1experiment package lock
@playwright/cli0.1.21experiment package lock
Puppeteer Core24.43.1expanded screen package lock
Selenium WebDriver4.49.0expanded screen package lock
ChromeDriver151.0.5 / 151.0.7922.77expanded screen package lock and binary
WebdriverIO9.31.9expanded screen package lock
Nightwatch3.16.0expanded screen package lock
Taiko1.5.0expanded screen package lock
TestCafe3.7.6bounded setup attempt
Cypress16.1.0 package; application binary absentbounded setup probe
Stagehand3.4.0experiment package lock
Browser Use0.13.10isolated Python environment
chrome-devtools-mcp1.9.0experiment package lock
Jev Ultrafast0.1.0 @ 1231850a0bf1a0c0341fe408ef1668dbbfdfac46isolated source checkout
AXe1.8.0pinned release archive
Maestro2.6.1, Java 21installed CLI
XcodeBuildMCP2.7.0installed CLI
ios-simulator-mcp2.1.0expanded native readiness screen
Mobile MCP1.0.4; device agent 0.0.26expanded native readiness screen
fb-idb1.1.7isolated experiment virtual environment
Chromium151.0.7922.34dedicated experiment browser
Codex0.155.1, gpt-6-astra, medium reasoningagent run metadata
Ternary Bonsai 2 27B PTQ1_0revision 6ed5e12bf84b7a63069882c91dd9e9218647d17b, SHA-256 53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3pinned model artifact
Prism llama.cpp0.2.0-dev build 10709 @ 9a9394a895b96003ca842a6041cb28ac49a108f7, archive SHA-256 f9cdf245fb7b832f1996dd776b321d4ae1f23b6d88c380100f636742c3a980ffpinned runtime artifact
Node.js app/build24.21.0experiment environment
npm11.19.1experiment environment
Node.js controller26.9.0initial measurement controller
zod3.25.76experiment package lock
Wrangler4.135.0site package lock

These pins make the historical result reproducible. They are not recommendations to avoid newer versions.

What would justify rerunning

Rerun when the application journey changes, a candidate has a material new release, the browser or simulator changes, or a five-run screen beats the current reliability and verified-feedback result. Do not rerun the whole catalogue merely because another tool exists.