I Gave an AI Agent a Browser. Can It Actually Test a Web Application?

Playwright MCP vs Vibium vs Selenium + Robot Framework: A Hands-On Guide to Agentic QA

AI can interact with a browser. But can it produce trustworthy test evidence? Let’s investigate.

The experiment: one task, three approaches

Imagine giving an AI agent access to a web application and asking it to:

Log in with a valid account, create a new record, verify that it appears in the application, attempt an invalid submission, and produce a test report with evidence for every assertion.

The agent opens the browser. It clicks the right controls. It submits the form. It returns a polished report.

Everything looks convincing.

But suppose the record was never persisted, the validation message was incorrect, or the agent silently skipped one of the acceptance criteria.

Would you trust the report?

This is the fundamental challenge of agentic QA: an agent can successfully operate a browser without successfully testing the application.

In this article, we will examine three approaches to browser automation:

• Playwright with its MCP server.
• Vibium and its agent-oriented browser interface.
• Selenium with Robot Framework, alongside the alternative of Robot Framework Browser Library.

We will define a common experiment, explore practical configurations, design a fair benchmark, and identify where agent autonomy genuinely adds value.

The goal is not to crown a winner based on a five-minute demonstration. It is to establish a repeatable method for choosing the right architecture.

1. First, understand what you are actually comparing

These technologies operate at different layers. Treating them as interchangeable creates misleading comparisons.

Layer	Technology	Responsibility
Agent interaction	MCP-compatible client or agent	Interprets tasks and decides which tools to invoke
Browser interaction	Playwright, Vibium, Selenium	Navigates and interacts with the browser
Test organization	Robot Framework, Playwright Test	Structures scenarios, assertions, fixtures, and execution
Evidence	Assertions, logs, traces, reports	Establishes what actually happened
Governance	Permissions, policies, CI/CD controls	Limits actions and determines what can be trusted

Playwright MCP exposes browser operations to compatible agents. Vibium provides an agent-oriented interface through MCP, a CLI, and client libraries. Selenium provides browser automation APIs, while Robot Framework organizes tests through a keyword-driven framework.

Robot Framework also has two distinct browser libraries worth separating:

• SeleniumLibrary, built on Selenium.
• Browser Library, powered by Playwright.

That distinction matters. If your organization already uses Robot Framework, you can adopt Playwright-based browser automation without abandoning your existing test organization.

The real comparison is between complete testing approaches, not just three browser engines.

2. Define a reproducible test scenario

Before installing anything, define the expected behavior.

Use a dedicated demo application, a controlled local application, or an isolated test environment. Do not experiment with production data or real customer accounts.

Our sample scenario is a customer-registration workflow.

Preconditions

• The application is reachable.
• A dedicated test account is available if authentication is required.
• The test environment can be reset.
• The expected validation messages and business rules are documented.
• The test data is unique to each execution.

Test flow

Test A — Valid submission

1. Open the customer-registration page.
2. Complete all mandatory fields.
3. Submit the form.
4. Verify that a success state is displayed.
5. Verify that the created record exists in the application’s observable state.
6. Record the execution evidence.

Test B — Invalid submission

1. Return to a clean registration form.
2. Leave a mandatory field empty or enter invalid data.
3. Submit the form.
4. Verify the expected validation message.
5. Verify that the application does not create an invalid record.

Test C — Recovery

1. Simulate a temporary interruption or unexpected page state.
2. Observe how the automation responds.
3. Determine whether the scenario can recover safely.
4. Check that recovery did not duplicate the operation.

This gives us more than a successful navigation. It evaluates business behavior, negative testing, evidence, and resilience.

For each test, define explicit acceptance criteria before executing the agent.

For example:

• AC-01: A valid record is created.
• AC-02: The record remains visible after a page refresh.
• AC-03: An empty mandatory field triggers the specified validation.
• AC-04: Invalid input does not create a record.
• AC-05: Every reported result has identifiable evidence.

The agent may help execute the scenario, but it must not invent the expected results.

3. Approach A — Playwright MCP

Why it is interesting

Playwright MCP exposes browser operations through the Model Context Protocol. Its interaction model uses structured accessibility snapshots rather than relying exclusively on screenshots.

The agent can inspect the page, identify accessible elements, interact with them through tool calls, and observe the resulting state.

This is useful when the objective is to let an agent explore an interface dynamically rather than provide it with a fully scripted sequence of browser actions.

Official documentation:

• Playwright MCP installation
• Playwright MCP snapshots

Step 1 — Configure the MCP server

With Node.js 20 or newer and a compatible MCP client, the official configuration can look like this:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": ["@playwright/mcp@latest"]
    }
  }
}

Use the installation instructions for your specific client, and pin a reviewed package version in a reproducible environment rather than leaving the configuration permanently on latest.

Step 2 — Give the agent an explicit test contract

Do not simply write:

“Test customer registration.”

That instruction leaves too much room for interpretation.

Instead, define the expected behavior and evidence:

ROLE
You are a QA agent testing a customer-registration workflow.

OBJECTIVE
Verify the documented registration acceptance criteria.

RULES
- Use only the designated test environment.
- Do not use real customer information.
- Inspect the page before interacting with it.
- Do not invent acceptance criteria.
- Distinguish observed facts from assumptions.
- Never report PASS without evidence.

TESTS
1. Submit valid registration data.
2. Verify the success state.
3. Verify that the record persists after refresh.
4. Submit invalid data.
5. Verify the expected validation message.
6. Verify that invalid data was not accepted.

OUTPUT
For each acceptance criterion, return:
- PASS, FAIL, or BLOCKED
- The action performed
- The expected result
- The observed result
- The evidence reference
- Any unresolved uncertainty

This makes the task more precise without prescribing every browser interaction.

The agent retains freedom to discover the interface, but the acceptance criteria remain fixed.

Step 3 — Understand the accessibility snapshot

Playwright MCP can return a structured representation such as:

- heading "Customer registration" [level=1]
- textbox "Customer name" [ref=e4]
- textbox "Email address" [ref=e5]
- button "Create customer" [ref=e6]

The agent can use the element references returned by the tool to interact with the relevant controls.

This is an important architectural difference from purely visual interaction: the agent can reason about accessible names, roles, and the structure of the interface.

However, the snapshot does not prove that the business operation succeeded. It only provides information about the interface state.

The same distinction applies to screenshots: visual evidence is valuable, but it does not necessarily establish persistence or business correctness.

Step 4 — Separate browser interaction from verification

Suppose the agent clicks Create customer and observes a success notification.

That observation is not sufficient to establish AC-01 and AC-02.

The workflow should also verify the created record and its persistence through a suitable application-level check.

Possible evidence sources include:

• A stable record visible in the application after refresh.
• A read-only API query in the test environment.
• A database assertion in a controlled integration-test setup.
• An application event or execution record, where appropriate.

The strongest verification method depends on the requirement. A UI requirement may need a UI assertion; a persistence requirement may need a persistence-level assertion.

Playwright MCP provides the interaction interface. Your test architecture must establish what constitutes proof.

4. Approach B — Vibium

Why it deserves a place in the experiment

Vibium is designed around browser automation for AI agents and humans. Its documented interfaces include a CLI, MCP integration, and client libraries, with browser automation built on WebDriver BiDi.

Its run and check approach is particularly interesting for QA.

The first operation asks a model to accomplish a goal. The second asks a fresh verifier to investigate a claim and return a verdict with evidence.

This introduces a useful testing question:

Can an independent verification step catch an incorrect or incomplete result produced during the execution step?

Official documentation:

• Vibium documentation
• Vibium Run and Check

Step 1 — Configure the environment

Follow Vibium’s current installation instructions and configure a supported model provider.

Check the availability of the required commands in the version you install. At the time of writing, the documentation identifies the AI run and check commands as nightly functionality rather than assuming they are available in every stable release.

This is an important practical consideration: an attractive capability in a nightly build is not automatically a production-ready capability.

Step 2 — Execute a goal

For example, against a dedicated application running locally:

vibium go http://localhost:3000/account

If the installed build supports the documented AI commands, a goal can be expressed as:

vibium run "Update the test account timezone and save it"

The goal-based interaction is convenient because the agent can determine the sequence of browser actions required to complete the task.

But the completion status is the model’s assessment of its execution, not conclusive proof of the business outcome.

Step 3 — Verify a claim independently

A useful follow-up claim is:

vibium check "The saved timezone remains correct after refresh"

This is stronger than asking whether clicking Save succeeded.

Why? Because the verification concerns the resulting state rather than the action itself.

The independent verifier uses a fresh conversation, but it can still make mistakes. Independence at the conversation level does not guarantee independent evidence or correctness.

For stronger validation, compare the verdict with a deterministic assertion or another authoritative source of application state.

Step 4 — Preserve execution evidence

The documented workflow supports recording output:

vibium run "Update the test account timezone and save it" \
  -o run.zip

And verification can also produce a recording:

vibium check "The timezone remains correct after refresh" \
  -o check.zip

These commands are illustrative and depend on the installed build supporting the documented functionality.

Review the generated evidence rather than relying only on the terminal summary. Record which version, model, browser, and environment were used.

What this experiment should reveal

Vibium provides an interesting opportunity to investigate a two-stage model-driven workflow:

1. Execute a browser goal.
2. Ask a verifier to investigate the resulting claim.
3. Compare the verdict with independently defined acceptance criteria.

Measure how often the verification catches deliberately introduced defects, how often it produces an inconclusive result, and how much additional time and model usage it requires.

That will tell you more than simply observing whether the agent can fill in a form.

5. Approach C – Selenium + Robot Framework

Agentic automation should not be evaluated in isolation from the deterministic testing systems that organizations already operate.

For teams with Selenium assets and established regression pipelines, Robot Framework remains an important option.

SeleniumLibrary exposes Selenium through Robot Framework keywords. Browser Library offers a separate option based on Playwright.

Official documentation:

• Robot Framework SeleniumLibrary
• Robot Framework Browser Library

Step 1 — Install the Selenium-based stack

Create an isolated Python environment, then install the required packages:

python -m venv .venv

Activate the environment using the appropriate command for your operating system.

Then install SeleniumLibrary:

python -m pip install robotframework-seleniumlibrary

Selenium Manager can manage browser drivers automatically in supported configurations, although enterprise environments may need explicit browser and driver policies.

Step 2 — Write a deterministic test

Here is an illustrative Robot Framework test using the public login demo:

*** Settings ***
Library    SeleniumLibrary

*** Variables ***
${URL}       https://the-internet.herokuapp.com/login
${BROWSER}   chrome

*** Test Cases ***
Invalid Login Must Be Rejected
    Open Browser    ${URL}    ${BROWSER}
    Input Text      username    invalid-user
    Input Text      password    invalid-password
    Click Button    class:radius
    Page Should Contain    Your username is invalid!
    [Teardown]    Close Browser

The expected validation text is explicit, and the test fails when the assertion does not hold.

This example tests a login validation scenario, not the customer-registration workflow. To compare approaches fairly, implement equivalent acceptance criteria in the same application.

Step 3 — Consider Browser Library

If your team prefers Robot Framework’s keyword-driven structure but wants Playwright-based browser automation, Browser Library is worth evaluating.

Its setup and browser installation requirements depend on the chosen package configuration. Follow the current official instructions.

The important point is that adopting a different browser engine does not automatically require replacing the test framework, and adding AI does not require abandoning deterministic regression suites.

Step 4 — Introduce AI around the test framework

A practical agentic workflow can use an AI agent to:

• Interpret a requirement and propose candidate scenarios.
• Generate a first draft of Robot Framework tests.
• Suggest additional negative cases.
• Analyze an existing execution report.
• Identify likely causes of a failure.
• Propose a correction for human review.

The generated test should still pass code review and run through the conventional test pipeline.

If you choose to expose existing test capabilities through MCP or another tool interface, expose narrowly scoped operations rather than unrestricted shell access.

For example, an agent might be allowed to run an approved test suite and read its report, while being prevented from modifying pipeline configuration or accessing production credentials.

This is a more controlled path to adoption than granting an agent unrestricted authority over the QA environment.

6. Build the benchmark: compare outcomes, not demonstrations

Now comes the most important part.

A benchmark should be designed to distinguish genuine success from plausible-looking output.

The experimental design

For each approach:

• Use the same application and initial state.
• Use equivalent acceptance criteria.
• Keep browser versions and environment settings controlled.
• Record the model and relevant configuration.
• Run repeated trials.
• Introduce known failures deliberately in a separate test dataset.
• Preserve the full execution evidence.

Separate deterministic automation results from model-driven results. A conventional test that executes exactly the same script should not be treated as equivalent to an agent that chooses its actions dynamically without acknowledging the difference.

Metrics to collect

1. Task completion rate

How often does the workflow reach its intended endpoint?

2. Assertion accuracy

How often does the reported verdict match the actual result established by the test oracle?

3. False-pass rate

How often does the workflow report success when an acceptance criterion is not satisfied?

This is one of the most important metrics for agentic QA.

4. False-fail rate

How often does the workflow report failure when the acceptance criteria are actually satisfied?

5. Recovery rate

How often can the workflow recover from a controlled transient failure without unsafe side effects?

6. Evidence completeness

How many results include sufficient evidence for independent review?

7. Execution time and cost

Record wall-clock time, model usage, infrastructure costs, and the effort required to investigate failures.

8. Maintenance effort

Measure the engineering effort needed to update the workflow when the application changes.

Do not claim that one approach is faster or cheaper until you have measured it under equivalent conditions.

A small but useful experiment

Start with 20 runs per approach, if your budget allows, and include both valid and invalid scenarios.

That is enough for an initial exploratory benchmark, not enough to establish production-grade reliability.

Track results in a CSV file with fields such as:

approach,run_id,scenario,expected_verdict,actual_verdict,
evidence_complete,duration_seconds,token_usage,
recovery_attempts,notes

Use a controlled test oracle to determine the expected verdict. Do not use another model’s opinion as the sole ground truth.

Calculate false-pass and false-fail rates separately. A tool that reports PASS frequently may appear productive while failing to detect invalid states.

7. Test the verifier itself

An AI verification workflow needs its own tests.

Introduce controlled defects and check whether the system detects them.

Injected defect	Expected result
Success notification appears, but no record is persisted	FAIL
Required field is empty and validation is absent	FAIL
Valid submission succeeds and the record persists	PASS
Browser becomes unavailable before the assertion can be checked	BLOCKED or INCONCLUSIVE
The test environment returns contradictory evidence	Escalate or return INCONCLUSIVE

The precise verdict depends on the test contract. For example, an unavailable environment should not automatically be classified as a business defect.

This experiment evaluates whether the verifier is capable of rejecting misleading evidence, not simply confirming successful scenarios.

8. MCP security: the part that should never be an afterthought

An MCP server exposes capabilities to an AI client. Depending on its configuration, those capabilities can include browser navigation, data entry, access to authenticated sessions, and other operations.

That makes tool permissions an engineering concern.

For an enterprise pilot:

• Use dedicated test accounts and synthetic data.
• Restrict access to approved environments and domains.
• Avoid production credentials in agent context.
• Review which browser tools and external connections are enabled.
• Prevent untrusted page content from overriding the test specification.
• Require explicit approval for sensitive or irreversible operations.
• Preserve logs and evidence in approved locations.
• Define what the agent is allowed to change and what it can only observe.

An agent that reads an instruction embedded in a web page must not automatically treat it as a trusted command. The application under test is an input source, not an authority over the agent’s operating policy.

Also consider the security implications of MCP tool output, local processes, and browser sessions. Restricting the agent to a test environment does not remove the need for access control.

9. The architecture I would recommend

For a mature QA organization, I would not begin by replacing the regression suite.

I would separate the workflow into four components.

Exploration agent

Investigates the application, proposes scenarios, and adapts to unexpected UI states.

Approved browser interface

Exposes a limited set of browser operations through MCP, a CLI, or a conventional automation API.

Verification layer

Runs explicit assertions against the documented acceptance criteria and, where necessary, authoritative application state.

Evidence and governance layer

Stores the source revision, environment, test results, screenshots or traces, model configuration, and approval decisions.

This design lets the agent be flexible without giving it authority to redefine correctness.

A useful CI/CD integration would let an agent propose or investigate a test, execute approved checks, and attach evidence to a report. The final pass/fail decision for critical acceptance criteria should be derived from validated results, not from the agent’s narrative alone.

Conclusion: make the PASS mean something

Playwright MCP, Vibium, and Selenium with Robot Framework offer different paths into browser automation and agentic QA.

Playwright MCP is compelling for structured browser interaction through a mature Playwright ecosystem. Vibium is worth evaluating for its goal-driven execution and verification workflow. Selenium with Robot Framework remains relevant for deterministic regression testing and established enterprise automation, while Browser Library offers a Playwright-based alternative within Robot Framework.

The most important decision is not which tool looks most autonomous.

It is how the architecture establishes correctness, handles uncertainty, controls permissions, and preserves evidence.

A browser agent that can perform ten actions is useful.

A testing system that can demonstrate why the result is correct and recognize when it cannot is much more valuable.

The challenge: choose one scenario, run it through two approaches, deliberately introduce a failure, and compare their evidence. Publish the results, including the failures.

That is a far more useful contribution to the QA community than another generic comparison of automation frameworks.