Browser agents inherit every reliability problem regular browser automation already has — flaky selectors, timing issues, unexpected page states — and add a new one on top: the model deciding what to click or type next based on a screenshot or DOM snapshot it's interpreting, which introduces its own failure mode.
A browser agent is an AI system that controls a real web browser — navigating pages, clicking elements, filling forms, reading content — typically through a browser automation library (Playwright or Puppeteer) wrapped as tools the model can call, deciding its own sequence of actions to accomplish a goal like "book this flight" or "find the pricing page."
Why Browser Agents Matter (and When a Direct API Integration Is Simply Better)
Browser agents matter specifically for tasks requiring interaction with sites that don't expose an API — automating workflows against legacy internal tools, scraping or interacting with sites without programmatic access, or testing a UI the way an actual user would — where the browser itself is the only available interface.
A direct API integration is simply better whenever the target system has one — calling a documented API is faster, more reliable, and cheaper than driving a browser to accomplish the same result, and reaching for browser automation when an API exists is solving a harder problem than necessary.
Getting Started with Browser Agents
Wrapping Playwright actions as tools for a model to call:
import { chromium } from "playwright";
const browser = await chromium.launch();
const page = await browser.newPage();
const tools = {
navigate: async ({ url }: { url: string }) => {
await page.goto(url);
return { title: await page.title() };
},
click: async ({ selector }: { selector: string }) => {
await page.click(selector);
return { success: true };
},
getPageContent: async () => {
return { text: await page.innerText("body") };
},
fillField: async ({ selector, value }: { selector: string; value: string }) => {
await page.fill(selector, value);
return { success: true };
},
};
Giving the model page state before each decision, since it needs current context to decide the next action:
async function agentStep(goal: string, history: string[]) {
const pageContent = await page.innerText("body");
const screenshot = await page.screenshot();
const response = await model.generate({
messages: [
{ role: "system", content: `Goal: ${goal}. History: ${history.join("\n")}` },
{ role: "user", content: [{ type: "text", text: pageContent }, { type: "image", data: screenshot }] },
],
tools: toolDefinitions,
});
return response;
}
Core Browser Agent Concepts Every Developer Should Know
The model needs current page state before every decision — a screenshot, DOM snapshot, or extracted text — since it can't reliably decide the next action based on stale context; each step needs fresh information about what the page actually looks like now, after any previous action's effects.
Selector-based actions are more reliable than coordinate-based clicking, since a semantic selector (an element's text, role, or attributes) survives minor layout changes better than fixed pixel coordinates — giving the model tools that operate on selectors, not raw coordinates, produces more robust agent behavior across page variations.
Timing and page-load state are a real source of failures distinct from the model's decision-making — an action attempted before a page has finished loading or an element has rendered fails regardless of whether the model's decision was correct. Explicit waits for load state or specific elements need to be part of your tool implementations, not left to the model to reason about.
Consequential actions (submitting a form, completing a purchase, deleting something) warrant the same confirmation-checkpoint pattern used elsewhere in agentic systems — a browser agent with the ability to click "confirm purchase" needs the same hard-to-reverse-action safeguard any other agentic tool with real-world side effects needs.
Common Mistakes Building Browser Agents and How to Fix Them
Mistake 1: reaching for browser automation when the target system has a usable API. Fix: check for an API first; browser automation should be the fallback for sites without programmatic access, not the default approach.
Mistake 2: giving the model coordinate-based clicking instead of selector-based actions, producing brittle behavior that breaks on minor layout changes. Fix: implement tools around semantic selectors (text, role, test IDs) rather than raw pixel coordinates.
Mistake 3: no explicit wait/load-state handling in tool implementations, causing actions to fail due to timing issues unrelated to the model's decision quality. Fix: build appropriate waits for page load and element visibility into your tool implementations, the same reliability practice any browser automation needs regardless of AI involvement.
When Should You Build a Browser Agent Instead of Automating Against an API?
Build a browser agent specifically when no API exists for the target system — legacy tools, sites without programmatic access, or tasks requiring genuine UI interaction (visual verification, testing). Automate against an API whenever one is available and documented — it will be faster, cheaper, and substantially more reliable than achieving the same result through browser automation.
Browser Agents in Production
Build reliability practices (selector-based actions, explicit load-state waits) into your tools regardless of AI involvement, since browser automation's classic failure modes don't go away just because a model is driving it. Add confirmation checkpoints for any consequential action a browser agent can take, and provide fresh page state before every decision the model makes.
If you're considering a browser agent, confirm first that no API exists for the target system — browser automation is the right tool specifically when it's the only interface available, not a default choice.