How LLMs Control Web Browsers

The Perception Layer: How LLMs See Web Pages

An LLM cannot look at a web browser. It processes text (and, for multimodal models, images). The perception layer is the system that converts the current browser state into a format the LLM can understand and reason about. There are three main approaches, each with distinct strengths.

DOM-Based Perception

The most common approach reads the page's Document Object Model and converts it into a simplified text representation. The raw DOM of a modern web page can contain thousands of elements, most of which are decorative or structural and irrelevant to the task at hand. A typical product page might have 3,000 DOM nodes, but only 20 or 30 of them are interactive elements the agent would ever need to interact with.

Frameworks like Browser Use apply aggressive filtering to the raw DOM. They strip out hidden elements, decorative divs, style attributes, and script tags. The remaining interactive elements, including buttons, links, input fields, dropdowns, and text areas, are extracted and labeled with numeric identifiers. The result is a compact representation that looks something like: "[1] Search input field, [2] 'Electronics' category link, [3] 'Sign In' button, [4] Shopping cart icon (3 items)."

This compressed DOM representation fits within the LLM's context window and gives it enough information to make decisions. When the LLM decides to "click element [3]," the framework maps that identifier back to the original DOM element and executes the click through Playwright.

DOM-based perception is fast and token-efficient. Parsing the DOM and generating the compressed representation takes milliseconds, and the resulting text uses far fewer tokens than an image would. The limitation is that it depends on the DOM being clean and accessible. Websites that render content through canvas elements, embed information in images, or use heavily obfuscated class names can produce poor DOM representations.

Vision-Based Perception

Vision-based perception captures a screenshot of the browser viewport and sends it to a multimodal LLM that can process images. The model sees exactly what a human user would see: buttons with their visual styling, text in its rendered form, images, charts, and layout structure.

Tools like OmniParser V2 enhance this process by pre-processing the screenshot to detect and label interactable elements before sending the annotated image to the LLM. The pre-processing step identifies buttons, text fields, links, and other clickable regions, draws bounding boxes around them, and assigns identifiers. This gives the LLM a structured overlay on top of the visual content.

Skyvern uses vision-based perception as its primary method. It captures a screenshot, sends it to a vision LLM, and receives back both a semantic understanding of the page and the coordinates of the element to interact with. This approach works on any website regardless of DOM structure, making it robust against obfuscated markup, canvas-rendered interfaces, and non-standard UI components.

The trade-offs are speed and cost. Screenshots are large, consuming significantly more tokens than text-based DOM representations. A single viewport screenshot might use 1,000 to 2,000 tokens depending on resolution, compared to 200 to 500 tokens for a compressed DOM. Processing takes longer because multimodal inference is slower than text-only inference. At scale, these differences compound into meaningful cost and latency increases.

Accessibility Tree Perception

A newer approach reads the browser's accessibility tree rather than the raw DOM. The accessibility tree is the browser's internal representation of the page for screen readers and assistive technologies. It contains semantic information about element roles (button, link, text input, heading), labels, states (checked, disabled, expanded), and relationships between elements.

The accessibility tree is often cleaner and more semantically meaningful than the raw DOM because browsers already perform significant processing to build it. An element that appears as a <div> with click handlers in the DOM shows up as a "button" in the accessibility tree because the browser recognizes its behavioral role. This semantic richness gives the LLM better context for making decisions.

This approach works well for well-structured websites that follow accessibility standards, but it can miss visual elements that are not properly annotated in the accessibility tree, such as decorative elements that convey information through visual design rather than semantic markup.

The Reasoning Layer: How LLMs Decide What to Do

The LLM receives a prompt that typically includes four components: the system instructions defining its role as a browser agent, the original task description from the user, the history of actions taken so far, and the current page representation.

From this context, the LLM generates a structured output specifying the next action. Modern browser agent frameworks constrain the LLM's output to a defined action schema. The LLM does not return free-form text. Instead, it produces a structured response like {"action": "click", "element": 7} or {"action": "type", "element": 3, "text": "New York"}. This structured output format prevents the LLM from generating invalid or ambiguous instructions.

The reasoning capability of the LLM determines the quality of these decisions. When a cookie consent dialog appears mid-task, a strong model recognizes it as an interruption, dismisses it, and resumes the original workflow. A weaker model might interpret the dialog as part of the task or get confused about which element to interact with. This is why frontier models consistently outperform smaller models on browser agent benchmarks, especially on complex, multi-step tasks.

Planning vs Reactive Approaches

Some frameworks have the LLM generate a multi-step plan before executing any actions. The plan outlines the sequence of pages to visit, forms to fill, and data to extract. This planning phase helps with coherence across long task sequences because each action is guided by the overall plan rather than being a purely reactive response to the current page.

Other frameworks operate reactively, where the LLM looks at the current page state and decides only the immediate next action without a broader plan. This approach is simpler and works well for straightforward tasks but can struggle with complex workflows where early decisions affect later options.

Hybrid approaches plan at a high level and react at the step level, adjusting the plan when unexpected situations arise. This provides the coherence benefits of planning with the adaptability of reactive decision-making.

The Action Layer: Translating Decisions to Browser Commands

The action layer receives the LLM's structured decision and translates it into actual browser automation calls. Most AI browser agents use Playwright as the underlying automation engine, though Puppeteer and Selenium are also supported by various frameworks.

The translation is straightforward for common actions. A "click element [7]" decision maps to finding the DOM element corresponding to identifier 7 and calling element.click(). A "type 'hello' into element [3]" decision maps to finding element 3 and calling element.fill('hello'). Scrolling, navigating to URLs, taking screenshots, and managing tabs all have direct Playwright API equivalents.

More complex actions require additional logic. File uploads need the action layer to handle file paths and browser dialog interactions. Drag-and-drop operations require coordinate calculations. Keyboard shortcuts need the action layer to translate semantic instructions ("press Enter") into the specific key combinations the browser expects.

The action layer also handles timing. After each action, it waits for the page to settle before capturing the new state for the next perception cycle. This involves waiting for network requests to complete, animations to finish, and dynamic content to render. Getting these waits right is critical for reliability, and Playwright's built-in auto-waiting mechanisms handle most cases, but some dynamic websites require custom wait logic.

The Feedback Loop: How Agents Self-Correct

After executing each action, the agent captures the new page state and feeds it back into the LLM along with the action that was just taken and its expected outcome. The LLM evaluates whether the action succeeded and decides the next step accordingly.

This feedback loop is what makes AI browser agents self-correcting. If a click did not produce the expected result, perhaps because the element was not fully loaded when clicked, the agent can try again or try an alternative approach. If the agent navigates to the wrong page, it can recognize the error and navigate back. This is fundamentally different from traditional scripts, which either succeed or throw an error with no recovery logic unless explicitly programmed.

The quality of self-correction depends entirely on the LLM's ability to compare expected outcomes with actual outcomes. Strong models do this well, noticing subtle differences like "I expected a search results page but this looks like an error page." Weaker models may not notice the discrepancy and continue with incorrect assumptions, leading to cascading errors.

Performance Considerations

Each cycle of the perception-reasoning-action loop involves at least one LLM API call, and complex decisions might require multiple calls. A simple task that takes five cycles means five API round trips, each taking 1 to 5 seconds depending on the model and provider. Add DOM parsing, screenshot capture, and browser interaction time, and a five-step task takes 15 to 30 seconds total.

Token consumption varies significantly between perception approaches. DOM-based agents typically use 500 to 2,000 tokens per cycle for the page representation, plus 200 to 500 tokens for the task context and action history. Vision-based agents add 1,000 to 2,000 tokens per screenshot on top of the text context. Over a 20-step task, the total token consumption can range from 15,000 to 80,000 tokens, which translates to costs between a few cents and a few dollars per task depending on the model and provider.

For latency-sensitive applications, the key optimization is minimizing LLM calls. Caching successful element selections (as Stagehand does), pre-planning multi-step sequences, and using hybrid DOM/vision perception all help reduce the number of round trips required.