Building an open-source Browser Agent on Fireworks AI

By Aishwarya Srinivasan|5/16/2025

Imagine an AI that doesn't just respond to your questions but can actively navigate the web for you - clicking buttons, filling forms, extracting information, and making decisions just like you would. That's the promise of AI agents with browser control capabilities, and it's becoming a reality with tools like Fireworks AI BrowserUse.

In this technical deep dive, we'll explore how large language models (LLMs) can be given the ability to "see" web content and take actions in real-time. We'll examine the architecture that makes this possible and show why Fireworks AI's inference capabilities are particularly well-suited for this challenging task.

Why Browsers Are Still the Universal API

Despite the push toward structured APIs, browsers remain the most universal interface to the web's vast information and services. Here's why building agents that can control browsers matters:

1. Universal Interface: Browsers can interact with any website regardless of its underlying technology stack. While some services offer well-documented APIs, countless others don't. A browser-based agent can interact with all of them.
2. Real-world UX Processing: Browser agents encounter the same interfaces humans do. This means they learn to navigate real-world UX patterns and challenges, including CAPTCHAs, cookie notices, modal dialogs, and dynamically loaded content.
3. Authentication & Complex Flows: Many websites implement complex authentication flows, session management, and multi-step processes. Browser agents can handle these just like a human would – navigating through OAuth screens, remembering cookies across sessions, and maintaining context across page transitions.
4. Dynamic Content Interpretation: Modern web applications render content dynamically through JavaScript execution. Traditional web scrapers often miss this content, but browser automation captures the fully rendered state that users actually see.
5. Multimedia & Interactive Elements: Browser agents can process rich media like images and videos, and interact with complex UI elements like drag-and-drop interfaces, canvas-based visualizations, and WebGL content.

This makes browser automation the most robust approach to web interaction, despite being technically more complex than API integration. The challenge has been making it intelligent enough to handle the unpredictability of the modern web.

Giving LLMs Eyes and Hands: The Agent Architecture

Creating an agent that can browse the web effectively requires solving several technical challenges simultaneously. Our architecture tackles these by implementing three core capabilities:

The Vision System

For an AI agent to understand a webpage, it needs to "see" the content. Our solution combines multiple techniques:

1. DOM Access: The agent extracts the Document Object Model (DOM) to understand the page structure, identifying interactive elements like buttons, forms, and links.
2. Visual Context: We capture screenshots of the visible viewport and convert them to base64 for the LLM to process. This gives the agent crucial visual context about layout, images, and design elements that pure HTML doesn't convey.
3. Spatial Awareness: The agent tracks viewport position, knowing how much content exists above and below the current view. This helps it understand when scrolling is needed to access more content.
4. Element Identification: Each interactive element receives a unique index, which is presented to the LLM along with its element type and text content. This creates an unambiguous way for the model to refer to specific elements it wants to interact with.

Here's a simplified illustration of how we capture browser state:

async def get_browser_state(self):
	"""Get the current browser state including DOM and screenshot."""
	# Capture screenshot and convert to base64
	screenshot = await self.page.screenshot(encoding="base64")
	# Get DOM elements with indices for interaction
	elements = await self._parse_dom_for_llm(self.page)
	# Track viewport position
	scroll_position = await self.page.evaluate("window.scrollY")
	page_height = await self.page.evaluate("document.body.scrollHeight")

	return {
		"url": self.page.url,
		"title": await self.page.title(),
		"screenshot": screenshot,
		"elements": elements,
		"pixels_above": scroll_position,
		"pixels_below": page_height - (scroll_position + viewport_height)
	}

This method gives the LLM all the context it needs to understand the current page state.

The Reasoning System

Once the agent can see the page, it needs to determine what to do next. This is where Fireworks AI's advanced reasoning capabilities come into play:

1. Context-Aware Decision Making: The agent doesn't just look at the current page in isolation. It maintains memory of previous actions, keeps track of its goals, and evaluates whether actions were successful.
2. Structured Thinking: We've designed the agent to follow a clear reasoning pattern: evaluate previous action → update memory → set next goal → choose specific action. This structure helps prevent the model from hallucinating or getting confused.
3. Task Planning: For complex multi-step tasks, the agent breaks down the overall goal into manageable sub-tasks and tracks progress through each step.
4. Error Handling Logic: The agent can recognize when actions fail and implement recovery strategies - retrying with variations, taking alternative paths, or diagnosing what went wrong.

Our implementation uses structured JSON outputs that force the LLM to maintain this reasoning framework, preventing it from defaulting to vague or unactionable responses.

The Action System

Finally, the agent needs to translate decisions into actual browser manipulations. Our BrowserUseTool handles this by providing a comprehensive set of atomic actions that can be combined to perform complex tasks:

1. Navigation Actions: Direct URL navigation, back/forward, refresh, and search capabilities.

2. Element Interaction: Clicking, text input, scrolling, and keyboard input tools that can manipulate any interactive element.

3. Content Extraction: Specialized functions for extracting structured data from pages based on semantic goals.

4. Multi-tab Management: The ability to open, close, and switch between multiple tabs for complex workflows.

This action system provides a clean interface that abstracts away the complexity of browser automation, letting the LLM focus on high-level decision making rather than implementation details.

The Observation-Decision-Action Loop

The core of our browser agent is the continuous loop of:

1. Observe: Capture the current state (DOM, screenshot, URL, title)
2. Decide: Process this information to determine the next step
3. Act: Execute the chosen action through browser automation
4. Evaluate: Assess whether the action achieved its intended result

This loop continues until the task is complete or requires human intervention. What makes this approach powerful is that the agent can adapt to unexpected situations - if a website changes its layout, introduces a new step, or behaves differently than expected, the agent can still navigate it successfully because it's responding to what it actually sees rather than following a pre-programmed script.

Fireworks AI: The Brain of the Browser Agent

One of the key technical challenges in building effective browser agents is the quality and speed of the underlying LLM. This is where Fireworks AI provides significant advantages.

Speed and Latency

Browser automation requires multiple back-and-forth interactions between the agent and the browser. Traditional LLM architectures introduce noticeable latency in this process, making the agent feel sluggish and unresponsive.

Fireworks AI models are optimized for inference speed, dramatically reducing the time between observation and action. In our testing, this resulted in:

• 40-60% faster page navigation sequences
• More responsive form filling (critical for time-sensitive operations)
• Ability to handle dynamic content that appears and changes quickly

This speed improvement isn't just about user experience - it fundamentally changes what the agent can accomplish by enabling it to keep up with modern, highly interactive websites.

JSON Mode and Structured Outputs

Browser automation demands precision. Vague instructions or formatting errors can cause actions to fail. Fireworks AI's JSON mode produces strictly formatted, valid JSON responses that are more reliable for parsing and executing actions.

Our system leverages this capability to enforce a structured thinking pattern where the LLM must generate valid JSON for every decision, with specific fields for evaluation, memory, goals, and actions. The response_format parameter ensures the model always returns properly structured data:

response = await llm_client.chat.completions.create(
	model=model_name,
	messages=messages,
	tools=tools,
	response_format={"type": "json_object"},
	temperature=0.2
)

This structured approach forces the model to be explicit about its reasoning and intended actions, reducing errors caused by ambiguous instructions.

Context Window Efficiency

Webpage content can be incredibly verbose. A single page might contain thousands of DOM elements, and screenshots add substantial token usage. Fireworks models handle long contexts efficiently, allowing our agent to process more information without hitting token limits.

This means the agent can:

• Process entire page contents rather than truncated snippets
• Maintain longer browsing histories for context
• Handle complex, content-heavy websites like documentation pages or data-rich dashboards

Multimodal Understanding

Modern websites are highly visual. Text alone often doesn't capture the full context of what's on a page. Fireworks AI's multimodal capabilities enable our agent to process screenshots alongside DOM data, giving it a more complete understanding of the page.

This visual understanding helps with:

• Identifying UI patterns that aren't clearly labeled in the DOM
• Understanding spatial relationships between elements
• Recognizing when content is obscured or needs scrolling to access
• Interpreting graphics, charts, and visual indicators

Our configuration allows for separate model selection for text reasoning and visual processing:

# Global LLM configuration
[llm]
model = "accounts/fireworks/models/deepseek-v3"
# Vision model configuration
[llm.vision]
model = "accounts/fireworks/models/firellava-13b"

This flexibility lets us optimize for each aspect of the agent's operation, using specialized models where they excel.

The Prompt Engineering Challenge

Perhaps the most technically nuanced aspect of building effective browser agents is designing prompts that guide the LLM effectively. Our system prompt is a carefully crafted set of instructions that shapes how the agent interprets what it sees and decides what to do.

The prompt structures the agent's input and output in specific ways:

1. Input Format: We present the webpage as a combination of:
   • Interactive elements with numeric indices
   • Current URL and page title
   • Available tabs and viewport position
   • Any previous action results or errors
2. Output Structure: The agent must respond with structured JSON that includes:

• Evaluation of previous actions (success/failure)
• Memory of what has been done and what remains
• Next goal to accomplish
• Specific actions to take with parameters

This strict structure guides the LLM's reasoning process, preventing common failure modes like hallucination, forgetting context, or generating vague commands.

Technical Challenges and Solutions

Building browser agents presents numerous technical challenges. Here are the most significant ones we've encountered and how we've addressed them:

Challenge 1: Element Selection Hallucinations

Problem: LLMs often hallucinate element indices or try to interact with non-existent elements, particularly when a desired element isn't visible in the current viewport.

Solution: We implemented several strategies to address this, including a precise DOM parsing system that creates unambiguous element references. Our parser traverses the DOM, identifies interactive elements, and assigns them unique indices, formatting each one as:

[15]<button>Submit Form</button>
[16]<input placeholder="Search..."></input>

This approach makes it crystal clear which elements can be interacted with and how to reference them, dramatically reducing hallucination issues.

Challenge 2: Dynamic Content and Timing Issues

Problem: Modern websites load content dynamically, making it challenging to know when a page is ready for interaction.

Solution: We implemented a sophisticated waiting system with multiple strategies:

1. Load State Tracking: Waiting for the page's load state (DOM, network, etc.)
2. Content-Based Waiting: Checking for specific text or selectors to appear
3. Dynamic Timeouts: Adjusting wait times based on page complexity
4. Progressive Interaction: Using strategic clicks to trigger content loading

Our system can adapt its waiting strategy based on the specific website and expected content, significantly improving reliability.

Challenge 3: Memory and Context Management

Problem: As agents navigate multiple pages and perform complex tasks, maintaining context becomes challenging.

Solution: We developed a structured memory system that tracks:

1. Navigation History: A record of previously visited URLs
2. Extracted Data: Information gathered from each page
3. Task Progress: Counters for multi-step operations
4. Session Notes: Important observations and decisions

This comprehensive memory enables the agent to maintain context across complex workflows, remembering what it has already learned and what remains to be done.

Challenge 4: Error Recovery

Problem: Web interactions frequently fail in unpredictable ways - elements disappear, pages redirect unexpectedly, forms reset, etc.

Solution: We implemented a multi-layered recovery system:

1. Proactive Validation: Checking preconditions before actions
2. Intelligent Retry Logic: Using different strategies based on error type
3. Adaptive Backoff: Increasing wait times between retries
4. Alternative Approaches: Trying different paths to the same goal

This robust error handling allows the agent to recover from many common failures without human intervention.

The Future of Browser Agents

This is just the beginning of what's possible with browser agents. Our research suggests several promising directions for future development:

Multi-agent Collaboration

Complex web tasks often involve multiple specialized skills. Future implementations could use multiple agents working together:

• Navigator Agents: Specialized in finding and navigating to relevant content
• Extractor Agents: Focused on efficiently parsing and structuring information
• Writer Agents: Converting extracted information into coherent summaries
• Coordinator Agents: Managing the overall workflow between other agents This division of labor would allow each agent to specialize in what it does best.

Learned Behaviors from Demonstrations

Currently, our agents rely primarily on in-context reasoning. Future versions could incorporate:

• Imitation Learning: Observing and learning from human demonstrations
• Experience Replay: Improving over time by analyzing past successes and failures
• Preference Learning: Adapting to individual user preferences for interaction style These approaches would make agents more personalized and efficient over time.

Enhanced Privacy and Security Controls

As browser agents gain capabilities, privacy and security become increasingly important:

• Sandboxed Execution: Running in isolated environments with limited permissions
• Data Minimization: Processing only essential information and discarding sensitive data
• Credential Management: Secure handling of authentication without exposing secrets
• Audit Trails: Transparent records of all agent actions for review

These controls would make browser agents suitable for more sensitive use cases.

Conclusion: The Augmented Web Experience

Browser agents represent a significant evolution in how we interact with the web. By combining the reasoning capabilities of LLMs with browser automation and enhancing them with Fireworks AI's efficient inference, we've created a system that can navigate the web with a level of understanding and adaptability approaching that of a human user.

The key innovations in our approach include:

1. Vision-Guided Navigation: Using visual and structural information for more robust interaction
2. Structured Reasoning Workflows: Guiding the LLM through a clear decision process
3. Adaptive Error Recovery: Handling the unpredictability of real-world websites
4. Optimized Inference Pipeline: Leveraging Fireworks AI for responsive, consistent performance

These capabilities open up new possibilities for automation, research, and accessibility. Browser agents can handle repetitive tasks like form filling, comparison shopping, or data collection. They can also assist users with disabilities by navigating complex interfaces on their behalf.

Remember: the goal isn't to replace human interaction with the web but to augment it- freeing us from repetitive tasks while maintaining the flexibility to handle the rich complexity of the modern web.

This project is open source and builds on technologies like Playwright, browser-use, and Fireworks AI. We welcome contributions and feedback from the community.

💡 GitHub Repository: https://github.com/shubcodes/fireworksai-browseruse