Computer Use Agents: Claude API and OpenAI Agents API Compared

Compare Anthropic computer use and OpenAI Agents API browser control: runtime architecture, request schemas, approval workflows, security, and token pricing.

By ailog · Published Oct 24, 2026 · 8 min read · How we work
In short
  • Anthropic delivers a client-side execution model where your software controls the desktop environment and executes tool calls locally.
  • OpenAI provides a cloud-hosted sandbox environment where the platform executes browser actions on remote infrastructure.
  • Claude uses a single toolset declaration offering 17 member tools with sequential batching, while OpenAI uses session-based event streams.
  • OpenAI requires origin-by-origin user approvals during web navigation, whereas Anthropic relies on local sandbox isolation and injection classifiers.

Architectural Foundation: Local Desktop Sandbox vs. Hosted Browser Runtime

Automating graphical user interfaces through language models has diverged into two architectural patterns across the developer ecosystem. Anthropic structures desktop interaction around external client tool execution, whereas OpenAI manages browser automation inside an isolated, provider-hosted session. Understanding this foundational divide dictates how an engineering team designs infrastructure, networks, and compliance policies before issuing a single API call.

In the Anthropic framework, the model never accesses an operating system directly. Instead, your client application acts as the mediator between the Messages API and a developer-maintained target environment. This host environment typically provisions a virtual display server using Xvfb on Linux, paired with a lightweight window manager such as Mutter and an interface panel like Tint2. The local runtime runs user applications including Firefox, LibreOffice, text editors, and terminal utilities. When the model requests an operation, your runner translates that request into operating system calls, captures updated visual output, and returns the response back to the provider.

Conversely, the OpenAI Agents API delegates desktop and browser management to remote provider infrastructure. When a program initializes a session via https://api.openai.com/v1/agents/sessions, OpenAI spins up a managed virtual container. Your application does not host virtual displays, install operating system dependencies, or run local browser processes. Instead, the runner monitors session lifecycle events over network connections while the remote agent navigates pages, renders visual DOM elements, and collects page context inside the hosted environment.

Tool Configuration and Request Payload Schemas

The contrast between external execution and hosted infrastructure is evident in the request schemas required to initiate agent workflows. Anthropic declares desktop capabilities directly within the standard messages payload, whereas OpenAI establishes a persistent agent session with environment declarations.

Anthropic provides access through a unified client toolset declaration named computer_toolset_20260801. Adding this single object to the tools array automatically grants the model access to 17 member tools, such as left_click, type, zoom, cursor_position, and screenshot. Developers frequently bundle this toolset with the companion tools text_editor_20250728 and bash_20250124 to allow simultaneous code modification and shell interaction inside the client sandbox.

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-5-5",
    "max_tokens": 1024,
    "tools": [
      {"type": "computer_toolset_20260801"},
      {"type": "text_editor_20250728", "name": "str_replace_based_edit_tool"},
      {"type": "bash_20250124", "name": "bash"}
    ],
    "messages": [
      {"role": "user", "content": "Save a picture of a cat to my desktop."}
    ]
  }'

OpenAI uses a stateful agent configuration. The request targets the sessions endpoint with the OpenAI-Beta: agents=v1 header. The configuration pairs the computer_use tool declaration with an explicit environment object that enables desktop rendering and network access.

curl https://api.openai.com/v1/agents/sessions \
  -H "OpenAI-Beta: agents=v1" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "agent": {
      "model": "gpt-6-astra",
      "instructions": "Read public documentation in the browser. Do not sign in or change any website data. Report the page title and URL you find.",
      "tools": [{"type": "computer_use", "include_screenshots": true}]
    },
    "environment": {
      "type": "openai_hosted",
      "desktop": {"enabled": true},
      "network": {"access": "enabled"}
    }
  }'

Execution Loops: Batch Action Processing vs. Origin Access Controls

Once an agent starts working, the operational loop differs significantly depending on where the execution logic resides. Claude communicates actions through structured tool blocks, while OpenAI utilizes interactive session approval events.

When Claude decides to interact with a system, the response carries a stop_reason set to tool_use. It often packages several operations into a single response, known as a batch action. For instance, the model may return a left_click block, followed by a type block, ending with a screenshot block. Each block specifies toolset_name: "computer". Your application must process these blocks strictly in sequence rather than in parallel, because subsequent actions depend directly on the state left by prior actions.

Every tool block generated in a Claude turn requires a matching tool_result entry in the subsequent user message, identified by tool_use_id and tagged with toolset_name: "computer". Text confirmations like OK suffice for mouse clicks and typing, while screenshot and zoom require base64-encoded image payloads. If an action in a batch fails, execution must halt immediately. The developer returns is_error: true along with an error description for the failed action, while all unexecuted following actions must return is_error: true and the exact text: Not executed: an earlier computer action in this turn failed.

Under the OpenAI Agents API, execution occurs remotely, but network navigation requires programmatic approvals. Whenever the hosted browser attempts to navigate to a new web domain, the session halts with agent.session.requires_action. The application must inspect the pending computer_use_approval_request, evaluate the requested origin and optional reason, and post a decision event back to the session events route.

curl https://api.openai.com/v1/agents/sessions/$session_id/events \
  -H "OpenAI-Beta: agents=v1" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "type": "agent.session.input.computer_use_approval_request_result",
      "request_id": "REQUEST_ID",
      "response": {"type": "browser_origin_access", "decision": "approve"}
    }]
  }'

The approval decision must specify approve, deny, or cancel. If the agent encounters a login screen and generates a browser_authentication approval request, workflows configured strictly for public data can respond with an action of cancel to terminate the authentication attempt without crashing the session.

Security Models, Sandbox Isolation, and Prompt Injection Defense

Granting language models the ability to parse external screens and browse the open web introduces serious security risks, particularly indirect prompt injection. Untrusted data contained in documents, email clients, or web pages can attempt to override system instructions and hijack agent behavior.

  • Dedicated sandbox environments: For Claude computer use, execute agent operations inside isolated virtual machines or unprivileged containers to limit the impact of destructive shell commands or file modifications.
  • Domain restriction policies: Limit internet connectivity from the agent environment using network firewalls or egress allowlists so the model cannot access unauthorized external hosts.
  • Credential protection: Avoid storing sensitive user credentials, payment details, or access tokens inside the execution environment to mitigate exfiltration risks.
  • Human oversight checkpoints: Require affirmative user confirmation before completing consequential actions, such as financial transactions, legal terms acceptance, or cookie consent banners.

Anthropic provides built-in classifier defenses for the computer_toolset_20260801 toolset. Incoming screenshots and tool returns are automatically scanned by safety classifiers before processing. If an injection pattern is detected, the classifier steers the model to verify whether the instruction originated from the authorized user. Teams running automated workflows without interactive operators can opt out of these steering classifiers by contacting support.

OpenAI enforces security boundaries at the browser network perimeter through its origin access approval pipeline. However, approving origin access grants the agent free range over that domain. The origin approval mechanism does not intercept individual clicks or form inputs once access to an origin is granted. When individual destructive actions require human confirmation, teams must enforce those checks through custom function tools or manage their own browser runtimes.

Side-by-Side Implementation and Specification Comparison

Choosing the appropriate provider depends on hosting constraints, model compatibility requirements, and the level of operating system access your agent demands.

Feature / Dimension Anthropic Claude API OpenAI Agents API
Hosting Location Client-managed sandbox (Xvfb, Mutter, Tint2 on Linux) OpenAI-hosted remote browser container
Tool Declaration {“type”: “computer_toolset_20260801”} {“type”: “computer_use”, “include_screenshots”: true}
Interaction Scope Full desktop OS control (17 member tools: mouse, keyboard, zoom, files) Browser-focused navigation and UI interaction
Turn Execution Sequential batch actions dispatched in caller loop Server-managed session event streams
Human Approval Mechanism Caller-implemented before running local tool actions Built-in browser_origin_access approval requests
Prompt Injection Mitigation Automated screenshot classifiers with instruction verification Origin boundary controls; custom function gates for internal actions
Primary Documented Models claude-opus-5-5, claude-sonnet-5-5, claude-fable-5-1, claude-haiku-5-5 gpt-6-astra

Compatibility also varies by cloud infrastructure provider. The computer_toolset_20260801 toolset is generally available on the standard Claude API and Claude Models on Google Cloud (Google managed). In contrast, Claude Platform on AWS, Amazon Bedrock, and Microsoft Foundry offer computer use in beta through earlier tool versions like computer_20251124, which require specific beta request headers. Furthermore, computer use is not currently supported in Claude Managed Agents.

Token Consumption and Session Cost Considerations

Computer use workflows consume substantial token volumes because multi-turn loops transmit repeated visual screenshots and lengthy execution histories. Managing cost efficiency requires careful model selection and aggressive caching strategies.

On the Anthropic platform, flagship reasoning tasks can leverage Claude Opus 5.5 (claude-opus-5-5) at $4 per 1M input tokens, $0.2 per 1M cached input tokens, and $20 per 1M output tokens. For high-volume UI automation where lower cost is critical, Claude Sonnet 5.5 (claude-sonnet-5-5) reduces costs to $2 per 1M input tokens, $0.2 per 1M cached input tokens, and $10 per 1M output tokens. Claude Fable 5.1 (claude-fable-5.1) operates at $10 per 1M input tokens, $0.25 per 1M cached input tokens, and $50 per 1M output tokens, while Claude Haiku 4.5 (claude-haiku-4.5) operates at $1 per 1M input tokens, $0.1 per 1M cached input tokens, and $5 per 1M output tokens.

For OpenAI browser agent sessions running GPT-6 Astra (gpt-6-astra), API usage is billed at $10 per 1M input tokens, $1 per 1M cached input tokens, and $50 per 1M output tokens. Because these sessions include image screenshots and growing message threads, repeated round trips can quickly accumulate significant token usage. To explore granular per-session cost modeling across these configurations, reference the /tools/llm-api-cost-calculator/ and review base rates at /tools/llm-api-pricing/.

To keep usage within bounds across both providers, consider implementing prompt caching on static system prompts and system tool definitions. Developers handling image payloads can consult image format guidelines to minimize payload overhead. For purely web-centric workflows in Claude, Anthropic also documents the browser use tool, which operates directly on page structures without demanding full desktop display rendering.

Implementation Checklist for Production Deployments

Before promoting an automated desktop or browser agent to production environments, verify that your integration satisfies essential security, error handling, and session management requirements.

  • Confirm the tool version: Ensure your Claude API calls declare computer_toolset_20260801 rather than retired tool identifiers like computer_20251124 or computer_20250124 on standard endpoints.
  • Enforce sequential batch execution: Iterate through Claude tool_use blocks sequentially, stopping at the first failure and populating all skipped items with the designated halt error message.
  • Ensure toolset_name consistency: Verify that every tool_result returned to Claude includes toolset_name: “computer” alongside tool_use_id to avoid invalid request rejections.
  • Implement approval loop handlers: For OpenAI Agents API sessions, register event listeners for agent.session.requires_action and handle browser_origin_access approval requests interactively.
  • Establish sandbox isolation: Deploy Linux desktop containers without administrative privileges, enforce domain firewalls, and separate customer credentials from agent workspaces.
  • Clean up session state: Explicitly delete inactive OpenAI sessions and terminate idle local X11 display servers once automated tasks conclude.
Sources
  1. Computer use tool
  2. Computer use

Related