Sending Images to Claude, GPT and Gemini: Formats and Limits

Ingestion Methods: Base64, Hosted URLs, and File IDs

Multimodal applications rely on three common mechanisms to transmit visual data to frontier vision models: direct inline byte encoding through base64, references to accessible public URLs, and pre-uploaded assets tracked via a provider-managed Files API. While all three platforms—Anthropic Claude, OpenAI GPT, and Google Gemini—accommodate these patterns, their internal handling and infrastructure constraints differ significantly.

Inline transmission through base64 data strings requires no external web hosting and guarantees that the target model receives the exact binary representation without external network dependencies. However, converting images into text representations increases raw network transfer weight and compounds message sizes over multi-turn interactions. By contrast, referencing public URLs removes payload bloat from the client request, though provider infrastructure must resolve and fetch the resource on every invocation. Hosted platforms like Amazon Bedrock and Google Cloud enforce stricter boundaries for Anthropic models, supporting only base64 data sources rather than public URLs or file references.

For conversational systems, agentic loops, and document pipelines, the Files API approach provides the cleanest architectural boundary. An image is transmitted once to an asset storage endpoint, which generates an identifier or resource URI. Subsequent model queries simply point to that identifier inside the content list. This avoids retransmitting identical megabytes of imagery on every turn while retaining access across the context window.

Request Structures Across Anthropic, OpenAI, and Google

Each provider organizes image payloads under different key structures within message arrays. Anthropic structures multimodal requests inside the Messages API by splitting the user message content into discrete typed dictionaries, categorizing the asset as an image content block with an explicit source object specifying encoding and media format.

{
  "model": "claude-opus-5-5",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image",
          "source": {
            "type": "base64",
            "media_type": "image/jpeg",
            "data": "$BASE64_IMAGE_DATA"
          }
        },
        {
          "type": "text",
          "text": "Describe this image."
        }
      ]
    }
  ]
}

OpenAI processes vision either through the Chat Completions endpoint or the Responses endpoint. In the Responses schema, input items are provided under an input array containing content objects typed as input_text and input_image. An image URL entry can supply either a web URI or an inline data URI formatted with the appropriate media type prefix, alongside optional parameter keys such as detail set to auto.

{
  "model": "gpt-6-astra",
  "input": [
    {
      "role": "user",
      "content": [
        {"type": "input_text", "text": "what is in this image?"},
        {
          "type": "input_image",
          "file_id": "file_id_string",
          "detail": "auto"
        }
      ]
    }
  ]
}

Google uses the Interactions interface for Gemini models. In this schema, client code provides an input list containing discrete content entries. Instead of nesting sources inside nested descriptor objects, Gemini takes type set to image, a mime_type indicator such as image/jpeg, and either a direct data string containing base64 content or a uri string pointing to an asset uploaded through the Files API or a public URL.

{
  "model": "gemini-3.8-flash",
  "input": [
    {"type": "text", "text": "Caption this image."},
    {
      "type": "image",
      "uri": "https://example.com/image1.jpg",
      "mime_type": "image/jpeg"
    }
  ]
}

Payload Limits, Memory Overhead, and Multi-Turn Workflows

Integrating visual assets into production calls requires tracking payload thresholds closely. Google explicitly caps total inline request sizes at 20MB for Gemini interactions. This 20MB boundary encompasses text prompts, system instructions, and inline base64 image bytes combined. Attempting to pass multiple high-resolution inline images in a single call risks hitting this ceiling. To handle larger volumes or multiple assets, Google recommends uploading images via the Files API prior to inference.

Anthropic documents a key operational consideration for stateful agent runs and extended chats. Standard chat loops retransmit the complete conversational history on every turn. When client code embeds base64 images directly into user messages, those megabytes of image bytes travel across the wire on every single round trip, causing unnecessary latency and network saturation. Leveraging the Files API allows callers to upload each image once, obtain a file_id, and reference that identifier across turns without inflating request transmission volume.

Multimodal Token Costs and Model Selection

Because image content consumes prompt tokens during multimodal processing, image processing costs scale directly with the underlying model token pricing. Developers can review pricing tiers on /tools/llm-api-pricing/ or simulate variable context sizes using /tools/llm-api-cost-calculator/. Understanding differences across token charges is vital when setting a safety boundary like a /posts/per-run-cost-cap-for-llm-api-calls/.

Under the Anthropic pricing schedule, Claude Opus 5.5 (claude-opus-5.5) charges $4 per 1M input tokens, $0.2 per 1M cached input tokens, and $20 per 1M output tokens. Claude Sonnet 5.5 (claude-sonnet-5.5) prices input at $2, cached input at $0.2, and output at $10. For high-volume inspection tasks, Claude Haiku 4.5 (claude-haiku-4.5) reduces standard input to $1 per 1M tokens, cached input to $0.1, and output to $5.

OpenAI positions GPT-6 Astra (gpt-6-astra) as its primary multimodal system, with standard input priced at $10 per 1M tokens, cached input at $1, and output at $50 per 1M tokens. For budget-conscious pipelines, alternate GPT models like GPT-6.1 Sol cost $2 per 1M input tokens and $10 for output, while GPT-6 Luna bills input at $0.1 and output at $0.5. Developers generating images directly alongside vision inspection can also note dedicated engines such as gpt-image-2.5-sunburst and gpt-image-2 available within OpenAI tools.

Google provides several multimodal models across its flash tiers. Gemini 3.8 Flash (gemini-3.8-flash) bills standard input tokens at $0.75, cached inputs at $0.075, and output tokens at $3.75 per 1M tokens. Lower latency options include Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite) at $0.3 per 1M input tokens and $2.5 per 1M output tokens, or Gemini 2.5 Flash-Lite (gemini-2.5-flash-lite) at $0.1 per 1M input tokens and $0.4 for output. Gemini documentation highlights that these native models provide enhanced accuracy for tasks such as object detection and segmentation beyond standard question answering.

Implementation Gotchas and Ordering Rules

Subtle differences in prompt structure and cloud platform support can introduce performance degradation or unexpected errors if overlooked during development.

• Content block ordering in Claude: Anthropic documentation emphasizes that Claude yields superior comprehension when images appear before associated query text in the content list. While text followed by images or interleaved blocks remain functional, an image-then-text ordering is explicitly recommended.
• Cloud platform restrictions: When invoking Claude models hosted on Amazon Bedrock or Google Cloud rather than the native Anthropic API, image URLs and Files API identifiers are unsupported; only base64-encoded strings can be accepted.
• Inline request ceiling in Gemini: Mixing multiple base64 images into standard text requests easily breaches the 20MB inline request boundary in Gemini, causing payload rejections. Applications handling batch images or high-resolution graphics should route files through Google Files API upload methods.
• State bloat in agent workflows: Passing base64 strings repeatedly inside long-running dialogs re-uploads raw image bytes on every single request, multiplying latency and network transit costs. Referencing static file identifiers eliminates redundant byte transmission across multi-turn sessions.
• Prompt caching dynamics: Caching can significantly lower recurring input expenses when identical system instructions and assets are evaluated repeatedly. Teams optimizing multi-image architectures should refer to /posts/prompt-caching-explained-claude-gpt-gemini/ to understand how repeated context windows interact with cache read pricing.

Multimodal API Implementation Checklist

Before deploying vision-enabled prompts into automated production services, verify the following configuration points across your codebase:

• Confirm whether your provider and host environment support URL references and file identifiers, or if you must fallback to base64 encoding (mandatory for Claude on Amazon Bedrock and Google Cloud).
• Verify that total inline request content stays strictly under 20MB when sending base64 payloads to Google Gemini interactions.
• Format Anthropic Claude message content arrays with image blocks positioned before instruction text blocks to optimize model attention.
• Replace base64 strings with Files API references (file_id or uploaded file URI) whenever assets persist across multi-turn dialogues or agent steps.
• Align model choice with workload economics, comparing models such as Claude Haiku 4.5 ($1 input), Gemini 3.8 Flash ($0.75 input), and GPT-6 Astra ($10 input) against task complexity requirements.