Batch Processing APIs Step by Step: Claude, OpenAI, and Gemini
Compare batch processing APIs across Anthropic Claude, OpenAI, and Google Gemini, covering request shapes, limits, 50% discounts, and polling workflows.
- All three platforms slash standard inference costs by 50% when requests run through their respective asynchronous batch systems.
- Anthropic accepts batch requests directly via an HTTP array, OpenAI mandates a pre-uploaded JSONL file, and Gemini supports both inlined payloads and uploaded files.
- Maximum batch sizes vary significantly: OpenAI caps submissions at 50,000 requests (200 MB), Anthropic permits up to 100,000 requests (256 MB), and Gemini accepts files up to 2GB.
- Every provider enforces a 24-hour completion target or expiration deadline, while Claude processes the majority of batch runs within 1 hour.
Architectural Concepts: How Claude, OpenAI, and Gemini Ingest Batches
Asynchronous batching allows engineering teams to submit large collections of model queries without tying up long-lived client HTTP connections. Rather than blocking until every response completes, the client offloads the workload to the provider infrastructure and fetches the results once processing finishes. While the fundamental pattern is identical across Anthropic, OpenAI, and Google Gemini, their architectural designs diverge in how jobs are ingested, staged, and referenced.
Anthropic implements the Message Batches API as a direct REST interface. You do not need to register an external file before queuing a job. Instead, you send an array of Message parameters directly to the batches endpoint inside a single JSON request payload. Anthropic then parses and validates each item asynchronously across its backend fleet.
OpenAI decouples payload storage from job orchestration. To execute a batch, developers must first write individual request bodies into a JSON Lines (.jsonl) file on local disk, transmit that file using the Files API with purpose set to batch, and then supply the resulting file identifier when instantiating the batch resource. OpenAI then validates the uploaded asset before moving the job into active processing.
Google Gemini combines both patterns through the Gemini Batch API. For compact workloads that keep the entire payload under 20MB, developers can pass an array of inline request objects directly into the batch creation call. For enterprise datasets, Gemini requires uploading a .jsonl document via the File API before referencing its unique URI in a batch job creation request.
Request Formats and Payload Shapes Side by Side
Each provider requires client systems to map individual requests to unique identifiers so that downstream parsers can reassemble records even if processing completes out of order. Here is how request declarations compare across the three platforms.
| Provider | Ingestion Mechanism | Request Identifier Key | Payload Wrapper |
|---|---|---|---|
| Anthropic Claude | Direct HTTP JSON array | custom_id | params object containing standard Messages API keys |
| OpenAI | Pre-uploaded JSONL file | custom_id | body object matching target endpoint schema |
| Google Gemini | Inline array or File API JSONL | key or metadata.key | request object containing GenerateContentRequest keys |
On Anthropic, every item within the batch array includes a custom_id string matching ^[a-zA-Z0-9_-]{1,64}$ and a nested params dictionary. Specific synchronous controls are strictly prohibited: setting stream: true, specifying speed for fast mode, or configuring max_tokens: 0 causes validation to fail. The minimum permitted max_tokens value is 1.
{
"requests": [
{
"custom_id": "request-1",
"params": {
"model": "claude-opus-5-5",
"max_tokens": 1024,
"messages": [{"role": "user", "content": "Hello, world"}]
}
}
]
}
OpenAI packages each operation as an individual line in a .jsonl file. Every record defines the target HTTP method, the endpoint path (such as /v1/chat/completions or /v1/responses), an arbitrary custom_id, and a body payload that mimics the synchronous REST endpoint. Batches cannot mix different models within the same file.
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-3.5-turbo-0125", "messages": [{"role": "user", "content": "Hello world!"}], "max_tokens": 1000}}
Gemini batch requests either embed items directly or structure a .jsonl document where each line pairs a user-defined key with a nested request object. Gemini supports general text generation via gemini-3.8-flash as well as dedicated embedding operations using gemini-embedding-2.
{"key": "request-1", "request": {"contents": [{"parts": [{"text": "Describe the process of photosynthesis."}]}], "generation_config": {"temperature": 0.7}}}
Limits, Expiration Windows, and Data Retention
Understanding throughput and boundary limits is critical when building ingestion workers. Batch pipelines cannot exceed maximum item counts or total byte volumes, and they must handle turnaround deadlines predictably.
| Specification | Anthropic Claude | OpenAI | Google Gemini |
|---|---|---|---|
| Per-Batch Request Limit | 100,000 Message requests | 50,000 requests | Unconstrained per-item cap (subject to file size) |
| File Size / Payload Ceiling | 256 MB | 200 MB | 20MB (inline) or 2GB (File API) |
| Completion Target Window | Most under 1 hour; expires at 24 hours | 24 hours (completion_window: 24h) | Target turnaround 24 hours |
| Results Retention | 29 days after creation | 30 days after completion | Retrievable via job resource |
| Concurrency / Creation Caps | Workspace queue and rate limits apply | 2,000 batches per hour | Standard API quota checks apply |
Anthropic enforces a hard cap of 100,000 requests or 256 MB per batch, whichever ceiling your payload hits first. If a batch remains unfinished when reaching 24 hours from creation, it transitions into an expired state. Results remain accessible for download during a 29 days window, after which output files are purged.
OpenAI restricts individual batch files to 50,000 requests and a 200 MB size limit. For embedding jobs, the entire batch is also constrained to at most 50,000 embedding inputs across all combined records. The job requires setting completion_window to 24h. OpenAI enforces a submission rate limit of 2,000 batches per hour. Completed output artifacts remain available on the Files API for 30 days before deletion.
Gemini provides notable flexibility on upload size, allowing input files up to 2GB through its File API. While inline submissions cannot surpass 20MB, the file-based route easily accommodates massive datasets under a single job descriptor.
Pricing Mechanics and Token Economics
Across Anthropic, OpenAI, and Google Gemini, batch endpoints provide an identical financial incentive: a 50% discount compared to standard synchronous rates. Because the provider processes queries opportunistically against idle computing capacity, users trade immediate latency for substantial cost savings.
For Anthropic workloads, batch pricing applies across both top-tier frontier models and lightweight alternatives. For example, standard synchronous rates for Claude Opus 5.5 run at $4 per million input tokens and $20 per million output tokens. Under the Message Batches API, Claude Opus 5.5 drops to $2 / MTok input and $10 / MTok output. Claude Sonnet 5.5 falls from $2 input and $10 output down to $1 / MTok input and $5 / MTok output. Claude Haiku 4.5 decreases from $1 input and $5 output to $0.50 / MTok input and $2.50 / MTok output, while Claude Fable 5.1 runs at $5 / MTok input and $25 / MTok output.
OpenAI and Gemini similarly apply a 50% discount to asynchronous runs. OpenAI models such as GPT-6.1 Sol list at $2 per million input tokens and $10 per million output tokens for standard synchronous inference, and Gemini 3.8 Flash lists at $0.75 per million input tokens and $3.75 per million output tokens. Running these workloads through asynchronous batching cuts token expenditures in half. To evaluate broader project budgets and model tradeoffs, review <a href=“/tools/llm-api-cost-calculator/”>the LLM API cost calculator</a> and compare standard base rates on <a href=“/tools/llm-api-pricing/”>the LLM API pricing overview</a>.
Prompt caching can further optimize batch runs on platforms that support it. On Anthropic, prompt caching remains active in batch workflows. Because jobs can take longer than 5 minutes to complete, Anthropic recommends evaluating the 1-hour cache duration to maintain hit rates across requests that share extensive system prompts or shared context.
Polling Lifecycle, Status Transitions, and Error Handling
Because batch execution is asynchronous, client architectures must implement robust status tracking and result extraction loops rather than expecting instant feedback.
Anthropic batch jobs initialize with a processing_status of in_progress. Clients poll the retrieve endpoint until the status shifts to ended. Once ended, the batch object yields counts for requests that succeeded, errored, canceled, or expired. Results are retrieved from results_url. If cancellation is required, Anthropic records when the cancel action was initiated.
OpenAI exposes a more granular set of state transitions. A batch moves through validating, in_progress, finalizing, and finally completed. If an invalid file format is detected during startup, the status shifts to failed. If the 24-hour window lapses before all items finish, the status becomes expired. Canceling an in-flight OpenAI batch triggers a cancelling phase that can take up to 10 minutes before reaching cancelled.
When parsing output files from OpenAI, engineers must not assume output order mirrors input line sequence. Responses often complete out of sequence. Each output row contains a custom_id mapping to the input record, alongside an execution object containing token usage figures, finish reasons, and status codes.
Partial expirations require explicit handling on both OpenAI and Claude. If a job expires at the 24 hours boundary, completed requests are still preserved in the output dataset, and organizations are billed only for the tokens consumed by those finished records. Incomplete items are appended to the error output with an expired failure reason, allowing ingestion workers to isolate unprocessed records and retry them in a subsequent batch.
Production Implementation Checklist
- Validate request bodies locally against standard synchronous endpoints prior to batching, ensuring parameters like max_tokens are at least 1 and streaming flags are removed.
- Enforce unique, deterministic custom identifiers matching provider character boundaries (such as 1 to 64 alphanumeric characters on Claude) to safely join completed responses back to internal database records.
- Verify file size thresholds before transmission: keep OpenAI files below 200 MB, Anthropic direct batches under 256 MB, Gemini inline requests under 20MB, and Gemini File API uploads under 2GB.
- Configure polling intervals to check status periodically (such as every 60 seconds) rather than aggressively hammering retrieve endpoints.
- Incorporate a dedicated error-file parser to separate genuine prompt errors from expired items that need re-queuing after the 24-hour turnaround window.
- Monitor organizational limits and spend caps, taking into account that parallel batch throughput can cause aggregate workspace token consumption to briefly outpace configured spend limits.
For teams managing large evaluation datasets, bulk moderation, or repository-wide embeddings, batching provides immediate 50% cost reductions. Review <a href=“/posts/rate-limits-and-usage-tiers-at-anthropic-openai-and-google/”>rate limits and usage tiers</a> and explore <a href=“/posts/files-apis-reusing-uploads-across-claude-gpt-and-gemini/”>files API handling across providers</a> to assemble reliable, automated batch processing pipelines.