AI Gateway Batch Processing

Use experimental_startBatch with a list of requests. Each request needs a unique id, a type of 'text', and the model to run:

import { experimental_startBatch as startBatch, gateway } from 'ai';
 
const instructions = 'Classify the sentiment as positive, negative, or neutral.';
 
const batch = await startBatch({
  provider: gateway,
  requests: [
    {
      id: 'review-1042',
      type: 'text',
      model: 'anthropic/claude-haiku-4.5',
      instructions,
      prompt: 'The checkout flow was fast and painless.',
    },
    {
      id: 'review-1043',
      type: 'text',
      model: 'anthropic/claude-haiku-4.5',
      instructions,
      prompt: 'The app crashed twice before I could pay.',
    },
  ],
});
 
console.log(batch.id, batch.status);

The provider option defaults to AI Gateway when you haven't configured a different global provider, so you can omit it. Pass it explicitly when your application sets a global provider, or when you use a custom gateway instance from createGateway.

The start call stays open until the provider accepts the batch, so large payloads take longer to submit. Once the call returns, batch.status starts as pending.

The returned batch object is a serializable reference containing the batch id, provider, and version. Batches can run for hours, so store it, for example in a database or queue message. Any process can then poll the batch later by passing the stored reference, with the same provider, to the status, results, and cancel functions.

Write one Chat Completions request per line of a JSONL file. Each line has a unique custom_id, method: "POST", url: "/v1/chat/completions", and the request body. Upload the file with purpose: "batch", then create a batch from the returned file ID:

import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const instructions = 'Classify the sentiment as positive, negative, or neutral.';
const reviews = [
  { id: 'review-1042', text: 'The checkout flow was fast and painless.' },
  { id: 'review-1043', text: 'The app crashed twice before I could pay.' },
];
 
const jsonl = reviews
  .map((review) =>
    JSON.stringify({
      custom_id: review.id,
      method: 'POST',
      url: '/v1/chat/completions',
      body: {
        model: 'openai/gpt-5.4-mini',
        messages: [
          { role: 'system', content: instructions },
          { role: 'user', content: review.text },
        ],
      },
    }),
  )
  .join('\n');
 
// Store this key with your job and reuse it if you retry these calls.
const key = 'sentiment-eval-2026-10-20';
 
const file = await client.files.create(
  {
    file: new File([jsonl], 'batch.jsonl', { type: 'application/jsonl' }),
    purpose: 'batch',
  },
  { headers: { 'Idempotency-Key': `${key}-file` } },
);
 
const batch = await client.batches.create(
  {
    input_file_id: file.id,
    endpoint: '/v1/chat/completions',
    completion_window: '24h',
  },
  { headers: { 'Idempotency-Key': `${key}-batch` } },
);
 
console.log(batch.id, batch.status);

Both calls require an Idempotency-Key header. See Retrying safely for how AI Gateway handles retries.

AI Gateway validates every line when you upload the file. All lines must use the same model and the same url, and that url must match the endpoint you pass when you create the batch. completion_window must be 24h. You can also pass metadata with up to 16 string key-value pairs.

Each uploaded file can start one batch. Create the batch within 23 hours of uploading its file. After that, upload the file again with a new idempotency key.

Write one Responses request per line of a JSONL file. Each line has a unique custom_id, method: "POST", url: "/v1/responses", and the request body. Upload the file with purpose: "batch", then create a batch from the returned file ID:

import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const instructions = 'Classify the sentiment as positive, negative, or neutral.';
const reviews = [
  { id: 'review-1042', text: 'The checkout flow was fast and painless.' },
  { id: 'review-1043', text: 'The app crashed twice before I could pay.' },
];
 
const jsonl = reviews
  .map((review) =>
    JSON.stringify({
      custom_id: review.id,
      method: 'POST',
      url: '/v1/responses',
      body: {
        model: 'openai/gpt-5.4-mini',
        instructions,
        input: review.text,
      },
    }),
  )
  .join('\n');
 
// Store this key with your job and reuse it if you retry these calls.
const key = 'sentiment-eval-2026-10-20';
 
const file = await client.files.create(
  {
    file: new File([jsonl], 'batch.jsonl', { type: 'application/jsonl' }),
    purpose: 'batch',
  },
  { headers: { 'Idempotency-Key': `${key}-file` } },
);
 
const batch = await client.batches.create(
  {
    input_file_id: file.id,
    endpoint: '/v1/responses',
    completion_window: '24h',
  },
  { headers: { 'Idempotency-Key': `${key}-batch` } },
);
 
console.log(batch.id, batch.status);

Both calls require an Idempotency-Key header. See Retrying safely for how AI Gateway handles retries.

AI Gateway validates every line when you upload the file. All lines must use the same model and the same url, and that url must match the endpoint you pass when you create the batch. completion_window must be 24h. You can also pass metadata with up to 16 string key-value pairs.

Each uploaded file can start one batch. Create the batch within 23 hours of uploading its file. After that, upload the file again with a new idempotency key.

Use client.messages.batches.create with a list of requests. Each request has a unique custom_id and the Messages API params to run. The call requires an Idempotency-Key header:

import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh',
});
 
const system = 'Classify the sentiment as positive, negative, or neutral.';
 
const batch = await client.messages.batches.create(
  {
    requests: [
      {
        custom_id: 'review-1042',
        params: {
          model: 'anthropic/claude-haiku-4.5',
          max_tokens: 64,
          system,
          messages: [
            { role: 'user', content: 'The checkout flow was fast and painless.' },
          ],
        },
      },
      {
        custom_id: 'review-1043',
        params: {
          model: 'anthropic/claude-haiku-4.5',
          max_tokens: 64,
          system,
          messages: [
            { role: 'user', content: 'The app crashed twice before I could pay.' },
          ],
        },
      },
    ],
  },
  // Store this key with your job and reuse it if you retry this call.
  { headers: { 'Idempotency-Key': 'sentiment-eval-2026-10-20' } },
);
 
console.log(batch.id, batch.processing_status);

For JavaScript and Python versions of this workflow, see Message batches.

Each request accepts the standard text generation options: prompt or messages, instructions, maxOutputTokens, temperature, topP, topK, seed, stopSequences, frequencyPenalty, presencePenalty, reasoning, structured output, and per-request providerOptions. Prompts can include images for vision models, either inline or by URL.

Requests can also define tools with tools and toolChoice. Batch tools are definition-only: the model can return tool calls, but the AI SDK never runs their execute functions or sends tool results back. Pass the same tools to getBatchResults to validate and normalize the returned tool calls.

The following are not supported in batch requests:

• Image generation requests (type: 'image')
• Provider-executed tools, such as web search or MCP servers
• Streaming
• References to provider-hosted resources, such as file IDs, stored responses, or containers. Batch inputs must be self-contained. This restriction doesn't apply when the batch runs on your own provider keys.

Each line's body accepts these Chat Completions fields:

• model (required): an OpenAI model, such as openai/gpt-5.4-mini or gpt-5.4-mini
• messages (required): system, developer, user, assistant, and tool messages with text content. Assistant messages can include function tool_calls.
• max_completion_tokens or max_tokens
• temperature, top_p, seed, frequency_penalty, presence_penalty, and logit_bias
• stop, with up to four sequences
• logprobs and top_logprobs
• reasoning_effort: none, minimal, low, medium, high, or xhigh
• response_format: text, json_object, or json_schema
• tools, tool_choice, and parallel_tool_calls, for function tools
• n, which must be 1
• safety_identifier and user, up to 64 characters each
• stream and store, which you can omit or set to false

The upload returns a 400 error if any line includes another field, such as image or audio content, or a hosted tool like web search. Each custom_id can be 1 to 64 characters.

Each line's body accepts these Responses fields:

• model (required): an OpenAI model, such as openai/gpt-5.4-mini or gpt-5.4-mini
• input (required): a string, or a list of system, developer, user, and assistant messages with text content. The list can also include function_call and function_call_output items.
• instructions
• max_output_tokens
• temperature and top_p
• reasoning, with effort and summary
• text, with format (text, json_object, or json_schema) and verbosity
• tools, tool_choice, and parallel_tool_calls, for function tools
• truncation: auto or disabled
• safety_identifier and user, up to 64 characters each
• stream and store, which you can omit or set to false

The upload returns a 400 error if any line includes another field, such as image or file inputs, a hosted tool like web search, or previous_response_id. Batch inputs must be self-contained, so they can't reference stored responses or conversations. Each custom_id can be 1 to 64 characters.

Each request's params accepts the same fields as POST /v1/messages, with these exceptions:

• stream isn't supported.
• Provider-executed tools, such as web search, and MCP servers aren't supported. Tools you define yourself work, and the model returns their tool calls in the results.
• Server-side model fallbacks aren't supported.
• Citations on user text aren't supported.
• Inside tool_result blocks, citations, search results, plain-text documents, and cache_control aren't supported. Set cache_control on the tool_result block itself instead.

Each custom_id can be 1 to 64 letters, numbers, underscores, or hyphens ([A-Za-z0-9_-]).

Use experimental_getBatchStatus to poll the batch:

import { experimental_getBatchStatus as getBatchStatus, gateway } from 'ai';
import { setTimeout } from 'node:timers/promises';
 
let status = batch.status;
while (status === 'pending') {
  await setTimeout(30_000);
  ({ status } = await getBatchStatus({ provider: gateway, batch }));
  console.log('Batch status:', status);
}

The status is one of:

The status response can also include requestCounts with total, pending, completed, and failed request tallies, and a rawStatus field with the provider's own status string.

All batch functions accept abortSignal, headers, and timeout options. getBatchStatus and getBatchResults also accept maxRetries.

Use client.batches.retrieve to poll the batch until it reaches a terminal status:

import { setTimeout } from 'node:timers/promises';
 
const terminal = ['completed', 'failed', 'expired', 'cancelled'];
 
let current = await client.batches.retrieve(batch.id);
while (!terminal.includes(current.status)) {
  await setTimeout(30_000);
  current = await client.batches.retrieve(batch.id);
  console.log('Batch status:', current.status, current.request_counts);
}

The status is one of:

The batch also reports request_counts with total, completed, and failed request tallies.

Use client.messages.batches.retrieve to poll the batch until processing_status is ended:

import { setTimeout } from 'node:timers/promises';
 
let current = await client.messages.batches.retrieve(batch.id);
while (current.processing_status !== 'ended') {
  await setTimeout(30_000);
  current = await client.messages.batches.retrieve(batch.id);
  console.log('Batch status:', current.processing_status, current.request_counts);
}

The processing status is one of:

The batch also reports request_counts with processing, succeeded, errored, canceled, and expired request tallies.

Once the batch is no longer pending, stream the results with experimental_getBatchResults. Results arrive as an async iterable:

import { experimental_getBatchResults as getBatchResults, gateway } from 'ai';
 
for await (const item of getBatchResults({ provider: gateway, batch })) {
  if (item.status !== 'succeeded') {
    console.error(item.id, item.status, item.error);
  } else if (item.type === 'text') {
    console.log(item.id, item.text, item.usage);
  }
}

Each item reports its own outcome:

Once the batch reaches a terminal status, download its output and error files. The file at output_file_id holds the requests that succeeded, and the file at error_file_id holds the requests that failed. Each line uses the OpenAI batch output format, with your custom_id and the full Chat Completions response in response.body:

const done = await client.batches.retrieve(batch.id);
 
for (const fileId of [done.output_file_id, done.error_file_id]) {
  if (!fileId) continue;
  const content = await client.files.content(fileId);
  for (const line of (await content.text()).split('\n').filter(Boolean)) {
    const item = JSON.parse(line);
    if (item.error || item.response.status_code !== 200) {
      console.error(item.custom_id, item.error ?? item.response.body.error);
    } else {
      console.log(item.custom_id, item.response.body.choices[0].message.content);
    }
  }
}

You can download output and error files until the expires_at time on the file object.

Once the batch reaches a terminal status, download its output and error files. The file at output_file_id holds the requests that succeeded, and the file at error_file_id holds the requests that failed. Each line uses the OpenAI batch output format, with your custom_id and the full Responses object in response.body:

const done = await client.batches.retrieve(batch.id);
 
for (const fileId of [done.output_file_id, done.error_file_id]) {
  if (!fileId) continue;
  const content = await client.files.content(fileId);
  for (const line of (await content.text()).split('\n').filter(Boolean)) {
    const item = JSON.parse(line);
    if (item.error || item.response.status_code !== 200) {
      console.error(item.custom_id, item.error ?? item.response.body.error);
      continue;
    }
    let text = '';
    for (const output of item.response.body.output) {
      if (output.type !== 'message') continue;
      for (const part of output.content) {
        if (part.type === 'output_text') text += part.text;
      }
    }
    console.log(item.custom_id, text);
  }
}

The raw Responses object doesn't include the SDK's output_text helper, so the example collects the text from the output items. You can download output and error files until the expires_at time on the file object.

Once processing_status is ended, a non-null results_url means results are available. Read them with client.messages.batches.results, which returns an async iterable:

const done = await client.messages.batches.retrieve(batch.id);
if (!done.results_url) {
  throw new Error(`No downloadable results for batch ${batch.id}.`);
}
 
const results = await client.messages.batches.results(batch.id);
for await (const item of results) {
  if (item.result.type === 'succeeded') {
    console.log(item.custom_id, item.result.message.content);
  } else {
    console.error(item.custom_id, item.result);
  }
}

Each item's result.type reports its outcome:

Use experimental_cancelBatch to stop a batch that's still pending:

import { experimental_cancelBatch as cancelBatch, gateway } from 'ai';
 
await cancelBatch({ provider: gateway, batch });

The batch stays pending until the provider confirms the cancellation. It then reports failed with a cancellation error.

Use client.batches.cancel:

await client.batches.cancel(batch.id);

The batch reports cancelling until the provider confirms the cancellation, and then cancelled. Requests that finished before the cancellation appear in the output file.

Use client.messages.batches.cancel:

await client.messages.batches.cancel(batch.id);

The batch reports canceling until the provider confirms the cancellation, and then ended. Requests the provider didn't run have a canceled result.

AI Gateway doesn't support experimental_listBatches, so store the batch references that startBatch returns.

Use client.batches.list. The SDK follows the after cursor to fetch later pages:

for await (const item of client.batches.list({ limit: 20 })) {
  console.log(item.id, item.status);
}

limit accepts 1 to 100.

Use client.messages.batches.list:

for await (const item of client.messages.batches.list({ limit: 20 })) {
  console.log(item.id, item.processing_status);
}

The default page size is 20, with a maximum of 100. To paginate manually, pass either after_id or before_id, never both.

Starting a batch with the AI SDK is not idempotent by default: calling startBatch twice submits two batches. To make retries safe, pass an idempotency key. AI Gateway returns the existing batch instead of creating a duplicate when it sees the same key with the same payload, and rejects a reused key whose payload differs:

import { experimental_startBatch as startBatch, gateway } from 'ai';
 
const batch = await startBatch({
  provider: gateway,
  requests,
  providerOptions: {
    gateway: {
      idempotencyKey: 'nightly-eval-2026-08-20',
    },
  },
});

POST /v1/files and POST /v1/batches both require an Idempotency-Key header. Use one key for each file upload and one for each batch, and resend the same key when you retry:

• Retrying an upload with the same key returns the original file. If the original upload is still in progress, AI Gateway returns a 409 error with a Retry-After header.
• Retrying batch creation with the same key returns the original batch.
• Reusing a key for a different request returns a 409 error.
• Each input file can start only one batch. Creating a second batch from the same file with a new key returns a 409 error.

If batch creation fails with an x-should-retry: false response header, the provider may have accepted the batch. Retry with the same key, which never submits a second batch, or check client.batches.list before you create a new batch with a new key.

POST /v1/messages/batches requires an Idempotency-Key header. Retrying with the same key and the same payload returns the existing batch. Reusing the key with a different payload returns an error.

If batch creation fails with an x-should-retry: false response header, the provider may have accepted the batch. Retry with the same key, which never submits a second batch, or check client.messages.batches.list before you submit again with a new key.