Amazon Bedrock Batch Inference: JSONL Format, Python and CLI, IAM Service Role, Job Statuses, Costs at 50% Off, Embeddings, Quotas and Every Error

Amazon Bedrock batch inference runs thousands of prompts in one job, without you calling the model for each one, at about half the on-demand price for supported models. You write the prompts to a JSONL file (one line per prompt: {"recordId": "...", "modelInput": {...}}), upload it to S3, and start a job with CreateModelInvocationJob, naming the model, the input and output S3 locations, and a service role Bedrock assumes to read and write them. The job moves from Submitted through Validating, Scheduled and InProgress to Completed, and writes one .jsonl.out file per input file plus a manifest.json.out summary. The modelInput can use each model's own request format (the InvokeModel type, the default) or the common Converse format. Results are not instant and not in input order, so match them by recordId. Below: the JSON format with examples, a complete Python script, the CLI, IAM and service role policies, statuses, quotas, embeddings in batch, EventBridge notifications, costs with worked numbers, and every common error.

Jake's computer shop has eight years of repair notes in a spreadsheet: 18,000 rows of things like "rplcd kb, sticky keys, cust says coffee." Ethan wanted every note tagged with a category, the part replaced and whether it looked like a warranty case, so the shop could finally see which laptops come back most and why. Calling a model 18,000 times from a script would work, slowly, and at full price. Bedrock's batch inference does the same job overnight for half the cost, as long as you get one JSON format right. That format, and the rest of the setup, is what this page covers.

New to Bedrock? Our plain-English Bedrock explainer covers what it is and how it bills, and the cards below fill in the rest.

What batch inference is, and when to use it

Normally you call a model and wait for the answer: that is on-demand (or real-time) inference, and it is what a chatbot needs. Batch inference is the other way round: you hand Bedrock a file of prompts and come back later for a file of answers. Bedrock processes the job when it has spare capacity, which is why it can charge less.

Batch fits work where nobody is waiting for each individual answer:

• Tagging, classifying or summarizing a backlog: support tickets, reviews, repair notes, call transcripts.
• Extracting fields from thousands of documents into a spreadsheet.
• Creating embeddings for a large document set before loading a vector database.
• Generating product descriptions, translations or test data in bulk.
• Running an evaluation set of hundreds of questions against several models.

It does not fit anything interactive, or small jobs: there is a minimum number of records per job, and even a small job spends time validating and waiting in the queue before it runs.

Jake: "So it's like dropping a stack of forms at the office and picking them up tomorrow?"

Ethan: "Exactly that. And the office charges half because it can do them whenever it's quiet."

How much batch inference costs, with worked numbers

For models that support it, batch inference is billed per token like on-demand, at roughly half the on-demand price. Each model has its own batch rate, roughly half of its on-demand one, so check the rate for the exact model you plan to use. There is no separate charge for the job itself; you pay normal S3 costs for the files, which for text are pennies.

Ethan's 18,000 repair notes average about 600 input tokens each (the instructions plus the note) and 100 output tokens (a short JSON answer). That is 10.8 million input tokens and 1.8 million output tokens:

Two lessons from the table. The model choice moves the bill far more than batch does, so test a cheap model on a sample first. And the output side is where long answers get expensive: asking for a compact JSON answer instead of a paragraph keeps output tokens, and cost, low. Our Bedrock pricing guide covers the other levers.

The whole process in six steps

1. Write the prompts to a JSONL file, one JSON object per line, each with a recordId and a modelInput.
2. Upload the file to S3, in a bucket in the same Region as the job. Several files in one folder are fine; the job reads all the JSONL files there.
3. Set up a service role that Bedrock can assume, with permission to read the input and write the output (the console can create it for you).
4. Create the job in the console, with the CLI, or from Python, naming the model, the input and output locations, the role and the input format.
5. Wait and watch: poll the job status, or get an EventBridge notification when it changes.
6. Read the results from S3: one output file per input file, matched to your prompts by recordId.

The rest of the page takes each step in turn, starting with the part that causes most failed jobs: the format.

The batch inference JSON format, with examples

The input is JSONL ("JSON Lines"): a text file where every line is one complete JSON object, with no commas between lines and no surrounding brackets. The file name must end in .jsonl. Each line has two fields:

• recordId: your identifier for this prompt, such as a row number or ticket ID. It is optional, and Bedrock adds one if you leave it out, but always set it: results come back in no guaranteed order, and the ID is how you match each answer to its source.
• modelInput: the request body, in one of two formats, chosen when you create the job.

Converse format (job invocation type Converse): the same body as Bedrock's Converse API, identical for every model. This is the easiest choice, because switching models never changes your file:

{"recordId": "NOTE-000001", "modelInput": {"system": [{"text": "Classify the repair note. Reply with JSON only: {\"category\": ..., \"part\": ..., \"warranty_likely\": true|false}"}], "messages": [{"role": "user", "content": [{"text": "rplcd kb, sticky keys, cust says coffee"}]}], "inferenceConfig": {"maxTokens": 120, "temperature": 0}}}
{"recordId": "NOTE-000002", "modelInput": {"system": [{"text": "Classify the repair note. Reply with JSON only: {\"category\": ..., \"part\": ..., \"warranty_likely\": true|false}"}], "messages": [{"role": "user", "content": [{"text": "no power, dc jack loose, 3 mo old"}]}], "inferenceConfig": {"maxTokens": 120, "temperature": 0}}}

(Each record is one line. They wrap on screen here, but in the file each must be on a single line.)

InvokeModel format (the default type): each model's own native request body, exactly what you would send to InvokeModel for that model. For Amazon Nova it looks like this:

{"recordId": "NOTE-000001", "modelInput": {"schemaVersion": "messages-v1", "system": [{"text": "Classify the repair note..."}], "messages": [{"role": "user", "content": [{"text": "rplcd kb, sticky keys, cust says coffee"}]}], "inferenceConfig": {"maxTokens": 120, "temperature": 0}}}

Other providers' native formats differ (Anthropic's uses anthropic_version and max_tokens, for example), which is the main reason to prefer the Converse type: one format for every model.

When you need the InvokeModel type anyway: tool calling. Batch jobs of the Converse type do not support tools, and records that include a toolConfig fail with a 400 error. To use tools in batch, choose the InvokeModel type and write each record in the model's native format. Structured output (getting validated JSON back) works with both types.

Images and video: models that read images or video, such as Amazon Nova Lite and Pro, accept them in batch, either inline as bytes or as S3 locations. When records point at S3 files, keep all of them in the same bucket and folder as the JSONL, and give the job the folder as its input location, not the single JSONL file. S3 paths are case-sensitive, a frequent cause of "file not found" errors in otherwise correct jobs.

A complete Python script: build, upload, start, wait, read

This script does the whole job for a CSV of notes. It uses boto3 (pip install boto3) and your normal AWS credentials:

import csv, json, time
import boto3

REGION = "us-east-1"
BUCKET = "shop-batch-jobs"                     # an S3 bucket in the same Region
ROLE_ARN = "arn:aws:iam::123456789012:role/BedrockBatchRole"
MODEL_ID = "us.amazon.nova-lite-v1:0"
SYSTEM = ("Classify the repair note. Reply with JSON only: "
          '{"category": "...", "part": "...", "warranty_likely": true or false}')

encode = json.JSONEncoder().encode            # one JSON object per line

# 1. Build the JSONL file
with open("notes.csv", newline="", encoding="utf-8") as src, \
     open("notes.jsonl", "w", encoding="utf-8") as out:
    for row in csv.DictReader(src):
        record = {
            "recordId": row["note_id"],
            "modelInput": {
                "system": [{"text": SYSTEM}],
                "messages": [{"role": "user", "content": [{"text": row["note"]}]}],
                "inferenceConfig": {"maxTokens": 120, "temperature": 0},
            },
        }
        out.write(encode(record) + "\n")

# 2. Upload it
s3 = boto3.client("s3", region_name=REGION)
s3.upload_file("notes.jsonl", BUCKET, "input/notes.jsonl")

# 3. Start the job
bedrock = boto3.client("bedrock", region_name=REGION)
job = bedrock.create_model_invocation_job(
    jobName=f"repair-notes-{int(time.time())}",
    roleArn=ROLE_ARN,
    modelId=MODEL_ID,
    modelInvocationType="Converse",
    inputDataConfig={"s3InputDataConfig": {"s3Uri": f"s3://{BUCKET}/input/"}},
    outputDataConfig={"s3OutputDataConfig": {"s3Uri": f"s3://{BUCKET}/output/"}},
)
arn = job["jobArn"]
print("Started", arn)

# 4. Wait (or use EventBridge instead of polling)
while True:
    info = bedrock.get_model_invocation_job(jobIdentifier=arn)
    status = info["status"]
    print(status, info.get("processedRecordCount", 0), "/", info.get("totalRecordCount", 0))
    if status in ("Completed", "PartiallyCompleted", "Failed", "Stopped", "Expired"):
        break
    time.sleep(300)

Note the client: jobs are created with boto3.client("bedrock"), the control-plane client, not bedrock-runtime, which is for on-demand calls. Mixing them up gives an "object has no attribute create_model_invocation_job" error. Checking every five minutes is plenty; the progress counters can lag by up to a minute anyway.

Reading the results once the job finishes:

job_id = arn.split("/")[-1]
prefix = f"output/{job_id}/"
answers, failures = {}, {}

for obj in s3.list_objects_v2(Bucket=BUCKET, Prefix=prefix).get("Contents", []):
    key = obj["Key"]
    body = s3.get_object(Bucket=BUCKET, Key=key)["Body"].read().decode("utf-8")
    if key.endswith("manifest.json.out"):
        print("Summary:", json.loads(body))
        continue
    for line in body.splitlines():
        rec = json.loads(line)
        if "error" in rec:
            failures[rec["recordId"]] = rec["error"]
        else:
            msg = rec["modelOutput"]["output"]["message"]
            answers[rec["recordId"]] = msg["content"][0]["text"]

print(len(answers), "answered,", len(failures), "failed")

Records that failed carry an error object instead of a modelOutput; collect their IDs and resubmit just those in a smaller follow-up job.

Jake: "Eighteen thousand notes, and the script is one page?"

Ethan: "One page, and most of it is reading the spreadsheet. The model part is three lines."

Starting a job with the AWS CLI or the console

The same job from the command line, after uploading the JSONL:

aws bedrock create-model-invocation-job \
  --region us-east-1 \
  --job-name repair-notes-oct19 \
  --role-arn arn:aws:iam::123456789012:role/BedrockBatchRole \
  --model-id us.amazon.nova-lite-v1:0 \
  --model-invocation-type Converse \
  --input-data-config '{"s3InputDataConfig": {"s3Uri": "s3://shop-batch-jobs/input/"}}' \
  --output-data-config '{"s3OutputDataConfig": {"s3Uri": "s3://shop-batch-jobs/output/"}}'

aws bedrock get-model-invocation-job --region us-east-1 --job-identifier JOB_ARN
aws bedrock list-model-invocation-jobs --region us-east-1 --max-results 10
aws bedrock stop-model-invocation-job --region us-east-1 --job-identifier JOB_ARN

On Windows PowerShell, put each JSON value in a file and pass file://input.json instead, which avoids quoting trouble.

In the console: open Bedrock, choose Batch inference in the left menu, then Create job. Name the job, pick the model, choose the Model invocation type (InvokeModel or Converse), browse to the S3 input and output locations, and either pick an existing service role or let the console create one. Two limits of the console: input or output buckets in a different AWS account, and running the job inside a VPC, both require the API.

Optional settings worth knowing: timeoutDurationInHours sets how long the job may run before it is cut off, tags label it for cost reports, vpcConfig keeps the job's data access inside your VPC, and clientRequestToken stops a retried script from accidentally starting the same job twice.

Already using OpenAI's Batch API? Bedrock also accepts batch jobs in that format through its OpenAI-compatible endpoint, so existing scripts can move with fewer changes. It is a separate path from CreateModelInvocationJob, suited to OpenAI models on Bedrock and to teams standardizing on OpenAI-style tooling. For anything new, the native jobs on this page give you the full set of models, S3 handling and EventBridge events.

Writing prompts that work well in batch

In a batch job nobody reads the answers one by one as they arrive, so problems surface only when you open 18,000 results at once. A few habits keep that moment pleasant:

• Fix the output format. Ask for a specific JSON shape, list the allowed values ("category must be one of: keyboard, battery, screen, charging, liquid, other"), and keep it short. Free-text answers are hard to count; categories are easy.
• Use temperature 0. Classification and extraction should give the same answer for the same input. Creativity is for product descriptions, not tagging.
• Keep the instructions short. The system prompt is repeated in every record, so 500 words of instructions across 18,000 records is 9 million extra input tokens. Say it once, clearly, with one or two examples.
• Cap the output. Set maxTokens just above what a good answer needs. It limits cost and stops a confused model from rambling.
• Allow "unknown." Give the model a way to say a note is empty or unclear, or it will guess. Ethan's 36 empty notes came back as clean "unknown" answers rather than invented categories.
• Try 50 records on-demand first. Run a small sample through the normal API, read every answer, fix the prompt, and only then submit the full job.

If you need guaranteed JSON, use the model's structured output support, which works in both job types; it rejects malformed answers instead of leaving you to clean them up afterwards.

IAM: your permissions and the batch service role

Two identities are involved, and most permission errors come from mixing them up.

1. You (or your script) need permission to create and manage jobs, and to pass the service role to Bedrock:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:CreateModelInvocationJob",
        "bedrock:GetModelInvocationJob",
        "bedrock:ListModelInvocationJobs",
        "bedrock:StopModelInvocationJob",
        "bedrock:ListFoundationModels",
        "bedrock:GetFoundationModel",
        "bedrock:ListInferenceProfiles",
        "bedrock:GetInferenceProfile"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/BedrockBatchRole"
    }
  ]
}

2. The service role is the identity Bedrock uses while the job runs. Its trust policy lets Bedrock assume it, ideally only for batch jobs in your account:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "bedrock.amazonaws.com"},
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {"aws:SourceAccount": "123456789012"},
      "ArnEquals": {"aws:SourceArn": "arn:aws:bedrock:us-east-1:123456789012:model-invocation-job/*"}
    }
  }]
}

Its permissions policy needs S3 access to both locations, and, when you use an inference profile such as us.amazon.nova-lite-v1:0, permission to invoke the profile and the model in every Region the profile can use:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::shop-batch-jobs", "arn:aws:s3:::shop-batch-jobs/*"],
      "Condition": {"StringEquals": {"aws:ResourceAccount": ["123456789012"]}}
    },
    {
      "Effect": "Allow",
      "Action": "bedrock:InvokeModel",
      "Resource": [
        "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.amazon.nova-lite-v1:0",
        "arn:aws:bedrock:*::foundation-model/amazon.nova-lite-v1:0"
      ]
    }
  ]
}

Encrypted buckets. If the input bucket uses a customer managed KMS key, the role also needs kms:Decrypt; if you choose a customer managed key for the output, it needs kms:GenerateDataKey. Missing key permissions produce a misleading failure: the job is accepted, then fails a few minutes later with a message about validating GetObject or PutObject permissions on the bucket, without mentioning KMS. If the S3 policy looks right, check the key.

Organization conditions. Foundation models are owned by AWS, not by your account, so a policy condition such as aws:ResourceOrgID on model resources makes batch jobs fail with AccessDeniedException. Remove that condition from statements that cover foundation models, or put models in a separate statement without it.

Batch inference job statuses

How long does batch inference take? There is no fixed completion time or SLA: batch work runs when capacity is available, which is the trade for the lower price. Small jobs often finish within an hour or two, and large ones over several hours; plan for "by tomorrow" rather than "by lunch," and use timeoutDurationInHours to bound how long a job may run. During InProgress, processedRecordCount divided by totalRecordCount gives the percentage done.

Get notified when a job finishes (EventBridge)

Instead of polling, let Bedrock tell you. Every status change emits an event to EventBridge's default bus with source aws.bedrock and detail-type Batch Inference Job State Change. A rule for finished jobs:

{
  "source": ["aws.bedrock"],
  "detail-type": ["Batch Inference Job State Change"],
  "detail": {
    "status": ["Completed", "PartiallyCompleted", "Failed", "Expired"]
  }
}

Save that pattern as pattern.json and create the rule and its target from the CLI:

aws events put-rule --name bedrock-batch-finished --event-pattern file://pattern.json
aws events put-targets --rule bedrock-batch-finished \
  --targets Id=email,Arn=arn:aws:sns:us-east-1:123456789012:batch-alerts

The SNS topic's access policy must allow events.amazonaws.com to publish to it; the console adds that automatically when you pick the topic there.

Point the rule at an SNS topic to get an email, or at a Lambda function that reads the results and writes them to your database. The event's detail includes batchJobArn, batchJobName, batchModelId, status and, on failure, failureMessage. These events cost nothing to receive. Our EventBridge explainer covers rules and targets if they are new to you.

Run it every night: a hands-off pipeline

Once the one-off backlog is done, the same pieces can tag each day's new notes automatically. Ethan's version has no server at all:

1. A scheduled rule in EventBridge Scheduler runs a small Lambda function at 7 p.m. every day.
2. That function exports the day's new notes to a JSONL file in S3 and starts a batch job with create_model_invocation_job, using the same service role as before. If there are fewer notes than the batch minimum, it calls the model directly instead.
3. The EventBridge rule above fires when the job reaches Completed or PartiallyCompleted and starts a second Lambda function.
4. The second function reads the .jsonl.out files, parses each answer, and writes the tags back next to the notes (a database table, or a CSV the spreadsheet imports).
5. Failed records are written to a separate file, and Failed or Expired jobs send an email through SNS, so a problem is noticed the next morning rather than the next month.

Each Lambda function needs only a few seconds per run, so the whole pipeline costs pennies beyond the tokens. Give the first function permission to create jobs and pass the service role, and the second permission to read the output prefix.

Reading the results: .jsonl.out and manifest.json.out

Results land under your output location, in a folder named after the job ID:

• One output file per input file, named after it with .out added (for example notes.jsonl.out). Each line holds the recordId, the original modelInput, and either a modelOutput or an error object with errorCode and errorMessage.
• manifest.json.out, a summary: totalRecordCount, processedRecordCount, successRecordCount, errorRecordCount, inputTokenCount and outputTokenCount. The token counts are the quickest way to check the bill before it arrives.

The shape of modelOutput follows the job type: for Converse jobs it is a Converse response (the text is at output.message.content[0].text, with token usage under usage); for InvokeModel jobs it is that model's native response body. Because lines are not in input order, always build a dictionary keyed by recordId before joining results back to your data.

If you asked for JSON answers, as Ethan did, parse each answer with a fallback: a small share of model outputs will have stray text around the JSON. Count those, inspect a few, and tighten the prompt for the next run.

Embeddings in batch

Batch is ideal for embedding a large document set, because embeddings are pure bulk work. With Titan Text Embeddings V2, use the InvokeModel type and the model's native body:

{"recordId": "DOC-0001-CHUNK-03", "modelInput": {"inputText": "Refurbished Lenovo laptops have a 12-month warranty...", "dimensions": 512, "normalize": true}}

Each output line then carries the vector in modelOutput.embedding. Use the same model and the same dimensions value you will use at search time; vectors from different settings cannot be compared. Splitting documents into chunks happens before the batch, so put the chunk position in the recordId (as above), and reassembling them afterwards is a simple lookup.

The economics are striking. Titan Text Embeddings V2 costs about $0.02 per million tokens on demand, so 10,000 chunks of 400 tokens (4 million tokens) cost around 8 cents, and less in batch. The reason to batch embeddings is less the price than the throughput: one job instead of 10,000 calls that each count against your per-minute quota. After the job, load the vectors and their chunk text into your vector store in one bulk import, keyed by the same recordId.

Keep the chunk text itself in your own records, not only in the batch output: the output repeats the modelInput, which is convenient, but your index should not depend on keeping a batch job's S3 folder forever.

Splitting very large jobs

When your data is bigger than one job allows, split it into several JSONL files and jobs. A few lines of Python cut a large file into parts of a chosen size:

PART_SIZE = 20000   # records per part; keep it under your per-job quota

with open("all_notes.jsonl", encoding="utf-8") as src:
    part, count, out = 0, 0, None
    for line in src:
        if count % PART_SIZE == 0:
            if out:
                out.close()
            part += 1
            out = open(f"notes_part{part:03}.jsonl", "w", encoding="utf-8")
        out.write(line)
        count += 1
    if out:
        out.close()
print(part, "parts written")

Upload each part to its own S3 folder and start one job per folder, keeping under your quota for jobs in progress at once. Jobs run independently, so one failed part only means rerunning that part. Name jobs and folders with the part number, and the results reassemble by recordId as usual.

For dozens of parts, an AWS Step Functions workflow is the tidy way to run them: it starts jobs up to your concurrency limit, waits for each to finish, retries failures, and collects the results, with every step visible in one diagram. AWS publishes a sample workflow for exactly this kind of batch orchestration, and it is a good starting point to adapt.

Tracking batch costs per project: tags and application inference profiles

When several teams or projects run batch jobs in one account, the Bedrock line on the bill is one big number. Two tools split it:

• Job tags. Add tags such as project=repair-notes when you create each job. Activate the tag key as a cost allocation tag in the Billing console, and Cost Explorer can then group spending by it.
• Application inference profiles. An application inference profile is a model ID you create yourself, pointing at a model, and you can tag it. Use its ARN as the job's modelId, and every token through it is attributed to its tags, in batch and on-demand alike.

aws bedrock create-inference-profile \
  --inference-profile-name repair-notes-nova-lite \
  --model-source copyFrom=arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0 \
  --tags key=project,value=repair-notes

The command returns an ARN ending in application-inference-profile/...; use it as the model in your job, and give the service role bedrock:InvokeModel on that ARN as well as on the model. For the quick check without any setup, the inputTokenCount and outputTokenCount in each job's manifest.json.out, multiplied by the model's batch rates, give the cost of that job within a few cents.

Using batch to compare models

Batch is also the cheapest way to answer "which model should we use?" with your own data. Because a Converse-format JSONL file works with any model, the same file can be submitted twice, once per candidate model, with no changes.

Ethan did this before the big run: 500 notes, already tagged by hand, sent through Nova Lite and Nova Pro as two jobs. Comparing each model's answers with the hand tags by recordId gave an accuracy figure for each, and the manifest token counts gave the cost of each. Put side by side, the two numbers show which judgments the cheaper model handles well enough and where a larger model earns its price, for your data rather than in general. That evidence took one evening and less than a dollar, and it is far better grounds for a decision than a public benchmark.

Run comparisons at temperature 0, with identical prompts, on records with known answers, and include some hard cases on purpose: easy records make every model look the same.

Keeping batch data safe

Batch files often hold exactly the data you would least like to leak: customer notes, tickets, transcripts. The defaults are sound, and a few settings make them tighter:

• Your data stays in your S3 buckets, and Bedrock does not use it to train models or share it with model providers.
• Encrypt the buckets with a KMS key if your policies require customer managed keys, and give the service role the two key permissions described above.
• Use vpcConfig when the job's access to S3 must stay inside your VPC, which is common in regulated environments.
• Delete what you no longer need. An S3 lifecycle rule that removes input/ and output/ objects after, say, 30 days stops old batches from piling up as a forgotten copy of sensitive data.
• Scope the service role to the specific buckets and, once created, specific job ARNs, rather than *.

For personal data in particular, consider stripping what the model does not need before writing the JSONL: a repair note does not need the customer's phone number to be classified.

Limits and quotas

Batch inference has its own quotas, separate from on-demand ones, and they vary by model and Region:

• Minimum number of records per job. Too few records and the job fails validation; for a handful of prompts, call the model directly.
• Records per input file and records per job, the maximum counts.
• Input file size and total job size.
• Jobs in progress at the same time.

To see your current values, open the Service Quotas console, choose Amazon Bedrock, and search for "batch," or list them from the CLI:

aws service-quotas list-service-quotas --service-code bedrock --region us-east-1 \
  --query "Quotas[?contains(QuotaName, 'atch')].[QuotaName,Value]" --output table

Jobs larger than the per-job maximum are split into several jobs, each with its own files; a Step Functions workflow or a simple script that starts the next job when the previous one completes handles that well. Batch inference is not available for provisioned (Provisioned Throughput) models.

Common batch inference errors and fixes

Before a big run, submit a test job just above the minimum record count with a sample of your real data. It catches format, permission and prompt problems for a few cents instead of after eight hours.

Frequently asked questions

Ethan's job went in at 6 p.m. By breakfast, the manifest said 17,964 records succeeded and 36 failed, all of them notes with nothing in them but a customer's name. The bill for the night was under a dollar on Nova Lite. By lunch, the shop had its first real chart: keyboards from one laptop model came back three times as often as any other part. Jake called the supplier that afternoon. "Eight years of notes," he said, "and the answer was in there the whole time."

Revision note. Written October 19, 2026. Batch inference rewards patience twice: once on the bill, and once more when you finally read eight years of your own notes in one chart.