DynamoDB BatchWriteItem UnprocessedItems: Why Writes Vanish Without an Error, "Provided List of Item Keys Contains Duplicates," and a Retry Loop That Works

DynamoDB's BatchWriteItem writes or deletes up to 25 items in one call, and it has a habit that loses data quietly: when some of those writes are throttled, the call still succeeds, with no error, and the writes that didn't happen come back in a field called UnprocessedItems. Code that doesn't check that field drops them. The fix is a loop: send the batch, and while UnprocessedItems isn't empty, wait a little longer each time and send what's left, because it's already in the right format to send again. The other errors you'll meet reject the whole batch instead: "Too many items requested for the BatchWriteItem call" means more than 25, and "Provided list of item keys contains duplicates" means the same key appears twice in one batch, even once as a put and once as a delete. This page has a retry loop for Python and Node.js, what boto3's batch_writer does and doesn't do for you, the AWS CLI check, and every error that rejects a batch, with its fix.

Ethan, a developer, moves the day's repair tickets for his friend Jake's phone repair shop into DynamoDB every night with a small script. One morning Jake asked where Tuesday's tickets were. The script's log said it had finished without an error. It had, and about a third of the tickets weren't there.

New to DynamoDB? Amazon DynamoDB is AWS's serverless database: you store items in tables, and you pay for the reads and writes you make. BatchWriteItem is how you write many items at once.

How BatchWriteItem works, and its limits

BatchWriteItem takes a list of PutRequest and DeleteRequest entries, for one table or several, and runs them in parallel on DynamoDB's side. That makes it much faster than a loop of single writes for loading data. It also comes with rules that single writes don't have:

If you need conditions or all-or-nothing behavior, BatchWriteItem is the wrong tool: TransactWriteItems gives you both, at a higher cost per write. Our ConditionalCheckFailedException guide covers when that trade is worth it.

The silent part: UnprocessedItems

When a batch is partly throttled, because the table or a partition is out of write capacity, or when DynamoDB has an internal problem with some of the requests, BatchWriteItem doesn't fail. It writes what it can and returns the rest:

{
  "UnprocessedItems": {
    "Repairs": [
      { "PutRequest": { "Item": { "ticket": { "S": "T-1043" }, "device": { "S": "iPhone 16" } } } }
    ]
  }
}

The response looks like a success, and to the SDK it is one: there's no exception to catch. So a script that sends batches and moves on loses every item listed there, and the log says everything went fine. That's what happened to Ethan's Tuesday tickets: a big evening of repairs, a table on provisioned capacity, and a third of the writes throttled into a field the script never read.

Only when DynamoDB can't process any of the batch because of throttling does the call fail, with ProvisionedThroughputExceededException or ThrottlingException. The SDKs retry those on their own. They don't retry UnprocessedItems, because nothing failed as far as they can tell.

Jake: "So it said 'done' and meant 'mostly done'?"

Ethan: "And it told me exactly which ones it skipped. I just never looked."

A retry loop that doesn't lose writes (Python)

The good news is that UnprocessedItems is in the same format as the request, so you can send it straight back. The rule is to wait between attempts, and to wait longer each time, because resending at once usually gets throttled again. Here's a complete helper for the boto3 client:

import random
import time

import boto3

dynamodb = boto3.client("dynamodb")


def write_all(table_name, requests, max_attempts=8):
    """Send PutRequest/DeleteRequest dicts in batches of 25, resending UnprocessedItems with backoff."""
    for start in range(0, len(requests), 25):
        pending = {table_name: requests[start:start + 25]}
        for attempt in range(max_attempts):
            response = dynamodb.batch_write_item(RequestItems=pending)
            pending = response.get("UnprocessedItems") or {}
            if not pending:
                break
            left = sum(len(v) for v in pending.values())
            delay = min(2 ** attempt * 0.05, 5) * random.uniform(0.5, 1.5)
            print(f"{left} unprocessed, retrying in {delay:.2f}s")
            time.sleep(delay)
        else:
            raise RuntimeError(f"Still unprocessed after {max_attempts} attempts: {pending}")

What it does, step by step:

1. Splits the list into batches of 25, the most one call accepts.
2. Sends a batch and replaces the pending list with whatever came back in UnprocessedItems.
3. Stops when nothing is left, which is the normal case.
4. Otherwise waits 0.05 seconds, then 0.1, 0.2 and so on up to 5 seconds, with some randomness so several workers don't retry in step, and sends only the leftovers.
5. Raises an error after eight attempts instead of dropping data quietly, so a long throttling spell shows up in your logs.

Each request is a dictionary in DynamoDB's typed format, for example {"PutRequest": {"Item": {"ticket": {"S": "T-1042"}}}}. Loading 60 tickets with it takes three calls, and when a response holds back one item, the next call sends exactly that one item.

boto3's batch_writer: what it does for you, and what it doesn't

If you use boto3's table resource, table.batch_writer() does most of this for you: it collects your writes, sends them 25 at a time, and puts any UnprocessedItems back into its queue to go out with the next batch. Leaving the with block sends everything that's still waiting.

import boto3

table = boto3.resource("dynamodb").Table("Repairs")

with table.batch_writer(overwrite_by_pkeys=["ticket"]) as batch:
    for row in tickets:
        batch.put_item(Item=row)

Two details matter. First, it resends unprocessed items with the next batch without pausing. For a table that's briefly busy, that's fine. Under sustained throttling, it keeps hammering, and the clean fix is more capacity or on-demand mode, not a faster loop. Second, it doesn't remove duplicate keys unless you pass overwrite_by_pkeys. Without it, two writes to the same key in one batch fail the whole batch:

ValidationException - Provided list of item keys contains duplicates

With overwrite_by_pkeys=["ticket"], the later write wins: a ticket put as "received" and then as "repaired" in the same run ends up "repaired". List every key attribute, for example ["pk", "sk"] for a table with a sort key.

The same loop in Node.js (AWS SDK v3)

In Node.js, the document client from @aws-sdk/lib-dynamodb lets you write items as plain objects, and the loop is the same idea:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, BatchWriteCommand } from "@aws-sdk/lib-dynamodb";

const doc = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function writeAll(tableName, requests, maxAttempts = 8) {
  for (let start = 0; start < requests.length; start += 25) {
    let pending = { [tableName]: requests.slice(start, start + 25) };
    for (let attempt = 0; ; attempt++) {
      const out = await doc.send(new BatchWriteCommand({ RequestItems: pending }));
      pending = out.UnprocessedItems ?? {};
      if (Object.keys(pending).length === 0) break;
      if (attempt + 1 >= maxAttempts) throw new Error("Still unprocessed: " + JSON.stringify(pending));
      await sleep(Math.min(2 ** attempt * 50, 5000) * (0.5 + Math.random()));
    }
  }
}

Requests look like { PutRequest: { Item: { ticket: "T-1042", device: "iPhone 16" } } }. The SDK for JavaScript v3 is included in the Lambda Node.js runtimes, so this works in a Lambda function without bundling it.

From the AWS CLI: don't ignore the output

The CLI prints the response, and an empty UnprocessedItems is the only sign that every write went through:

aws dynamodb batch-write-item --request-items file://batch.json

{
    "UnprocessedItems": {}
}

In a script, save the leftovers to a file and resend them. When nothing is left, the CLI writes nothing at all, so the file is empty, and the shell's -s test ("file exists and isn't empty") tells you whether a second pass is needed:

aws dynamodb batch-write-item --request-items file://batch.json \
  --query UnprocessedItems --output json > unprocessed.json
if [ -s unprocessed.json ]; then
  sleep 2
  aws dynamodb batch-write-item --request-items file://unprocessed.json
fi

For loads bigger than a few batches, use the Python or Node.js loop instead of chaining shell commands. To try any of this without touching a real table, our DynamoDB Local guide runs the same API on your computer.

Errors that reject the whole batch

Unlike throttling, these mistakes stop the entire call, and nothing in the batch is written. Each one is easy to fix once you know which it is:

An empty key value, such as a ticket ID of "", is also rejected: key attributes can't hold an empty string. And boto3 refuses an empty list for a table before it even sends the request, with a ParamValidationError. The 400 KB limit has its own guide: "Item size has exceeded the maximum allowed size", and key mismatches are covered in our schema mismatch guide.

"Provided list of item keys contains duplicates": where they come from

This is the most common batch error, and the duplicate is rarely obvious, because it only has to be in the same batch of 25:

• Repeated rows in the source. A CSV export with the same ticket twice, once from each update during the day.
• A put and a delete of the same item. Syncing scripts that "delete, then re-add" a changed record in one batch.
• Composite keys. On a table with a partition and a sort key, two items are duplicates only if both values match, so check both when you deduplicate.

The fix is to deduplicate before you batch, keeping the last version of each key. In Python, a dictionary keyed by the key values does it in one line: list({row["ticket"]: row for row in rows}.values()). With batch_writer, overwrite_by_pkeys does the same thing as items arrive.

Jake: "So a customer who came back twice in one day broke the whole night's import?"

Ethan: "Only their batch of 25, but yes. Now the script keeps the latest version of each ticket before it sends anything."

Capacity, cost and why throttling happens

Batching doesn't make writes cheaper: each put or delete in a batch uses the same write capacity it would use on its own, because DynamoDB runs them in parallel rather than as one combined write. What changes is speed. A batch of 25 can hit a table hard enough to be throttled where 25 single writes, spread out, would not.

On a table with provisioned capacity, a nightly load is the classic case: quiet all day, then a burst. Raise the write capacity for the load, turn on auto scaling, or switch the table to on-demand mode, which handles bursts without capacity planning. Our on-demand vs provisioned comparison shows what each costs.

Before a big load, this order avoids most surprises:

1. Check whether the table uses on-demand or provisioned capacity; the console shows it under the table's capacity settings.
2. On provisioned capacity, raise the write capacity or turn on auto scaling for the load window.
3. Deduplicate the rows by key.
4. Write with a loop that resends UnprocessedItems with backoff, like the ones above.
5. Make the loop fail loudly when items are still left after its last attempt, and alert on that error.

If only some keys are throttled while the table has capacity to spare, the writes are piling onto one partition; our hot partition guide covers that case, and the ProvisionedThroughputExceededException guide covers the rest.

Troubleshooting by symptom

Some items are missing, but there was no error

They came back in UnprocessedItems. Resend them in a loop with exponential backoff until it's empty.

Too many items requested for the BatchWriteItem call

The batch has more than 25 requests. Split it into batches of 25.

Provided list of item keys contains duplicates

A key appears twice in one batch. Deduplicate by key first, or use batch_writer with overwrite_by_pkeys.

ProvisionedThroughputExceededException or ThrottlingException on the whole batch

Nothing could be written because of throttling. Add capacity or use on-demand mode, and let the SDK retry.

One of the required keys was not given a value

An item lacks its partition or sort key, or the attribute name differs in case from the table's key.

batch_writer keeps running during heavy throttling

It resends unprocessed items without pausing. Add capacity, switch to on-demand, or use a loop with backoff.

DynamoDB BatchWriteItem: frequently asked questions

Ethan replaced the script's single batch_write_item call with the retry loop and added one line to deduplicate tickets by ID. That night's log showed eleven items held back on the first pass and none on the second. Tuesday's tickets went in from the old CSV the same morning. Jake's request was simple: "Make it tell me when something didn't go in." It does now: after eight tries, the script stops with an error.

Revision note. Written October 13, 2026, for DynamoDB BatchWriteItem with boto3 and the AWS SDK for JavaScript v3. "Trust, but verify," says the Russian proverb; a batch that says it's done deserves a look at what it left behind.