DynamoDB Scan Filter Expression Not Working? Why Limit Returns Nothing, Reserved Words, Not Equal and Every Fix (boto3, Node.js)

When a DynamoDB FilterExpression seems not to work, the expression is usually fine. DynamoDB reads a page of items first, up to your Limit or 1 MB of data, and only then applies the filter to that page. So a scan with Limit=10 and a filter returns the matches among those first 10 items, which can be one item or none, plus a LastEvaluatedKey that means "there's more, ask again." The fix is to keep scanning until LastEvaluatedKey disappears, with a paginator or a short loop. Here is the surprise that catches nearly everyone: Limit is not "give me 10 matches." It's "read 10 items, then filter." In the example below, a table with 6 open repair tickets answered a Limit=10 scan for open tickets with exactly one, and nothing in the response looked wrong. The other usual suspects are a reserved word such as status or name (DynamoDB refuses it with "Attribute name is a reserved keyword"), a number stored as a string, which silently matches nothing, and contains, which is case-sensitive. And the filter doesn't save money: you pay for every item read, not every item returned.

Jake runs a phone repair shop. Every repair ticket lives in a DynamoDB table, and his friend Ethan, a developer, built him a small page that lists the open ones. It asked DynamoDB for ten tickets where the status is "open." On a busy Monday Jake had six open repairs and the page showed one. On Tuesday it showed none, and Jake spent the morning convinced the system was losing his jobs. When Ethan tried to fix it, DynamoDB complained that status was a reserved keyword. When Jake added a price filter for big jobs, it found nothing, because some prices had been saved as text. This page is what Ethan explained over the counter: how scans and filters really run, the Limit trap and the pagination fix in Python, Node.js and the CLI, reserved words, type mismatches, case, nested attributes, the Query versions of the same problems, why filters don't lower your bill, and what to build instead when you scan the same thing every day.

New to DynamoDB? Our plain-English guide to DynamoDB explains tables, keys and items in about ten minutes. You can follow this page without it; every step starts from zero.

How a DynamoDB scan with a filter really runs

Picture Jake's tickets as a long row of folders on a shelf. A scan is DynamoDB walking along that shelf. A filter is a sticky note that says "only hand me the ones marked open." The important part is the order of the steps:

1. DynamoDB reads a page of items from the table, in its own internal order. A page ends after Limit items, or after 1 MB of data, whichever comes first.
2. It applies the FilterExpression to the items in that page and drops the ones that don't match.
3. It returns whatever survived, along with two counts and, if the table isn't finished, a bookmark called LastEvaluatedKey.

Every scan response tells you what happened, if you look:

• ScannedCount is how many items DynamoDB read in this page.
• Count is how many of those matched the filter and came back.
• LastEvaluatedKey is present when there are more items to read. Pass it back as ExclusiveStartKey to read the next page. When it's missing, you've seen the whole table.

A filter never makes DynamoDB look further. It only throws away items from a page that's already been read. That's the root of nearly every "my filter isn't working" problem, and it's why the fix is almost always about pages, not about the expression.

Jake: "So it doesn't search the whole shelf for open tickets."

Ethan: "It grabs an armful of folders, picks out the open ones, and hands you those. If the armful had one open ticket, you get one. The rest of the shelf is still there, and it gave you a note saying where it stopped."

The Limit trap: why your scan returns fewer items than match

Here is the scan behind Jake's page, on a table of 103 repair tickets, 6 of them open:

import boto3

client = boto3.client("dynamodb")
response = client.scan(
    TableName="repairs",
    Limit=10,
    FilterExpression="#s = :open",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}},
)
print(response["Count"], response["ScannedCount"], "LastEvaluatedKey" in response)

1 10 True

One open ticket out of six. DynamoDB read 10 items, found 1 open among them, returned it, and handed back a LastEvaluatedKey to say there was more. Nothing about the response is an error, and that's what makes it so convincing. On a different day, the first 10 items might hold no open tickets at all, and the same scan returns zero.

Limit is the most misunderstood parameter in DynamoDB. It caps how many items are read per page, before filtering, and it has nothing to do with how many come back. The same is true of the 1 MB page size: even without a Limit, a large table is read 1 MB at a time, so a filtered scan of a big table returns its matches spread across many pages, and stopping after the first page loses the rest.

The fix in Python: a paginator

Boto3's paginator follows LastEvaluatedKey for you, page after page, until the table is finished:

paginator = client.get_paginator("scan")
open_tickets = []
for page in paginator.paginate(
    TableName="repairs",
    FilterExpression="#s = :open",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}},
):
    open_tickets.extend(page["Items"])

print(len(open_tickets))

6

All six. If you use the higher-level Table resource instead of the client, there's no paginator, so loop on LastEvaluatedKey yourself:

from boto3.dynamodb.conditions import Attr

table = boto3.resource("dynamodb").Table("repairs")
items = []
kwargs = {"FilterExpression": Attr("status").eq("open")}
while True:
    page = table.scan(**kwargs)
    items.extend(page["Items"])
    if "LastEvaluatedKey" not in page:
        break
    kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"]

When you really want "the first N matches"

A page that shows ten open tickets needs ten matches, not ten reads. Keep reading pages until you have enough or the table ends, then trim:

def first_matches(table, n, **scan_kwargs):
    """Keep scanning until n matching items are found or the table ends."""
    found = []
    while len(found) < n:
        page = table.scan(**scan_kwargs)
        found.extend(page["Items"])
        if "LastEvaluatedKey" not in page:
            break
        scan_kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"]
    return found[:n]


open_tickets = first_matches(table, 3, FilterExpression=Attr("status").eq("open"))

On Jake's table that returned three open tickets, t007, t027 and t047. Leave Limit out of the scan itself, or set it to a few hundred; a small Limit only means more round trips to collect the same matches.

The fix in Node.js: paginateScan

The AWS SDK for JavaScript v3 has a paginator too, used with for await:

import { DynamoDBClient, paginateScan } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({});

const matches = [];
for await (const page of paginateScan({ client, pageSize: 10 }, {
  TableName: "repairs",
  FilterExpression: "#s = :open",
  ExpressionAttributeNames: { "#s": "status" },
  ExpressionAttributeValues: { ":open": { S: "open" } },
})) {
  matches.push(...page.Items);
}
console.log(matches.length, "open tickets");

6 open tickets

The pageSize here is the same as Limit: items read per request. The paginator keeps going until there are no pages left, so the total is right whatever the page size.

In the AWS CLI, paging is automatic

The CLI follows LastEvaluatedKey on its own and adds up the counts, so the CLI and your code can disagree about the same filter:

aws dynamodb scan --table-name repairs \
  --filter-expression "#s = :open" \
  --expression-attribute-names '{"#s":"status"}' \
  --expression-attribute-values '{":open":{"S":"open"}}' \
  --page-size 10 --query "{Count: Count, ScannedCount: ScannedCount}"

{
    "Count": 6,
    "ScannedCount": 103
}

That's a common source of confusion: "the CLI finds six, my code finds one." The CLI read every page; your code read one.

Jake: "So the page wasn't losing my jobs. It just stopped after the first armful."

Ethan: "Exactly. And it left a note saying 'there's more,' which my code ignored. Now it reads the note."

"Attribute name is a reserved keyword": status, name, date and friends

Fixing Jake's filter the obvious way gets this:

client.scan(
    TableName="repairs",
    FilterExpression="status = :open",
    ExpressionAttributeValues={":open": {"S": "open"}},
)

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Scan operation: Invalid FilterExpression: Attribute name is a reserved keyword; reserved keyword: status

DynamoDB's expression language reserves hundreds of words, and the list isn't case-sensitive. Plenty of natural attribute names are on it: status, name, type, date, timestamp, value, ttl, comment, count, data, key, user, size, year, location, owner, group, role, url, source, result and state among them. Words like price, device and customer aren't.

The fix is an expression attribute name: a placeholder that starts with #, defined in ExpressionAttributeNames, which DynamoDB swaps for the real name after it parses the expression:

client.scan(
    TableName="repairs",
    FilterExpression="#s = :open",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}},
)

You can use a placeholder for every name, reserved or not; many teams do, so nobody has to check the list. And if you use boto3's Attr conditions with the Table resource, boto3 builds the placeholders for you, so Attr("status").eq("open") works without any ExpressionAttributeNames at all.

Nested attributes and names with dots

Dots in an expression mean "go inside a map." customer.name asks for the name field inside the customer map, and because name is reserved, it fails too:

An error occurred (ValidationException) when calling the Scan operation: Invalid FilterExpression: Attribute name is a reserved keyword; reserved keyword: name

For a nested path, use one placeholder per part, joined by the dot: #c.#n with {"#c": "customer", "#n": "name"}. But if an attribute's actual name contains a dot, as a top-level attribute literally called customer.name, you must use a single placeholder for the whole thing, #n with {"#n": "customer.name"}. Without it, DynamoDB reads the dot as a path and looks inside a map that doesn't exist.

Zero results and no error: type and case mismatches

The most frustrating filter failures don't raise anything. The scan runs, reads everything, and quietly matches nothing. Three causes account for almost all of them.

A number compared with a string

DynamoDB compares values only within the same type. A price stored as the number 59 doesn't equal the string "59", and the filter simply doesn't match:

client.scan(TableName="repairs", FilterExpression="price = :p",
            ExpressionAttributeValues={":p": {"S": "59"}})["Count"]   # 1
client.scan(TableName="repairs", FilterExpression="price = :p",
            ExpressionAttributeValues={":p": {"N": "59"}})["Count"]   # 15

Fifteen tickets cost $59, stored as numbers. One more had been saved by an older form as the text "59". The string filter finds only that odd one out; the number filter finds the fifteen and misses it. Mixed types in one attribute are common when data arrives from several apps or from CSV imports, and the fix is to make every writer store the same type, then repair the old items.

With the low-level client, the type is the letter in each value: {"N": "59"} is a number (sent as a string, which is normal), {"S": "59"} is text. With the Table resource, boto3 picks the type from your Python value, so pass a Decimal or an int for numbers, not a str.

Numbers stored as text sort like words

If prices are stored as strings, comparisons work alphabetically, character by character, not numerically. "100" comes before "60", because "1" comes before "6":

table.scan(FilterExpression=Attr("price_text").gt("60"))["Items"]   # []

Neither "100" nor "59" is "greater than" "60" as text, so nothing matched, although 100 is obviously bigger. Store numbers as numbers. If you're stuck with text for now, pad them to the same length, such as "0100" and "0059", so alphabetical order matches numeric order.

The same rule explains why begins_with never matches a number attribute: it's a string function, and a number isn't a string. On Jake's table, begins_with(price, :p) with "5" found only the one price saved as text.

contains is case-sensitive

contains compares characters exactly. Searching devices for "iPhone" and for "iphone" gave very different answers:

contains(device, "iPhone")   # 34 tickets
contains(device, "iphone")   # 1 ticket

DynamoDB has no lowercase function in expressions. The usual fix is to store a lowercase copy of searchable text when you write the item, such as device_lc, and filter on that. contains also works on lists and sets: contains(tags, "screen") finds every ticket whose tags list includes "screen".

Null is not the same as missing

DynamoDB has a real NULL type, and an attribute set to null still exists. That splits "items with no notes" into two different filters:

• attribute_not_exists(notes) matches items that have no notes attribute at all.
• attribute_type(notes, :t) with {":t": {"S": "NULL"}} matches items where notes is present and set to null.
• attribute_exists(notes) matches both the null one and any with real text.

With three test tickets, one with null notes, one with text and one with none, attribute_exists matched two and the null check matched one. Code that writes None for empty fields creates nulls; code that leaves the field out creates missing attributes. Pick one style for each attribute, or filter for both with OR.

Jake: "So 'iPhone' and 'iphone' are different words to it."

Ethan: "Completely different, like '59' the number and '59' typed on a label. DynamoDB is a very literal shop assistant. If you want it to ignore case, you have to write the label in lowercase too."

Not equal, IN, AND, OR: writing the conditions themselves

Once pagination and placeholders are right, the remaining questions are about syntax. DynamoDB's expression language looks like SQL but differs in small ways that cause syntax errors.

The != mistake comes up constantly, because every programming language uses it. DynamoDB answers with a syntax error that points straight at it:

An error occurred (ValidationException) when calling the Scan operation: Invalid FilterExpression: Syntax error; token: "!", near: "#s !="

One more detail about "not equal": items that don't have the attribute at all count as not equal. On Jake's table, #s <> :closed returned 8 items: the 6 open tickets plus 2 records that had no status at all. If you only want items that have the attribute, add it explicitly: attribute_exists(#s) AND #s <> :closed.

Here is a combined filter in boto3 with the client: open tickets that are either over $80 or for an iPhone. It matched 4 tickets on Jake's table:

paginator = client.get_paginator("scan")
matches = []
for page in paginator.paginate(
    TableName="repairs",
    FilterExpression="#s = :open AND (price > :p OR contains(device, :d))",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}, ":p": {"N": "80"}, ":d": {"S": "iPhone"}},
):
    matches.extend(page["Items"])
print(len(matches))

4

With the Table resource, the same condition reads more like Python: Attr("status").eq("open") & (Attr("price").gt(80) | Attr("device").contains("iPhone")). Use &, | and ~, not the words and, or and not, and keep the parentheses.

Scan vs Query, and sorting results

The cleanest fix for most filter problems is to stop scanning. Here's how the two reads compare:

Scans have no order you can rely on, so there's no "descending scan." If you need newest first, query a partition with a sort key and set ScanIndexForward=False:

client.query(
    TableName="repairs",
    KeyConditionExpression="shop_id = :s",
    ExpressionAttributeValues={":s": {"S": "shop-1"}},
    ScanIndexForward=False,
    Limit=3,
)

That returned shop 1's three highest ticket IDs, t999, t099 and t097, instead of the lowest three. Despite its name, ScanIndexForward belongs to Query only; passing it to a scan fails before the request is sent:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in input: "ScanIndexForward", must be one of: TableName, IndexName, AttributesToGet, Limit, Select, ScanFilter, ConditionalOperator, ExclusiveStartKey, ...

A five-step check when a filter returns nothing

When a filter that "should" match returns nothing, work through these in order. Each step rules out one cause, and most people stop at step 1 or 2:

1. Read every page. Run the same filter with a paginator and add up Count. If the total is right, your code was stopping after the first page.
2. Look at one real item. Fetch an item you expect to match with get_item and print it raw. Check the attribute's exact name, its type letter (S or N) and its exact value, including case and spaces.
3. Match the type. Make the filter value the same type as the stored value: {"N": "59"} for a number, {"S": "59"} for text.
4. Simplify the expression. Remove every condition but one. If that matches, add the others back one at a time until the count drops; the last one you added is the problem.
5. Check where you're reading. Print the table name, Region and endpoint. A filter on the wrong table succeeds and finds nothing.

Jake: "Step two found my price problem in about a minute."

Ethan: "It usually does. Looking at one real item beats staring at the expression. The item tells you what's actually stored, and the expression only tells you what you hoped was stored."

Query filters: the same trap, plus the key-attribute rule

A Query reads only the items with one partition key value, which makes it far cheaper than a scan, but filters work the same way: read a page, then filter. Limit caps the reads, not the matches:

{'Count': 1, 'ScannedCount': 5, 'LastEvaluatedKey': {...}}

That's a Query for shop 1 with Limit=5 and an "open" filter: five tickets read, one returned, more to come. The same paginator and loop fixes apply; use get_paginator("query") or paginateQuery.

Query adds one rule of its own. Key attributes can't go in the filter:

An error occurred (ValidationException) when calling the Query operation: Filter Expression can only contain non-primary key attributes: Primary key attribute: ticket_id

Conditions on the partition key and sort key belong in KeyConditionExpression, where DynamoDB uses them to decide what to read at all. For a sort key, that's =, <, <=, >, >=, BETWEEN or begins_with:

client.query(
    TableName="repairs",
    KeyConditionExpression="shop_id = :s AND begins_with(ticket_id, :t)",
    ExpressionAttributeValues={":s": {"S": "shop-1"}, ":t": {"S": "t00"}},
)

A scan has no such rule; it can filter on key attributes, because it reads everything anyway. Our guide to "The provided key element does not match the schema" covers the other key-related ValidationException.

Every filter ValidationException, decoded

Filter mistakes all come back as ValidationException, and the message after "Invalid FilterExpression" says exactly what's wrong:

The "unused values" rule surprises people who build expressions from optional search fields. If a user leaves the price box empty and your code drops the price condition, drop :price from the values map too, or DynamoDB rejects the whole request.

Filters don't save money: what a filtered scan really costs

Because the filter runs after the read, you pay for everything read. Here is the consumed capacity of a full scan of Jake's table, with and without the "open" filter:

no filter:     {'TableName': 'repairs', 'CapacityUnits': 1.5}
"open" filter: {'TableName': 'repairs', 'CapacityUnits': 1.5}

Identical. The filter shrank the response from 103 items to 6 and cost exactly the same, because DynamoDB charges for the items it reads, measured by their size, not the ones it hands back. You can see this on any request by adding ReturnConsumedCapacity="TOTAL".

On a small table that's fine. On a table with millions of items, a "find the open tickets" scan reads every single item every time, which gets slow and expensive and can use up a provisioned table's read capacity. Our guide to DynamoDB throughput explains what each read costs.

What to build instead: query an index

If you filter on the same attribute over and over, make it a key. A global secondary index with status as its partition key lets you query for open tickets, reading only the open ones:

client.query(
    TableName="repairs",
    IndexName="status-index",
    KeyConditionExpression="#s = :open",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}},
)

On Jake's table, that query returned all six open tickets and read only those six. An even cheaper pattern is a sparse index: write an attribute such as open_since only while a ticket is open and remove it when it closes. An index keyed on open_since then holds only open tickets, however large the table grows. Index reads are eventually consistent, though; ConsistentRead=True on a GSI is refused, as the error table shows.

Return less data with ProjectionExpression

If you only need a few attributes, ask for just those. A ProjectionExpression trims each returned item, which makes responses smaller and faster to send, though not cheaper to read:

client.scan(
    TableName="repairs",
    ProjectionExpression="ticket_id, #s, price",
    ExpressionAttributeNames={"#s": "status"},
    FilterExpression="#s = :open",
    ExpressionAttributeValues={":open": {"S": "open"}},
)

Each returned item then holds only ticket_id, status and price, and the consumed capacity is still the same 1.5 units, because the full items were read before anything was trimmed. Projections follow the same reserved-word rule as filters: ProjectionExpression="ticket_id, status" fails with Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved keyword: status, so reuse the #s placeholder.

Make a necessary full scan faster: parallel scan

Sometimes you really do need to read everything, for a migration, a report or a one-off cleanup. A parallel scan splits the table into segments and reads them at the same time. Each worker passes its own Segment number and the same TotalSegments, and paginates its segment to the end:

from concurrent.futures import ThreadPoolExecutor


def scan_segment(segment, total_segments):
    items = []
    paginator = client.get_paginator("scan")
    for page in paginator.paginate(
        TableName="repairs",
        Segment=segment,
        TotalSegments=total_segments,
        FilterExpression="#s = :open",
        ExpressionAttributeNames={"#s": "status"},
        ExpressionAttributeValues={":open": {"S": "open"}},
    ):
        items.extend(page["Items"])
    return items


segments = 4
with ThreadPoolExecutor(max_workers=segments) as pool:
    results = pool.map(lambda seg: scan_segment(seg, segments), range(segments))
open_tickets = [item for segment in results for item in segment]
print(len(open_tickets))

6

A parallel scan finishes sooner but reads exactly as much as a normal one, so it costs the same, and on a provisioned table it uses up read capacity faster. Keep it for jobs that genuinely need the whole table, and keep everyday lookups on queries.

Counting matches correctly

"How many open tickets are there?" has the same trap. Count on a single scan response counts the matches in that page only. To count everything, ask for counts only, so DynamoDB doesn't send the items back, and add up every page:

paginator = client.get_paginator("scan")
total = 0
for page in paginator.paginate(
    TableName="repairs",
    Select="COUNT",
    FilterExpression="#s = :open",
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={":open": {"S": "open"}},
):
    total += page["Count"]
print(total)

6

Select="COUNT" makes the responses smaller, but it still reads every item, so it costs the same as a full scan. For a number you need often, keep a counter item that you update whenever a ticket opens or closes, or query a sparse index with Select="COUNT".

"My new item isn't in the results": consistency and the wrong table

Sometimes the filter is right and the item genuinely isn't there yet, from the read's point of view.

• Eventual consistency. Scans and queries are eventually consistent by default, so an item written a moment ago may not show up for a short time. On the base table, ConsistentRead=True returns the latest data, at twice the read cost. Global secondary indexes can't do consistent reads at all.
• The wrong table, Region or account. A scan of an empty or different table filters nothing successfully. Print the table name and Region your code uses, and compare with where you wrote the item.
• DynamoDB Local versus the real service. Code pointed at a local endpoint, through an endpoint_url or an environment variable, reads the local copy, not your real table.

Test filters safely with DynamoDB Local

Filter experiments on a production table cost real reads and can stall a busy table. DynamoDB Local, a downloadable copy of DynamoDB that runs on your own computer, is a free place to try them. Every example on this page runs against it unchanged.

With Docker, one command starts it on port 8000:

docker run -p 8000:8000 amazon/dynamodb-local

Without Docker, download DynamoDB Local from AWS, unpack it, and run the JAR with Java, from the folder you unpacked it into. This works on Windows, macOS and Linux:

java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -inMemory -port 8000

Then point your code at it with endpoint_url. Any access key and secret work locally:

client = boto3.client("dynamodb", endpoint_url="http://localhost:8000", region_name="us-east-1",
                      aws_access_key_id="local", aws_secret_access_key="local")

The -inMemory option throws everything away when it stops, which is ideal for experiments. Two cautions. Remove the endpoint_url before deploying, because code still pointed at a local endpoint reads the local copy, not your real table. And treat Local as a very good copy, not the real thing: the filter behavior and error messages on this page match the real service, but some other error wordings and limits can differ.

Jake: "So I can break things on my laptop instead of on the shop's table."

Ethan: "That's the idea. Load a few hundred fake tickets, try every filter you like, and nothing you do costs a cent or slows down the real counter."

Still stuck? Troubleshooting by symptom

Fewer items than the console shows

Your code read one page. Check for LastEvaluatedKey in the response and keep scanning, or use a paginator. The console and the CLI page through automatically.

Zero items, no error

Compare types: is the value a number in the table and a string in your filter, or the other way round? Then check case for contains and begins_with, and print the exact values with repr() to spot trailing spaces.

Different results every time

A small Limit plus a filter returns whatever matched in the first page, and the first page's contents shift as items are written. Paginate to the end for a stable answer.

The scan is slow

A filtered scan still reads the whole table. Use a Query on a key or an index. For a large one-off job, a parallel scan with Segment and TotalSegments splits the reading across workers.

It works with the CLI but not in code

The CLI follows every page automatically; your code probably stops after the first. Add pagination.

Lambda times out while scanning

Scanning a large table page by page can take longer than the function's timeout. Query an index instead, or process pages across several invocations by saving LastEvaluatedKey between them.

DynamoDB scan and filter expressions: frequently asked questions

Jake's open-tickets page now reads every page until DynamoDB says there's nothing left, so the six Monday repairs showed as six. A week later Ethan added a small index on a field that only open tickets carry, and the page stopped scanning the whole table at all. The stray prices saved as text were fixed with a one-off script, and new tickets store prices as numbers only. Ethan taped the line below inside the cupboard where Jake keeps the spare screens.

Revision note. Written October 11, 2026, for every list that showed one result when there were six. "Don't judge a book by its cover," the old proverb says; with DynamoDB, don't judge a table by its first page.