boto3 Pagination: How to Use get_paginator With Examples (S3, SSM, DynamoDB, IAM), PaginationConfig, search(), and a Loop for the 716 Calls With No Paginator

boto3 pagination means fetching a long list from AWS in pages, because most list and describe calls return only part of the result per call. The simple fix is a paginator: for page in client.get_paginator('list_objects_v2').paginate(Bucket='my-bucket'): ... keeps calling until there are no more pages, so you never handle tokens yourself. Two facts explain why so many scripts get this wrong. First, the page sizes are small and inconsistent: in our test, SSM's get_parameters_by_path returned 10 parameters out of 25 in one call, because 10 is its maximum page size. Second, AWS does not even agree on what the "next page" token is called: across the 435 services in boto3 1.43.118, we counted 37 different names (NextToken, nextToken, Marker, NextMarker, nextPageToken and 32 more), and 716 list-style operations that return a token but have no paginator at all, so you have to loop yourself. This guide shows paginators with examples, PaginationConfig, search(), resuming with StartingToken, a loop for operations with no paginator, and the bugs we reproduced.

Ethan's nightly script was supposed to copy every configuration value for Jake's two shops out of Parameter Store. The north shop's settings arrived; most of the south shop's did not. "It ran without an error," Jake said. "That's the problem," Ethan said. "It asked for everything, AWS sent the first ten, and my script believed that was all of it."

New to boto3? It is the AWS SDK for Python. You create a client for a service (boto3.client('s3')) and call its methods, each of which is one AWS API call; our session vs client vs resource guide explains the pieces. When an API call can return more items than fit in one response, AWS returns a page and a token for the next page. A paginator is a boto3 helper that makes the repeated calls for you.

Why one call is not enough

Every list-style API has a maximum page size, and many are small. A few that catch people, with their limits from the API models in boto3 1.43.118:

Look at the second column too: the setting that controls page size is called MaxResults in one service, limit in another, Limit, MaxItems, MaxKeys or MaxRecords in others. Across all paginated operations, we counted 1,422 that use MaxResults, 1,241 maxResults, 160 MaxRecords, 138 Limit, 90 MaxItems and 56 PageSize, and 177 with no page-size setting at all. A paginator hides all of this, which is the main reason to use one.

How to use a boto3 paginator, with an example

1. Create the client as usual: ssm = boto3.client('ssm', region_name='us-east-1').
2. Get a paginator by the method's name as a string: pg = ssm.get_paginator('get_parameters_by_path').
3. Call paginate() with the same arguments you would pass to the method, minus any token.
4. Loop over pages, and over the items in each page.

import boto3

ssm = boto3.client('ssm', region_name='us-east-1')
paginator = ssm.get_paginator('get_parameters_by_path')
names = []
for page in paginator.paginate(Path='/shop', Recursive=True, WithDecryption=True):
    for p in page['Parameters']:
        names.append(p['Name'])
print(len(names), 'parameters')

In our test (25 parameters under /shop, using moto's in-process mock of AWS), a plain get_parameters_by_path call returned 10 parameters plus a NextToken. The paginator returned all 25. Same arguments, no token handling. That is exactly Ethan's bug and its fix.

Each page is the normal response dictionary of that API, so the items are under the same key as in a single call (Parameters here). Use page.get('Contents', []) rather than page['Contents'] where a page can be empty: an empty S3 prefix, for example, returns a page with no Contents key at all.

PaginationConfig: MaxItems, PageSize and StartingToken

paginate() accepts one extra argument, PaginationConfig, with three settings that work the same way in every service:

• MaxItems: stop after this many items in total, across pages.
• PageSize: how many items to ask for per call. boto3 translates it to that service's own setting (MaxResults, limit, MaxKeys...), so you can skip learning the name.
• StartingToken: resume from where an earlier run stopped.

pg = ssm.get_paginator('get_parameters_by_path')
part = pg.paginate(Path='/shop', Recursive=True,
                   PaginationConfig={'MaxItems': 12, 'PageSize': 5}).build_full_result()
print(len(part['Parameters']), 'items; resume token:', 'NextToken' in part)

rest = pg.paginate(Path='/shop', Recursive=True,
                   PaginationConfig={'StartingToken': part['NextToken'], 'PageSize': 5}).build_full_result()
print(len(rest['Parameters']), 'more')

Our run printed 12 items with a resume token, then 13 more, which adds up to the 25. Two details matter. The token that comes back when you stop at MaxItems is boto3's own resume token, which can encode "stopped in the middle of a page"; pass it back only to the same paginator with the same arguments, not to the raw API call. And PageSize is a request, not a promise: services can return fewer items per page, and IAM explicitly may return fewer results even when more are available.

Filter and flatten with search() and build_full_result()

Two helpers save loops:

• build_full_result() collects every page into one dictionary, with the items under their usual key. Convenient for small results; for large ones it holds everything in memory, so prefer a loop.
• search(expression) applies a JMESPath expression to every page and yields the matching values one by one, across pages.

pg = ssm.get_paginator('get_parameters_by_path')
names = list(pg.paginate(Path='/shop', Recursive=True).search('Parameters[].Name'))

ec2 = boto3.client('ec2', region_name='us-east-1')
running = ec2.get_paginator('describe_instances').paginate(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
).search('Reservations[].Instances[].InstanceId')
for instance_id in running:
    print(instance_id)

s3 = boto3.client('s3')
big = s3.get_paginator('list_objects_v2').paginate(Bucket='shop-backups', Prefix='2026/').search(
    'Contents[?Size > `104857600`].Key')          # objects over 100 MB

Our search('Parameters[].Name') returned all 25 names. Filtering in JMESPath happens on your machine after each page arrives, so it does not reduce what AWS sends or bills; where an API has its own server-side filter (EC2's Filters, S3's Prefix, SSM's ParameterFilters), use that first.

"How many pages are there?" is a common question, and there is no shortcut: AWS does not tell you the total in advance. Count while you iterate, or use a service's own count or summary API if it has one.

37 names for "next page": why hand-written loops break

When you write the token loop yourself, you have to know what the service calls its token, in the response and in the request. We read the paginator definitions of every service in botocore 1.43.118:

The request side is just as varied (26 different input names), and some services send the token back under a different name than they accept it: S3's list_objects_v2 returns NextContinuationToken and expects ContinuationToken; DynamoDB's scan returns LastEvaluatedKey and expects ExclusiveStartKey. A loop that checks 'NextToken' in response on a service that uses nextToken stops after the first page, silently, because the key it looks for never appears. Python dictionary keys are case-sensitive, and so is the bug.

Operations with no paginator: a loop that is safe

Not every list call has a paginator. Across all services we found 716 List or Describe operations whose response includes a NextToken, Marker or similar, but which have no paginator in boto3 1.43.118. Some familiar ones:

• EventBridge: list_event_buses, list_connections, list_archives, list_api_destinations
• DynamoDB: list_exports, list_imports, list_contributor_insights
• Step Functions: list_state_machine_versions, list_state_machine_aliases
• CloudWatch: list_metric_streams; Route 53: list_hosted_zones_by_vpc; Secrets Manager: list_secret_version_ids

Check with client.can_paginate('list_event_buses'), which returned False in our test. Calling get_paginator on such an operation raises OperationNotPageableError. For these, use a small generic loop:

def manual_pages(call, items_key, token_in='NextToken', token_out='NextToken', **kw):
    """Generic loop for operations that have no paginator."""
    token = None
    while True:
        args = dict(kw, **({token_in: token} if token else {}))   # never send NextToken=None
        page = call(**args)
        yield from page.get(items_key, [])
        token = page.get(token_out)
        if not token:
            break

events = boto3.client('events', region_name='us-east-1')
buses = list(manual_pages(events.list_event_buses, 'EventBuses', Limit=10))

With 23 event buses created (plus the default bus), one list_event_buses(Limit=10) call returned 10 and a NextToken; the loop returned all 24. Pass token_in and token_out when the service uses other names, for example token_in='nextToken', token_out='nextToken'. The loop stops when the token is missing or empty, and sends no token on the first call.

The pagination bugs we reproduced

Only the first page: "It only returns 10 (or 50, or 100, or 1,000) items"

The script calls the method once. The size of the "missing" list is a clue: 10 is SSM's maximum, 50 is CloudWatch Logs and Lambda, 100 is IAM's default and DynamoDB's list_tables, 1,000 is S3. Use a paginator.

ParamValidationError: Invalid type for parameter NextToken, value: None

The first call sent NextToken=None. boto3 rejects it before sending: Invalid type for parameter NextToken, value: None, type: <class 'NoneType'>, valid types: <class 'str'>. Leave the argument out entirely until you have a real token, as the loop above does.

ValidationException: Member must have value less than or equal to 10

You asked for a bigger page than the service allows, for example MaxResults=50 on SSM. boto3 passed it through and the service rejected it. The SDK checks minimums before sending (it refused MaxResults=0 with "valid min value: 1") but leaves maximums to the service, so read the limit or let the paginator choose.

The loop stops early with no error

It checks the wrong token name (NextToken instead of nextToken or Marker), or it stops when a page is empty. Some APIs, such as CloudWatch Logs' filter_log_events and DynamoDB scans with a filter, can return empty pages that still carry a token. Stop on a missing token, not on an empty page.

An infinite loop

The loop sends the same token every time, usually because it reads the token from the first response instead of the latest one. Update the token from each new page.

S3: list_objects_v2 with a paginator, Prefix and Delimiter

s3 = boto3.client('s3')
pg = s3.get_paginator('list_objects_v2')
total, size = 0, 0
for page in pg.paginate(Bucket='shop-backups', Prefix='2026/10/'):
    for obj in page.get('Contents', []):
        total += 1; size += obj['Size']
print(total, 'objects,', round(size / 1e9, 2), 'GB')

# "folders": one level at a time
for page in pg.paginate(Bucket='shop-backups', Prefix='2026/', Delimiter='/'):
    for cp in page.get('CommonPrefixes', []):
        print(cp['Prefix'])

S3 returns up to 1,000 keys per page, and its paginator handles ContinuationToken for you. Pass Prefix to the paginator rather than filtering keys afterwards: it is applied by S3, so you fetch and pay for fewer list requests. Our S3 listing guide goes deeper on prefixes, versions and very large buckets, and the folder download guide uses this pattern to fetch a whole prefix.

DynamoDB: scan and query pagination

DynamoDB pages by size, not count: each scan or query page holds up to 1 MB of data, and the token is the LastEvaluatedKey of the last item read. The paginators handle that:

ddb = boto3.client('dynamodb', region_name='us-east-1')
items = []
for page in ddb.get_paginator('query').paginate(
        TableName='orders', KeyConditionExpression='shop = :s',
        ExpressionAttributeValues={':s': {'S': 'north'}}):
    items.extend(page['Items'])

The table resource (boto3.resource('dynamodb').Table('orders')) has no paginators, so with it you loop on LastEvaluatedKey yourself, passing it back as ExclusiveStartKey. A filter expression is applied after the page is read, so a page can be empty and still have a LastEvaluatedKey; keep going until the key is absent. Our DynamoDB pagination guide covers the resource loop and page tokens for APIs.

IAM: Marker and IsTruncated

IAM's list calls use Marker as the token in both directions and add IsTruncated, which is the field to trust. The MaxItems documentation in the API model says the page size defaults to 100, and that "IAM might return fewer results, even when there are more results available", with IsTruncated set to true. So a page of 37 users does not mean there are only 37. The paginator follows IsTruncated for you:

iam = boto3.client('iam')
users = [u['UserName'] for page in iam.get_paginator('list_users').paginate() for u in page['Users']]
roles = list(iam.get_paginator('list_roles').paginate().search('Roles[].RoleName'))

A hand-written IAM loop must keep going while IsTruncated is true and pass Marker, not NextToken.

Testing pagination: mocks can hide the bug

We ran these examples against moto, the popular library that mocks AWS in-process, because it lets you create hundreds of resources in seconds. It reproduced SSM's 10-item pages and EventBridge's tokens faithfully. But when we created 130 IAM users and called list_users once, moto returned all 130 users in one response, with no Marker and IsTruncated: False. Real IAM returns 100 by default and a Marker for the rest. A test suite that only runs against that mock would pass a script that loses every user after the hundredth.

Two habits catch this:

• Force small pages in tests: pass PaginationConfig={'PageSize': 2} (or the raw page-size argument) so every listing in your tests spans several pages, whatever the mock's default.
• Test the loop with Stubber: botocore's Stubber lets you queue exact responses, such as a first page with a token, an empty middle page with a token and a last page without one, and check that your code makes three calls.

Our boto3 sessions guide shows Stubber and moto setup for any client.

Making long listings faster

1. Filter on the server (Prefix, Filters, Path) before paginating; every page you skip is a call you do not make.
2. Use the largest page size the service allows; fewer pages means fewer round trips and less throttling.
3. Process pages as they arrive instead of building the full list first; memory stays flat and work starts sooner.
4. Handle throttling with the SDK's retry modes (Config(retries={'mode': 'adaptive'})) on very long listings, rather than your own sleeps.
5. For huge S3 buckets, an S3 Inventory report is cheaper and faster than listing millions of keys every night.

Ethan's Parameter Store copy now runs through the paginator and finishes in under a second for both shops. "All of them this time?" Jake asked. "Twenty-five out of twenty-five," Ethan said.

boto3 pagination: frequently asked questions

The next morning, Ethan ran the copy with a page size of 2 in the test suite, just to watch it make thirteen calls and still end with twenty-five parameters. "You're testing that it can count," Jake said. "I'm testing that it doesn't stop counting," Ethan said. If your results differ from what this page describes, please tell us. Guides like this stay right because readers write in.

Revision note. Written October 24, 2026, from a count of every paginator in botocore 1.43.118 and tests against moto 5.2.4. "Don't judge a book by its cover," the old saying goes, and a listing deserves the same patience: read past the first page.