Python Lambda Layer: Structure, How to Create One, Max Size and Limits, No module named _cffi_backend, Why a Layer Built for the Wrong Python Version Fails (pydantic, numpy), AWS-Provided Layers, Python 3.14 and Downloading a Layer

A Python Lambda layer is a zip file with one rule: your packages go inside a top-level folder named python, so the zip contains python/requests/, python/pydantic/ and so on. Build it with pip install requests -t python/, zip -r layer.zip python, publish it with aws lambda publish-layer-version, and attach the version ARN to your function. The catch that costs people an afternoon: a layer does not know which Python it was built for. "Compatible runtimes" is a label, not a check. Lambda will happily attach a layer built on Python 3.11 to a Python 3.13 function, the deploy succeeds, and the first invocation fails with Runtime.ImportModuleError: cannot import name 'ArgsKwargs' from 'pydantic_core._pydantic_core', because compiled packages such as pydantic and numpy ship a .so file named for one exact Python version. Build the layer with the same Python version and CPU architecture as the function, every time.

Jake's shop has one Lambda function that validates orders with pydantic before they go into the database. When Ethan moved it from Python 3.11 to 3.13, the console said "Successfully updated". The next order that came in got a 502. "It didn't say a word when I changed it," Ethan said, reading the log. "It just waited for a customer."

This guide is for anyone adding Python packages to AWS Lambda: the folder structure, the commands, the compiled-package trap, the Python version rules, and the error messages, each one produced in the official Lambda runtime images so you can match them against your own log. If Lambda itself is new to you, start with what AWS Lambda is.

What a Lambda layer is, in plain English

A Lambda layer is a zip file of files that Lambda unpacks into the /opt directory of your function's environment before your code starts. You use it to keep libraries out of your function's own zip, so the function stays small and several functions can share one set of dependencies. For Python, Lambda adds /opt/python and /opt/python/lib/python3.x/site-packages to the module search path, which is why the folder name matters so much.

The numbers to know: a function can use up to five layers, and the unzipped size of the function plus all its layers cannot exceed 250 MB. Layers are versioned and immutable: publishing a layer again creates version 2 with a new ARN, and your functions keep using version 1 until you change them. Layers work only with functions deployed as zip files; a function deployed as a container image bakes its dependencies into the image instead.

Jake: "So the layer is the toolbox, and the function is the job."

Ethan: "And the toolbox has to be packed for the exact van it's going in."

The Python Lambda layer structure

Inside the zip, your packages must sit under a top-level folder called python:

layer.zip
└── python/
    ├── requests/
    ├── urllib3/
    ├── certifi/
    ├── pydantic/
    ├── pydantic_core/
    └── (everything else pip installed)

Lambda unpacks that to /opt/python/requests/, and because /opt/python is on the path, import requests works. The longer form that a virtual environment produces, python/lib/python3.13/site-packages/requests/, also works, as long as the version number in the path matches the function's runtime. Two layouts do not work: packages at the zip root (layer.zip/requests/), and a folder with a different name such as site-packages/ or lib/. Both unpack to places Python never looks.

Here is the proof, from a function that returns its own search path, run in the official Python 3.13 runtime image through the Lambda Runtime Interface Emulator:

{
 "python": "3.13.16",
 "PYTHONPATH": "/var/runtime",
 "sys_path": [
  "/var/task",
  "/opt/python/lib/python3.13/site-packages",
  "/opt/python",
  "/var/lang/lib/python3.13/site-packages",
  "/var/runtime",
  ...
 ]
}

Read the order: your function's own code (/var/task) first, then the two layer paths, then the runtime's own packages in /var/runtime. The layer paths are hard-coded by the runtime, which is why only python and python/lib/python3.13/site-packages work, and why a layer's copy of a package beats the runtime's copy, a point that matters in the boto3 section.

Lambda layer tutorial: create a Python layer step by step

Everything below is for a function on Python 3.13 and x86_64. Change 3.13 to your runtime wherever it appears.

1. List the packages in a requirements.txt, pinned, so the layer is reproducible:

   requests==2.32.5
   pydantic==2.12.3

2. Install them into a python folder, for Linux and the function's exact Python, whatever machine you are on:

   mkdir python
   pip install -r requirements.txt -t python/ \
     --platform manylinux2014_x86_64 --implementation cp --python-version 3.13 --only-binary=:all: --upgrade

   On Windows PowerShell, replace the backslash line breaks with backticks or put the command on one line.
3. Zip the folder, keeping python as the top level: zip -r layer.zip python (PowerShell: Compress-Archive -Path python -DestinationPath layer.zip).
4. Publish the layer:

   aws lambda publish-layer-version --layer-name shop-deps \
     --zip-file fileb://layer.zip \
     --compatible-runtimes python3.13 python3.14 \
     --compatible-architectures x86_64

   The response contains the version ARN, ending in :1 the first time. For a zip over 50 MB, upload it to S3 first and use --content S3Bucket=my-bucket,S3Key=layer.zip instead of --zip-file.
5. Attach it to the function:

   aws lambda update-function-configuration --function-name shop-orders \
     --layers arn:aws:lambda:us-east-1:123456789012:layer:shop-deps:1

   --layers replaces the whole list, so include every layer the function should keep.
6. Test with a real invocation, not just a successful update: aws lambda invoke --function-name shop-orders out.json && cat out.json.

In the console, the same steps are: Lambda, Layers, Create layer, upload the zip (or point at S3), tick the compatible runtimes and architecture, then on the function's page scroll to Layers, Add a layer, Custom layers, pick the layer and version. The console does exactly what the CLI does and checks exactly as little.

Here is the publish and attach sequence as the CLI prints it, run against a local Lambda API mock to show the shape of the responses (the account number is the mock's):

$ aws lambda publish-layer-version --layer-name shop-deps --zip-file fileb://layer.zip \
    --compatible-runtimes python3.13 python3.14 --compatible-architectures x86_64 \
    --query "[LayerVersionArn,Version,CompatibleRuntimes]"
[
    "arn:aws:lambda:us-east-1:123456789012:layer:shop-deps:1",
    1,
    [
        "python3.13",
        "python3.14"
    ]
]
$ aws lambda update-function-configuration --function-name shop-fn \
    --layers arn:aws:lambda:us-east-1:123456789012:layer:shop-deps:1 --query "Layers[].Arn" --output text
arn:aws:lambda:us-east-1:123456789012:layer:shop-deps:1

Adding packages with compiled code: pydantic, numpy, pandas, psycopg2

Pure-Python packages such as requests run on any Python 3 and any CPU. Packages with compiled parts do not: pydantic (through pydantic_core), numpy, pandas, psycopg2, cryptography, Pillow and lxml all ship a shared library (.so file on Linux) built for one Python version and one CPU architecture, with the version in the file name, such as _pydantic_core.cpython-313-x86_64-linux-gnu.so. A plain pip install -t python/ on your laptop picks the wheel for your Python and your operating system: a Mac wheel, a Windows wheel, or a Linux wheel for Python 3.12 when the function runs 3.13. Any of those fails at import time in Lambda.

The pip options from the create section fix all three mistakes at once:

If --only-binary fails because a package has no wheel for that platform, build the layer inside the matching Lambda image instead, which gives you the right compiler and libraries:

docker run --rm -v "$PWD":/out --entrypoint bash public.ecr.aws/lambda/python:3.13 \
  -c "pip install -r /out/requirements.txt -t /out/python"
zip -r layer.zip python

That command works the same with podman. For arm64 functions add --platform linux/arm64 to the docker run. For psycopg2 specifically, the psycopg2-binary package has wheels and installs with the pip line above; plain psycopg2 needs PostgreSQL's client libraries and is the classic "no wheel" case.

The version mismatch: what a layer built for the wrong Python looks like

To show exactly what happens, this layer was built inside the Python 3.11 Lambda image with pip install pydantic numpy requests -t python/, then attached to functions running in the official Lambda runtime images and invoked through the Lambda Runtime Interface Emulator, the same code path a real invocation takes.

First, building it. On Python 3.11, pip install pydantic numpy requests --only-binary=:all: stopped at numpy, because numpy 2.4.6 publishes no Python 3.11 wheel any more; without --only-binary pip had tried to compile numpy from source and failed on the missing compiler instead. With numpy<2.4 the install succeeded, and the two compiled files in the layer carry the version in their names:

python/numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so
python/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-gnu.so

The same layer, invoked on the Python 3.11 runtime, works. Invoked on Python 3.13, the response for the pydantic function is:

{
 "errorMessage": "Unable to import module 'app_pyd': cannot import name 'ArgsKwargs' from 'pydantic_core._pydantic_core' (unknown location)",
 "errorType": "Runtime.ImportModuleError",
 "requestId": "",
 "stackTrace": []
}

And for the numpy function, a message that sends most people looking in the wrong place:

{
 "errorMessage": "Unable to import module 'app_np': Error importing numpy: you should not try to import numpy from\n        its source directory; please exit the numpy source tree, and relaunch\n        your python interpreter from there.",
 "errorType": "Runtime.ImportModuleError",
 "requestId": "",
 "stackTrace": []
}

Meanwhile requests, pure Python, from the very same layer on the very same 3.13 runtime:

{
 "requests": "2.34.2"
}

And a layer whose packages sit at the zip root, with no python/ folder, on any runtime:

{
 "errorMessage": "Unable to import module 'app_req': No module named 'requests'",
 "errorType": "Runtime.ImportModuleError",
 "requestId": "",
 "stackTrace": []
}

Three things to notice. The deploy step never complains: the layer carried python3.11 in its compatible runtimes, and the function was on 3.13, and nothing stopped the attach. Pure-Python requests from the same layer imports fine on 3.13, so a layer can be half-working, which is more confusing than a layer that fails completely. And neither error says "wrong Python version". pydantic reports a missing name inside pydantic_core._pydantic_core, because Python 3.13 cannot load the cpython-311 library and falls back to an empty placeholder; numpy blames its "source directory" for the same reason. Whenever an import error points at a leading-underscore module (_pydantic_core, _multiarray_umath) or talks about a source tree in a function that has no source tree, suspect the Python version or the architecture first.

The fix is always a rebuild for the right version, never a patch: rerun the pip line with the function's --python-version, publish a new layer version, and point the function at it. If you upgrade the function's runtime, the layer needs rebuilding in the same change.

Ethan: "The console checked the zip was a zip. That's all it checked."

Jake: "So the test is the order. Run a fake one before a real one."

No module named '_cffi_backend' in Lambda: the cause and the fix

This is one of the most searched Lambda layer errors, and it has nothing to do with your code. _cffi_backend is the compiled part of cffi, a package that many others depend on without you asking for it: cryptography requires cffi>=2.0.0, and so does PyNaCl, which means anything that signs, encrypts or connects over SSH can pull it in. The pure-Python half of cffi imports fine; the compiled half does not, because it was built for a different machine or a different Python.

Here are the two ways it happens, reproduced in the official Lambda images through the runtime emulator. First, a layer built on a Mac, which gets a macOS binary:

$ ls layer-built-on-mac/python | grep cffi
cffi
cffi-2.1.1.dist-info
_cffi_backend.cpython-313-darwin.so

# invoked on the python3.13 runtime
{"errorMessage": "No module named '_cffi_backend'", "errorType": "ModuleNotFoundError",
 "stackTrace": ["  File \"/var/task/app.py\", line 4, in handler\n    ffi = cffi.FFI()\n",
                "  File \"/opt/python/cffi/api.py\", line 41, in __init__\n    import _cffi_backend as backend\n"]}

Second, a Linux build for the wrong Python version, here built for 3.13 and run on the python3.11 runtime. The file is a Linux binary this time, but its name says cpython-313, so Python 3.11 does not even look at it, and the error is identical:

$ ls layer-built-for-313/python | grep cffi
_cffi_backend.cpython-313-x86_64-linux-gnu.so

# invoked on the python3.11 runtime
{"errorMessage": "No module named '_cffi_backend'", "errorType": "ModuleNotFoundError", ...}

Notice that import cffi itself succeeded in both cases; the failure came when the code first used it (cffi.FFI()), because cffi loads its compiled backend lazily. With cryptography the failure usually shows up at import time instead, as a Runtime.ImportModuleError during init. Either way, the cause is the same.

The fix is to ask pip for the Linux build that matches your function's runtime and architecture, whatever machine you are on:

pip install cffi cryptography \
  --platform manylinux2014_x86_64 \
  --implementation cp \
  --python-version 3.11 \
  --only-binary=:all: \
  -t layer/python

With that command, the same layer contains _cffi_backend.cpython-311-x86_64-linux-gnu.so and the function returns {"cffi": "2.1.1"}. The same command fetched cryptography 50.0.2, and a Fernet encrypt-and-decrypt round trip worked on the python3.11 runtime. Change --python-version to your runtime (3.12, 3.13, 3.14), and use manylinux2014_aarch64 for arm64 functions. If you would rather not think about platforms at all, build the layer inside the official image for your runtime, for example public.ecr.aws/lambda/python:3.13, so pip sees exactly the system Lambda runs.

Jake hit this one with a payment library on his MacBook. "It works on my laptop," he said, which Ethan pointed out was exactly the problem: the layer was built for his laptop.

Which Python version to build for: Lambda's runtimes in October 2026

Two rules follow. A layer is built for exactly one Python version, so a layer for a 3.13 function and a 3.14 function means two layers, or one layer with two python/lib/python3.x/site-packages folders, which doubles its size. And the Amazon Linux column matters for compiled packages: 3.11 and older run on Amazon Linux 2 with an older glibc, 3.12 and newer on Amazon Linux 2023, so a wheel built from source on one does not always run on the other. The manylinux2014 wheels from PyPI run on both. Python 3.10 is deprecated on October 31, 2026, so functions on it should move to 3.12 or newer, with their layers rebuilt; "runtime is no longer supported" explains what the deprecation dates block and when.

Layer versions: ARNs, listing, updating and deleting

Every publish-layer-version call creates a new, immutable version with the next number, and functions reference a specific version by ARN. That is a feature: a function on shop-deps:1 keeps working exactly as tested until you move it to :2, even if someone deletes version 1 later. The commands:

# every version of a layer, newest first
aws lambda list-layer-versions --layer-name shop-deps \
  --query "LayerVersions[].[Version,CompatibleRuntimes[0]]" --output text

# only the versions marked compatible with one runtime
aws lambda list-layer-versions --layer-name shop-deps --compatible-runtime python3.13

# which layer versions a function uses right now
aws lambda get-function-configuration --function-name shop-orders --query "Layers[].Arn"

# move the function to version 2 (the list is replaced, so name every layer)
aws lambda update-function-configuration --function-name shop-orders \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:shop-deps:2

# delete an old version nobody uses
aws lambda delete-layer-version --layer-name shop-deps --version-number 1

There is no "latest" alias: a function always pins a number, which is why CloudFormation, SAM and Terraform templates that build layers output the new ARN and feed it to the function in the same deployment. SAM's AWS::Serverless::LayerVersion with BuildMethod: python3.13 in its metadata also runs the pip install for the right version for you, inside a matching image when you pass --use-container to sam build.

Lambda layer max size and limits

Every limit that matters for layers, in one place:

To get a feel for the 250 MB, here is a real layer with three popular packages for Python 3.11, pydantic, numpy and requests: 86.9 MB unzipped and 26.8 MB zipped. Of that, numpy alone is about 46 MB plus 27 MB of bundled libraries, test folders are 15 MB, and pip's precompiled __pycache__ files are 18.7 MB. Two layers like that and a modest function already sit near the ceiling, and adding pandas usually pushes past it.

When a layer exceeds the maximum size, work through these in order:

1. Drop what the runtime already has. The Python runtimes include boto3; leave it out unless you need a newer version than the runtime's.
2. Delete test folders (tests directories inside packages), which code never imports; that was 15 MB in the example. Keep __pycache__ unless you are desperate: those files save compile time at every cold start.
3. Use an AWS-provided layer for heavy, common stacks: AWS SDK for pandas bundles pandas, NumPy and PyArrow, built and tested for Lambda (next section).
4. Split by function. If only one function needs the heavy package, give it its own layer instead of putting everything in one shared layer.
5. Switch to a container image when the dependencies are simply big (machine learning libraries, browsers): up to 10 GB, no layers needed.

AWS-provided Lambda layers you can use instead of building your own

For two common stacks you do not need to build a layer at all. AWS publishes managed, public layers that you add by ARN (in the console: Layers, Add a layer, Specify an ARN). The current versions in US East (N. Virginia), October 2026:

The ARNs follow a pattern, so the other versions are easy to work out. Powertools uses python310 to python314 and x86_64 or arm64 in the name (Python 3.12 on x86_64 is ...AWSLambdaPowertoolsPythonV3-python312-x86_64:40). AWS SDK for pandas uses Python312, Python313, Python314, with -Arm64 added for arm64; in us-east-1 those are versions 31, 16 and 11. The version number at the end changes with each release and differs by Region, so copy the current ARN for your Region from each project's documentation rather than reusing an old one.

Two things to know before you rely on them. They count toward your 5 layers and your 250 MB like any other layer, and AWS SDK for pandas is large, so it leaves less room for your own packages. And because the ARN pins a version, your function does not change when a new version is published; updating is your decision.

Lambda layer pricing: what layers cost

Layers have no price of their own: the Lambda price list has no charge for creating, storing or attaching layers. You pay for your function's requests and duration as usual. Layers do affect the bill in two indirect ways:

• Initialization time is billed. Duration charges cover the code outside your handler too, including the imports that load packages from your layers. A layer that takes a second to import costs a second at every cold start.
• Extensions are billed for their time. If a layer contains an extension, you pay for the execution time it uses, in 1 ms increments, including the time the last extension takes to finish at shutdown. Installing your own extensions costs nothing; partner extensions may have their own pricing.

Layer versions also use the Region's 300 GB code storage quota, which is free but finite. A deployment pipeline that publishes a new layer version on every commit can fill it over a year or two, so delete old versions you no longer attach.

Do Lambda layers reduce cold start?

No, and they do not make it much worse either, for Python. A layer is not a cache: at the start of each new execution environment, Lambda downloads your code and all its layers and extracts them, then your code imports what it needs. The same files load either way; only their location differs (/opt for layers, /var/task for the function).

Here is the same function, importing numpy 2.2.6 and pydantic, run as a fresh cold start several times through the Lambda runtime emulator, once with the packages in a layer and once bundled in the function:

Once the files were in the disk cache, the two were the same within noise; the first-run gap came from the files being read from disk for the first time, not from layers. The emulator does not model Lambda's download step, but the lesson holds: what slows a cold start is how much you import, not where it lives. For Go and Rust, putting dependencies in a layer can increase cold start time, because the runtime must load them separately during initialization; for those languages, bundle dependencies in the deployment package.

To actually reduce Python cold starts: import heavy packages only where needed, trim unused dependencies, and consider SnapStart or provisioned concurrency for functions where latency matters.

Lambda layer vs extension vs container image

So "layer vs extension" is not really a choice: an extension is a special kind of layer. If the zip contains an extensions/ folder, Lambda finds each executable there during init and starts it in parallel with your runtime, and an external extension can keep running after your handler returns, for example to flush logs or metrics. If the zip contains python/ with packages, it is a plain layer your code imports.

How to download a Lambda layer

Sometimes you inherit a function whose layer source is long gone and need to see what is inside. get-layer-version returns a link to the layer zip in S3 that is valid for 10 minutes:

URL=$(aws lambda get-layer-version --layer-name shop-deps --version-number 1 \
  --query Content.Location --output text)
curl -s -o shop-deps-1.zip "$URL"
unzip -l shop-deps-1.zip | head

The listing shows the folder structure and, for compiled packages, the cpython-3xx in the .so names, which tells you which Python it was built for without guessing. For a layer from another account, such as a public layer, use get-layer-version-by-arn with the full ARN. The console shows the same information on the layer version's page.

When your layer breaks boto3: the runtime's own packages

The Python runtime comes with boto3, botocore and their dependencies, including urllib3, preinstalled under /var/runtime. If your layer also contains urllib3 (it will, if you installed requests), Python now has two copies, and which one wins depends on the search order. A layer with a newer urllib3 than the runtime's botocore expects produced the most-viewed Lambda layer question of all, ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_', where botocore from the runtime imported urllib3 2.x from the layer. The usual fixes, in order:

1. Do not ship what the runtime already has. Install with pip install requests -t python/ --no-deps only when you know the runtime provides the dependencies, or simply leave boto3 and botocore out of requirements.txt.
2. If you need a newer boto3 than the runtime's, put the whole set (boto3, botocore, s3transfer, urllib3) in the layer together so the versions match each other.
3. Check the search order from inside a function with import sys; print(sys.path), and import urllib3; print(urllib3.__file__) to see which copy is loaded.

Layers also merge in the order you added them, and a later layer's file overwrites an earlier layer's file at the same path, so two layers that both contain urllib3/ give you whichever came last. The console lets you reorder them; the CLI's --layers list is the order.

Lambda layer errors for Python and what each one means

Runtime.ImportModuleError: Unable to import module 'app': No module named 'requests'
The package is not where Python looks. Usually the zip has no python/ top folder, or the folder is misspelled, or the layer was never attached (check get-function-configuration --query Layers).

cannot import name 'ArgsKwargs' from 'pydantic_core._pydantic_core' (unknown location), Error importing numpy: you should not try to import numpy from its source directory, or No module named 'something._something' inside a package that is clearly present
The compiled part was built for a different Python version or CPU architecture. Rebuild with --python-version and --platform for the function.

pip: no matching distribution, or a long compiler error during pip install -t python/
The package has no wheel for the Python version you asked for (numpy 2.4 dropped Python 3.11, for example). Pin an older release that still ships one, or move the function to a newer runtime; never let pip compile from source on your laptop for Lambda.

ImportError: ... cannot import name ... from 'urllib3...', cannot import name 'DEFAULT_CIPHERS'
Your layer's urllib3 (or another dependency) is shadowing the runtime's. See the section above.

Unable to import module 'app': /lib64/libc.so.6: version 'GLIBC_2.28' not found
A wheel built for Amazon Linux 2023 (or a modern distribution) on a function running Python 3.11 or older, which is on Amazon Linux 2. Use the manylinux2014 wheels, or move the function to Python 3.12+.

InvalidParameterValueException: Unzipped size must be smaller than 262144000 bytes
Function plus layers exceed 250 MB unpacked. Remove packages you do not import (pandas pulls in a lot), strip tests and .dist-info you do not need, or move to a container image; the unzipped-size post shows what to cut.

Layers are not supported for functions defined as container images
Exactly that. Install the packages in the Dockerfile instead.

Cannot find module in a Node.js function
The Node.js rule is a nodejs/node_modules/ folder, not python/; the Node.js layer post covers it.

ModuleNotFoundError: No module named '_lzma'
Not a layer problem on the managed runtimes: lzma, _lzma, bz2 and sqlite3 all import on the python3.9, 3.11 and 3.13 images. If you see it, the Python that raised it was built by hand, usually a custom runtime on provided.al2023 or a pyenv Python on your build machine, compiled without the liblzma headers. Install xz-devel (Amazon Linux, Fedora) or liblzma-dev (Debian, Ubuntu) and rebuild that Python, or switch to the managed Python runtime.

Lambda layer best practices: the Python checklist

Pin this next to your deploy script. Every layer problem on this page is caught by one of these seven lines, and most of them take under a minute to check before you publish rather than an afternoon after you deploy.

1. Top-level folder named python inside the zip.
2. Install with --platform manylinux2014_x86_64 (or _aarch64), --python-version equal to the runtime, --only-binary=:all:.
3. Leave boto3 and botocore out unless you ship the whole matching set.
4. Publish, attach the version ARN, and invoke once before you call it done.
5. When you change the runtime version, rebuild the layer in the same change.
6. Keep function plus layers under 250 MB unzipped, five layers at most.
7. Inherited a layer? get-layer-version, download, and read the .so names.

Frequently asked questions

Ethan rebuilt Jake's layer in the 3.13 image, published version 4, attached it and sent a fake order through before touching anything else. "Took four minutes," he said. "The first time took four hours." Jake asked what the difference was. "The first time I trusted the green tick."

Revision note. Written October 23, 2026, with the Lambda Python runtime table of that date, and every error reproduced in the public.ecr.aws/lambda/python 3.11 and 3.13 images through the Runtime Interface Emulator. "Measure twice, cut once," the carpenters say; with layers it is build once, for the right Python.