ECS CannotPullContainerError: "Pull Access Denied", "Failed to Resolve Ref ... Not Found", "Request Canceled While Waiting for Connection", "toomanyrequests" and "No Space Left on Device", Every Cause Fixed on Fargate and EC2
CannotPullContainerError means your ECS task stopped before any of your code ran, because the agent could not download the container image named in the task definition. The text after the colon tells you which of six things went wrong: no network path to the registry ("request canceled while waiting for connection", "i/o timeout", "context canceled"), no permission ("pull access denied", "not authorized to perform: ecr:BatchGetImage"), no such image ("failed to resolve ref ... not found", "invalid reference format"), a rate limit ("toomanyrequests: You have reached your pull rate limit"), no disk space ("no space left on device"), or a private registry without credentials. Here is the case that fools experienced people: the tag can exist in ECR, and ECS still reports "not found", because when a service deploys, ECS resolves your tag to an image digest once and keeps using that digest for every task it starts later. Re-push :latest, let a lifecycle policy clean up the old untagged image, and the next scale-out asks for a digest that no longer exists. This page shows you how to read the full stopped reason, then walks every branch with the command that proves it and the fix, on Fargate and on EC2.
Jake's booking site runs as a small Fargate service that Ethan set up in the spring. At 9:05 on a busy Saturday the service scaled out, and every new task went from PENDING to STOPPED in under a minute. Jake had the ECR console open on his phone. "The image is right there. It says latest, pushed Thursday." Ethan looked at the stopped reason and said, "Your image is there. The one your service asked for is not. Those are two different images with the same name." The rest of this page is how he knew, and what he changed so it would not happen twice.
Read the whole message first: six families of CannotPullContainerError
Every one of these errors starts the same way, which is why people paste the first ten words into a search box and get the wrong fix. The useful part is the end of the message. ECS rewrote many of these messages in June 2024, so you may see an older raw Docker daemon string or the newer form that begins with a plain-English hint such as "The task can't pull the image. Check your network configuration." Both carry the same underlying error after the hint, and the stopCode stayed the same when the wording changed. On Fargate platform version 1.4, long messages are truncated, so the containers' own reason field is worth reading too.
Match the tail of your message to a row below, then jump to that section. If two rows seem to fit, take the network row first: a missing route produces timeouts that look like everything else.
| The end of your message says | What actually failed | Where to go |
|---|---|---|
| request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) | No route from the task to the registry | Network |
| dial tcp ... i/o timeout, or "There is a connection issue between the task and the registry" | Route or security group blocks HTTPS to the registry or to S3 | Network |
| context canceled | Usually no route to ECR from the task's VPC | Network |
| pull command failed: panic: runtime error: invalid memory address or nil pointer dereference | The layer download from S3 is blocked (security group or a gateway endpoint with no route) | Network |
| pull access denied for REPO, repository does not exist or may require 'docker login' | The role cannot pull, the repository name is wrong, or a private non-AWS registry needs credentials | Permissions, Private registries |
| is not authorized to perform: ecr:BatchGetImage | Execution role or instance role is missing ECR actions, or a repository policy denies it | Permissions |
| failed to resolve ref "...:TAG": ... not found | The tag or digest does not exist where the task definition points | Image not found, Digest trap |
| API error (400): invalid reference format | The image string itself is malformed | Image not found |
| toomanyrequests: You have reached your pull rate limit | Docker Hub rate limit | Docker Hub |
| no space left on device / failed to extract layer / Thin Pool has N free data blocks | Disk full: Fargate ephemeral storage or the EC2 host volume | Disk space |
| httpReaderSeeker: failed open: unexpected status code | The copy of a layer failed: often network, sometimes a rate limit or a 403 from an S3 endpoint policy | Network, Docker Hub |
Ethan's message on Saturday ended in failed to resolve ref "ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/booking:latest@sha256:...": not found. Jake: "Why is there a long hash glued to the end of latest?" Ethan: "That hash is the whole story. Hold that thought for three sections."
Find the full stopped reason in the console and the CLI
The task list hides stopped tasks by default, and stopped tasks only stay visible for about an hour, so look soon after the failure. If the task belongs to a service, the service's Events tab also shows the pattern ("has started 1 tasks" followed by stops), but the event text is shorter than the task's own reason.
To read it in the console:
- Open the ECS console, choose Clusters, and open your cluster.
- Choose the Tasks tab and set Filter desired status to Stopped.
- Open the most recent stopped task. In the row, choose Stopped in the Last status column to see the stopped reason in a pop-up.
- Inside the task, open the Containers section and read each container's status reason as well. With a sidecar, the failing image may be the sidecar's, not your app's.
- Copy the whole message into a text file before the task drops off the list.
The same thing from a terminal, which also gives you text you can search:
aws ecs list-tasks --cluster CLUSTER --desired-status STOPPED
aws ecs describe-tasks --cluster CLUSTER --tasks TASK_ARN \
--query "tasks[].[stopCode,stoppedReason,containers[].[name,image,reason]]"
A pull failure shows stopCode as TaskFailedToStart, and the image field shows the exact string ECS tried to pull. Compare that string character by character with what you think you deployed; you will catch a surprising number of Region and account mix-ups right there.
If the reason is ambiguous, the AWS Systems Manager runbook AWSSupport-TroubleshootECSTaskFailedToStart checks the network path, roles and endpoints for you. Run it in the same Region as the cluster, give it the most recent failed task ID, and start it while that task is still visible to DescribeTasks.
"Request canceled while waiting for connection", "i/o timeout" and "context canceled": no route to the registry
A Fargate task, and an EC2 instance pulling for a task, needs an HTTPS path to the registry. For ECR that means three destinations: the ECR API (for the login token), the Docker registry endpoint (for the image manifest), and Amazon S3 (where ECR keeps the layers). For Docker Hub, GitHub Container Registry or any other public registry, it means the internet. Without a path, the pull waits, times out, and the task stops with one of the timeout messages.
Which path you need depends on the subnet the task lands in:
- Public subnet (its route table sends
0.0.0.0/0to an internet gateway): the task needs a public IP. Set Auto-assign public IP to ENABLED in the service or run-task networking settings. A task in a public subnet without a public IP has no way out, and this is the single most common cause of the error on a first Fargate deployment. - Private subnet with a NAT gateway (route
0.0.0.0/0to a NAT gateway): leave public IP DISABLED. The NAT gateway itself must sit in a public subnet. - Private subnet with no internet route: create VPC endpoints so traffic to ECR and S3 stays inside AWS. This works only for images in ECR, not for Docker Hub.
Check the route table of the exact subnets your service uses:
aws ec2 describe-route-tables \
--filters Name=association.subnet-id,Values=subnet-0abc123 \
--query "RouteTables[].Routes[].[DestinationCidrBlock,GatewayId,NatGatewayId,State]"
A 0.0.0.0/0 row pointing at an igw- means public; a row pointing at a nat- means private with NAT; no 0.0.0.0/0 row means you need endpoints. A route with State blackhole points at a NAT gateway that was deleted, which fails exactly like no route at all.
For a private subnet with no internet route, the set of endpoints depends on your Fargate platform version and logging:
| Endpoint | Type | Needed when | Detail that breaks it |
|---|---|---|---|
com.amazonaws.REGION.ecr.dkr | Interface | Always, for ECR images | Private DNS must be enabled |
com.amazonaws.REGION.ecr.api | Interface | Fargate platform 1.4.0 or later | Security group must allow 443 from the tasks |
com.amazonaws.REGION.s3 | Gateway | Always, for ECR images (layers live in S3) | Must be attached to the subnets' route tables; its policy must allow arn:aws:s3:::prod-REGION-starport-layer-bucket/* |
com.amazonaws.REGION.logs | Interface | The task uses the awslogs log driver | Missing logs endpoint fails startup with a different error |
com.amazonaws.REGION.secretsmanager / ssm | Interface | The task injects secrets or uses private registry credentials | Covered in the ResourceInitializationError guide |
List what the VPC already has:
aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-0abc123 \
--query "VpcEndpoints[].[ServiceName,VpcEndpointType,PrivateDnsEnabled,State]"
Two quieter causes belong here too. First, the VPC's DHCP options set should use AmazonProvidedDNS; if it points at your own DNS servers, those servers must forward AWS names (conditional forwarding), or the endpoint's private DNS name never resolves. Second, the endpoint's security group must accept HTTPS (443) from the task's security group or subnet, and the task's security group must allow outbound 443. The "panic: runtime error: invalid memory address" message is usually this second problem on the S3 side: the manifest arrived, the layer download could not.
The full endpoint walkthrough, with the console clicks for each one, lives in our ResourceInitializationError guide, because the same plumbing serves registry auth, secrets and logs. If you are weighing a NAT gateway against endpoints on cost, the numbers are in why the NAT gateway line on your bill is so large.
Jake: "So the network one is the boring one?" Ethan: "The boring one is the common one. Check it first, because a missing route makes the permission and image errors impossible to see."
"Pull access denied" and "not authorized to perform: ecr:BatchGetImage": the role, the repository, or the name
On Fargate, the identity that pulls your image is the task execution role, the one in executionRoleArn. It is not the task role, which your application uses after it starts. On EC2, the pull uses the container instance role attached to the instance. Mixing these up is the classic version of this error: someone adds ECR permissions to the task role, the error stays, and everyone concludes IAM is haunted.
The newer message names the role and the missing action directly:
CannotPullContainerError: The task can't pull the image. Check that the role has the
permissions to pull images from the registry. Error response from daemon: pull access
denied for REPOSITORY, repository does not exist or may require 'docker login': denied:
User: ROLE_ARN is not authorized to perform: ecr:BatchGetImage on resource: IMAGE because
no identity-based policy allows the ecr:BatchGetImage action.
To pull from a private ECR repository, the role needs three actions: ecr:GetAuthorizationToken, ecr:BatchGetImage and ecr:GetDownloadUrlForLayer. The managed policy AmazonECSTaskExecutionRolePolicy covers Fargate (it also includes ecr:BatchCheckLayerAvailability and the CloudWatch Logs actions). On EC2, the instance role normally carries AmazonEC2ContainerServiceforEC2Role; AmazonEC2ContainerRegistryReadOnly is the minimum for pulling. ecr:GetAuthorizationToken has to come from an identity-based policy on the role; a repository policy cannot grant it.
Check what the execution role really has:
aws ecs describe-task-definition --task-definition FAMILY \
--query "taskDefinition.[executionRoleArn,containerDefinitions[].image]"
aws iam list-attached-role-policies --role-name ecsTaskExecutionRole
If executionRoleArn is empty on a Fargate task definition, that is the whole problem: there is no identity to pull with. Create or pick the role, add it to the task definition, register a new revision and update the service.
When the role looks right and the pull is still denied, look at the repository side. By default, only the account that owns an ECR repository can pull from it. A task in a second account needs a repository policy in the owning account that names the other account or role, plus the ECR actions on its own execution role. Within one account, either an identity policy or a repository policy can allow the pull, but an explicit deny in either one blocks it, so check for a Deny statement:
aws ecr get-repository-policy --repository-name booking --region us-east-1
Finally, "repository does not exist or may require 'docker login'" is Docker's catch-all for "I could not get this image." It also appears when the repository name is simply wrong, when the repository lives in another Region, and when the image is in a private registry outside AWS that needs credentials. The first two are in the next section; the third has its own section below.
Ethan, to Jake: "The execution role is the delivery driver. The task role is you, after the package arrives. The driver needs the key to the warehouse, not you."
"Failed to resolve ref ... not found" and "invalid reference format": the image is not where the task definition points
An image reference has up to four parts, and every one of them can be wrong: the registry host (ACCOUNT.dkr.ecr.REGION.amazonaws.com), the repository path (booking or team/booking), and either a tag (:v42) or a digest (@sha256:...). With no tag at all, the agent pulls :latest. Docker Hub official images use a single name (nginx), other Docker Hub images add the organization (amazon/amazon-ecs-agent), and every other registry needs its host name (quay.io/org/image).
Prove the tag exists, in the account and Region in the reference:
aws ecr describe-images --repository-name booking --region us-east-1 \
--image-ids imageTag=v42 \
--query "imageDetails[].[imageTags,imageDigest,imagePushedAt]"
An ImageNotFoundException here means the task definition is pointing at something that does not exist. The usual reasons, in order of how often they bite:
- Region or account mismatch. The image was pushed to
us-east-2and the task definition saysus-east-1, or a copied task definition still carries the old account ID. - The pipeline never pushed the tag. A build step failed after the task definition was registered with the new tag.
- A lifecycle policy deleted it. Rules such as "keep the last 10 images" or "expire untagged images after 1 day" remove images that a running service still depends on. Check with
aws ecr get-lifecycle-policy --repository-name booking. - The digest is gone. This one deserves its own section, below.
"API error (400): invalid reference format" is different: the string is not a valid image reference at all, so nothing was even looked up. Look for a leading https://, a trailing space or newline from a copy-paste, uppercase letters in the repository path (repository names are lowercase), or an unexpanded variable such as ${IMAGE_TAG} left in by a template that did not run. The image field allows up to 255 characters of letters, numbers, hyphens, underscores, colons, periods, forward slashes and number signs.
The tag exists and ECS still says "not found": how digest pinning works
This is Ethan's Saturday, and it is the most confusing version of the error because everything you check by hand looks fine. When a service deployment starts, ECS takes the tag in your task definition (say booking:latest) and resolves it to the image digest it points at right then. The first task the scheduler starts in that deployment establishes the digest. From then on, ECS starts every other task in that service, including scale-outs and replacements for crashed tasks, from that digest, not from the tag. The point is consistency: every task in a service runs byte-identical images, even if someone pushes a new :latest in the middle of the day.
Now replay Jake's week. Monday's deployment pinned latest to digest A. On Thursday, Ethan pushed a new build and moved the latest tag to digest B, but did not redeploy the service. Image A was now untagged, and the repository's lifecycle rule "expire untagged images after 1 day" deleted it on Friday. On Saturday the service scaled out and asked ECR for digest A. ECR answered "not found," with the tag sitting right there pointing at B. The stopped reason even showed it: booking:latest@sha256:..., the tag with the pinned digest glued on.
Jake: "So the robot was being loyal to Monday's image." Ethan: "Loyal to an image we had thrown away. The fix is to tell it which image we mean today."
The immediate fix is a fresh deployment, which resolves the tag again:
aws ecs update-service --cluster CLUSTER --service booking --force-new-deployment
The lasting fix is to stop reusing tags. Tag each build with something unique (the git commit SHA is the common choice), register a new task definition revision per release, and turn on immutable tags so a tag can only ever point at one image:
aws ecr put-image-tag-mutability --repository-name booking --image-tag-mutability IMMUTABLE
aws ecr describe-repositories --repository-names booking \
--query "repositories[].imageTagMutability"
With immutable tags, pushing an existing tag fails with ImageTagAlreadyExistsException, which is exactly what you want: the pipeline fails loudly at push time instead of the service failing quietly at scale-out time. If you need a few moving tags (a dev tag, say), immutable repositories accept tag exclusion filters for those names. And write lifecycle rules so they keep at least as many images as you could plausibly roll back to.
Some details worth knowing about digest resolution:
- It is on by default. A container can opt out with
"versionConsistency": "disabled"in its container definition; ECS then pulls whatever the tag points at for each new task. That trades the "not found" failure for tasks in one service running different images, which is a debugging problem of its own. - If ECS fails three times to resolve the digests at deployment, the deployment continues without them; with the deployment circuit breaker on, it fails and rolls back instead.
- On EC2, agent versions below 1.31.0 do not resolve digests; 1.31.0 to 1.69.0 resolve them only for ECR images; 1.70.0 and later do it for every registry. On Fargate Linux, platform version 1.3.0 is the minimum.
- Sidecars that ECS manages for you (the GuardDuty agent, the Service Connect proxy) are not pinned.
"toomanyrequests: You have reached your pull rate limit": Docker Hub limits
If any container in your task pulls from Docker Hub (a plain nginx, redis or datadog/agent image counts), you share Docker Hub's pull limits with everyone else coming from the same IP address. The full message is:
toomanyrequests: You have reached your pull rate limit. You may increase the limit by
authenticating and upgrading: https://www.docker.com/increase-rate-limits
The published limits, counted per 6 hours:
| Who is pulling | Limit |
|---|---|
| Unauthenticated | 100 per IPv4 address (or per IPv6 /64 subnet) |
| Docker Personal, signed in | 200 |
| Docker Pro, Team or Business, signed in | Unlimited (fair use) |
Here is the part that surprises people. A check we ran on October 16, 2026 asked Docker Hub's registry for its live rate-limit header as an anonymous user, and the answer used a one-hour window, not six hours:
ratelimit-limit: 100;w=3600
ratelimit-remaining: 100;w=3600
The w= value is the window in seconds, so that is 100 pulls per hour for that address. Treat the header your own address gets as the truth, because it is what the registry enforces. You can check yours with two commands (a HEAD request, which does not count as a pull):
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl -s --head -H "Authorization: Bearer $TOKEN" \
https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i ratelimit
Why ECS hits the limit so easily: every task in a private subnet goes out through the same NAT gateway, so the whole service, and every other service in that VPC, looks like one IP to Docker Hub. Fargate also keeps no image cache between tasks, so every new task pulls every image in full. A deployment that replaces 20 tasks, each with an app image and two sidecars from Docker Hub, can be 60 pulls before the first health check. Two deployments and a scale-out on a busy morning, and you are over.
Three fixes, from quickest to most durable:
- Authenticate the pulls with a Docker account, using
repositoryCredentialsin the container definition (the private registry section shows how). A paid Docker plan removes the limit; a free account raises it. - Copy the image into ECR once and point the task definition at your copy. You control when it updates, and pulls from ECR do not count against Docker Hub.
- Use an ECR pull-through cache rule, which does that copying automatically on first pull and refreshes it. Next section.
Pull-through cache for Docker Hub, and the two permissions everyone forgets
A pull-through cache rule tells ECR: "when someone pulls docker-hub/... from my registry, fetch it from Docker Hub the first time and keep a copy." After that, your tasks pull from ECR, inside AWS, and ECR checks the upstream for a newer version at most once every 24 hours. If Docker Hub is down or rate-limiting at that moment, ECR still serves the last cached copy.
Docker Hub as an upstream requires credentials, stored in Secrets Manager. The secret's name must start with ecr-pullthroughcache/, and it must live in the same account and Region as the rule:
aws secretsmanager create-secret --name ecr-pullthroughcache/docker-hub \
--secret-string '{"username":"YOUR_DOCKER_ID","accessToken":"YOUR_DOCKER_ACCESS_TOKEN"}'
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url registry-1.docker.io \
--credential-arn arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:ecr-pullthroughcache/docker-hub-AbCdEf
Then change the image in your task definition. Official Docker Hub images need the library/ segment; everything else does not:
ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/docker-hub/library/nginx:1.29
ACCOUNT.dkr.ecr.us-east-1.amazonaws.com/docker-hub/datadog/agent:7
Now the trap. The first pull of each image does more than read: ECR has to import the upstream image and, if the repository does not exist yet, create it. Those two actions are checked against whoever is pulling, which for a Fargate task is the execution role, and AmazonECSTaskExecutionRolePolicy includes neither. Add them, scoped to the prefix:
{
"Effect": "Allow",
"Action": ["ecr:BatchImportUpstreamImage", "ecr:CreateRepository"],
"Resource": "arn:aws:ecr:us-east-1:ACCOUNT:repository/docker-hub/*"
}
ecr:CreateRepository is only needed until the cached repository exists, so another option is to pre-create the repositories (or pull each image once from a machine with broader rights) and grant only ecr:BatchImportUpstreamImage. Three more details catch people:
- If your tasks use the ECR VPC endpoints with no internet route, the first pull of a new image may still need a route to the internet. Do the first pull from somewhere that has one, or give the subnet a NAT path during setup.
- Repositories created by the cache have mutable tags, no lifecycle policy and no repository policy by default, unless you add a repository creation template.
- Pulling a multi-architecture tag caches the whole manifest list and every architecture in it. Pull a specific architecture's tag or digest if you only want one.
Ethan moved the booking service's nginx sidecar to the cache on Saturday afternoon. Jake asked whether that meant they now ran their own Docker Hub. "A small shelf of it," said Ethan, "with only the five images we use."
Private registries outside AWS: GitLab, Quay, Artifactory and private Docker Hub repos
For an image in a private registry that is not ECR, the agent needs a username and password, and ECS reads them from Secrets Manager at pull time. Without them you get "pull access denied for registry.gitlab.com/..., repository does not exist or may require 'docker login'", which is the registry refusing an anonymous request.
Store the credentials as a secret in this exact shape:
{
"username": "privateRegistryUsername",
"password": "privateRegistryPassword"
}
Point the container at it:
"containerDefinitions": [
{
"name": "app",
"image": "registry.gitlab.com/team/booking:1.4.2",
"repositoryCredentials": {
"credentialsParameter": "arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:gitlab-pull-AbCdEf"
}
}
]
Then give the task execution role secretsmanager:GetSecretValue on that secret, plus kms:Decrypt on the key if the secret uses a customer-managed KMS key rather than the default one. The secret may live in another Region or another account; in the console, and for any secret in another account, use the full ARN rather than the name. This needs Fargate platform version 1.2.0 or later, or container agent 1.19.0 or later on EC2. On EC2 there is also an older method using agent environment variables on the instance, but the per-container secret is easier to rotate and audit.
If the task now fails with a ResourceInitializationError that mentions "unable to pull secrets or registry auth," the credentials exist but the task cannot reach Secrets Manager; that is a network or permission issue on the secret, covered in the ResourceInitializationError guide.
"No space left on device": Fargate ephemeral storage and full EC2 disks
The image has to be downloaded and unpacked somewhere before the container starts. On Fargate (platform 1.4.0 or later) each task gets 20 GiB of ephemeral storage by default, shared by the image layers, the container's writable layer and anything the task writes to its local disk. A large image (think machine learning images with CUDA libraries) can fill it while extracting:
CannotPullContainerError: ref pull has been retried 1 time(s): failed to extract layer
no space left on device: unknown
Raise the allocation in the task definition, up to 200 GiB:
"ephemeralStorage": { "sizeInGiB": 60 }
Storage above the default is billed, so size it to the image plus what the task writes, not to the maximum. The error can appear even when the image is well under 20 GiB, because the extracted layers take more room than the compressed download and other files share the space.
On EC2, the host's volume is shared by every container on it, plus all their logs. Find what is eating it on an ECS-optimized AMI:
du -Sh / | sort -rh | head -20
A frequent culprit is container log files from the default json-file driver with no size cap. Either cap them in the container definition, or send logs to CloudWatch with awslogs so they do not land on the disk at all:
"logConfiguration": {
"logDriver": "json-file",
"options": { "max-size": "256m" }
}
Older hosts using the devicemapper storage driver fail with "Thin Pool has 4405 free data blocks which is less than minimum required 4480 free data blocks." That is the same problem in a different storage layout: clean up unused images and stopped containers (the ECS agent's automated image cleanup settings handle this over time), or move to a current ECS-optimized AMI with a larger volume.
Slow pulls that time out: big images, Windows and lazy loading
Sometimes nothing is misconfigured and the pull is simply too slow. Fargate keeps no image cache between tasks, so every task downloads every image in full, and the time it takes is part of every task's startup. A few things shorten it:
- Keep the image in the same Region as the cluster. Cross-Region pulls are slower and add data transfer.
- Shrink the image. A minimal base image (the minimal Amazon Linux 2023 container image, or a distroless image) can be a fraction of the size of a full OS base.
- Use zstd compression for layers, which decompresses faster than gzip; a bigger task size also gives the runtime more CPU to decompress.
- Lazy-load large images with SOCI (Seekable OCI). For images over about 250 MB, Fargate can start the container before the whole image has arrived, fetching files as they are read.
Windows containers have an extra limit: the agent setting ECS_IMAGE_PULL_INACTIVITY_TIMEOUT defaults to 3 minutes on Windows, and Windows images are large. If a pull stalls past that, it is canceled and the task fails. Keep Windows images lean and in the same Region, or raise the setting on EC2 hosts you manage.
Errors that look like pull failures but are not
A few errors show up in the same searches and get the same wrong fixes. Telling them apart saves an hour:
- ResourceInitializationError: unable to pull secrets or registry auth. The task failed earlier, while fetching the ECR login token or a secret, before any image pull. Same plumbing, different stage; see the ResourceInitializationError guide.
- "exec user process caused: exec format error". The pull worked. The image was built for a different CPU architecture than the task runs on, for example an image built on an Apple Silicon Mac (arm64) running on the default x86_64 Fargate. Either build for the target with
docker buildx build --platform linux/amd64, or run the task on Graviton by setting"runtimePlatform": {"cpuArchitecture": "ARM64", "operatingSystemFamily": "LINUX"}. Every task definition used by a service must use the same architecture. - "Essential container in task exited". The image pulled and started, and your process ended. That is an application or configuration problem; the essential container guide walks the exit codes, and exit code 137 is usually memory.
- "Your authorization token has expired" on your laptop or CI. That is
docker pushordocker pullon a machine with a 12-hour ECR login, not ECS; ECS refreshes its own tokens. See the ECR token guide.
Jake: "How do you keep all these straight?" Ethan: "One question. Did my code ever run? If no, it's a pull, a secret or the network. If yes, it's my code."
The ten-minute fix order
When a task will not start and you have not yet read the reason, this order clears the large majority of cases, because each step removes a cause that can hide the next:
- Read the full
stoppedReasonand each container'sreasonwithdescribe-tasks, and copy the exactimagestring ECS tried to pull. - Check the subnet's route table: public subnet means public IP ENABLED; private means a NAT route or the ECR, ECR API and S3 gateway endpoints (plus logs).
- Confirm the task definition has an
executionRoleArn(Fargate) and that the role hasAmazonECSTaskExecutionRolePolicyor the three ECR actions; on EC2, check the instance role. - Run
aws ecr describe-imageswith the exact repository, Region and tag from step 1. If the tag exists but the reason shows@sha256:, compare digests and force a new deployment. - If any image comes from Docker Hub, read the reason for "toomanyrequests" and move that image to ECR or a pull-through cache.
- For non-AWS private registries, confirm
repositoryCredentialsand the role'ssecretsmanager:GetSecretValue. - For "no space left," raise
ephemeralStorage(Fargate) or clear and resize the disk (EC2). - Still stuck: run the
AWSSupport-TroubleshootECSTaskFailedToStartrunbook against the newest failed task.
Fixes by symptom
| Symptom | Cause | Fix |
|---|---|---|
| Timeout on a brand-new Fargate service in a public subnet | No public IP on the task | Auto-assign public IP: ENABLED |
| Timeout after moving tasks to private subnets | No NAT route, no endpoints | NAT gateway route, or ECR, ECR API and S3 gateway endpoints |
| Timeout with endpoints present | Private DNS off on ecr.dkr, endpoint security group blocks 443, S3 gateway not on the route table | Enable private DNS, allow 443 from tasks, associate the route table |
| "not authorized to perform: ecr:BatchGetImage" | ECR actions on the wrong role, or no execution role | Attach AmazonECSTaskExecutionRolePolicy to the execution role |
| Denied from a second account | Repository policy only allows the owner | Repository policy naming the other account's role |
| "not found" with the tag visible in ECR | Pinned digest deleted after the tag moved | Force new deployment; unique, immutable tags |
| "not found" and the tag is missing | Wrong Region or account, pipeline did not push, lifecycle policy | Fix the URI, push, adjust lifecycle rules |
| "invalid reference format" | https:// prefix, space, uppercase, unexpanded variable | Clean the image string |
| "toomanyrequests" | Docker Hub anonymous limit through one NAT IP | Authenticate, copy to ECR, or pull-through cache |
| Pull-through cache image fails on first pull | Execution role lacks BatchImportUpstreamImage, CreateRepository | Add both, scoped to the cache prefix |
| GitLab or Quay "pull access denied" | No registry credentials | repositoryCredentials plus GetSecretValue on the role |
| "no space left on device" on Fargate | 20 GiB default ephemeral storage | ephemeralStorage up to 200 GiB |
| "no space left" or thin pool on EC2 | Full host disk, uncapped json-file logs | Clean up, cap logs or use awslogs, bigger volume |
Frequently asked questions about ECS CannotPullContainerError
What does CannotPullContainerError mean in ECS?
The task stopped before your container started because the agent could not download the image in the task definition. The text after the colon names the reason: no network path, no permission, image not found, a Docker Hub rate limit, no disk space, or missing private registry credentials.
How do I see the full CannotPullContainerError message?
Run aws ecs describe-tasks on the stopped task and read stoppedReason and each container's reason, or open the stopped task in the console's Tasks tab with the Stopped filter. Stopped tasks stay visible for about an hour.
How do I fix "request canceled while waiting for connection" on Fargate?
The task has no route to the registry. In a public subnet, set Auto-assign public IP to ENABLED. In a private subnet, add a NAT gateway route or, for ECR images, the ecr.dkr and ecr.api interface endpoints plus the S3 gateway endpoint.
Which VPC endpoints does Fargate need to pull from ECR?
com.amazonaws.REGION.ecr.dkr with private DNS enabled, com.amazonaws.REGION.ecr.api for platform 1.4.0 and later, and the com.amazonaws.REGION.s3 gateway endpoint on the subnets' route tables. Add the logs endpoint if the task uses awslogs.
Which role pulls the image on Fargate, the task role or the execution role?
The task execution role in executionRoleArn. The task role is for your application after it starts. On EC2 the container instance role pulls the image.
What permissions does ECS need to pull from ECR?
ecr:GetAuthorizationToken, ecr:BatchGetImage and ecr:GetDownloadUrlForLayer. AmazonECSTaskExecutionRolePolicy includes them for Fargate; AmazonEC2ContainerRegistryReadOnly is the minimum on the EC2 instance role.
Why does ECS say "not found" when the image tag exists in ECR?
ECS pinned the tag to a digest when the service deployed. If the tag later moved to a new image and the old one was deleted, new tasks ask for the deleted digest. Force a new deployment, then use unique, immutable tags.
What does "failed to resolve ref" mean?
The registry has no image at that reference: the repository, Region, account, tag or digest does not match anything. Check with aws ecr describe-images using the exact string from the stopped task.
How do I fix "invalid reference format" in an ECS task definition?
Clean the image string: remove any https:// prefix, spaces or newlines, use lowercase repository names, and make sure template variables such as ${IMAGE_TAG} were replaced before the task definition was registered.
What is the Docker Hub pull rate limit for ECS tasks?
The published limits are 100 pulls per 6 hours per IP for anonymous users and 200 for signed-in Personal accounts, unlimited on paid plans. A live header check on October 16, 2026 showed 100 per hour for an anonymous address. All tasks behind one NAT gateway share one IP.
How do I stop hitting Docker Hub rate limits on ECS?
Authenticate pulls with repositoryCredentials, copy the images into ECR, or create an ECR pull-through cache rule for Docker Hub and point the task definition at the cached path.
Why does my ECR pull-through cache image fail on the first pull?
The pulling role also needs ecr:BatchImportUpstreamImage, and ecr:CreateRepository if the cached repository does not exist yet. AmazonECSTaskExecutionRolePolicy has neither. Also check the secret name starts with ecr-pullthroughcache/.
How do I pull a Docker Hub official image through ECR pull-through cache?
Include library in the path: ACCOUNT.dkr.ecr.REGION.amazonaws.com/docker-hub/library/nginx:TAG. Non-official images omit library.
How do I pull from GitLab or another private registry on Fargate?
Store {"username","password"} in Secrets Manager, set repositoryCredentials.credentialsParameter to the secret ARN, and give the execution role secretsmanager:GetSecretValue (plus kms:Decrypt for a customer-managed key). Fargate 1.2.0 or later.
How do I fix "no space left on device" when a Fargate task pulls an image?
Raise ephemeralStorage.sizeInGiB in the task definition above the 20 GiB default, up to 200 GiB, or shrink the image.
How do I fix "no space left on device" on ECS EC2 instances?
Find large files with du -Sh / | sort -rh | head -20, cap json-file logs with max-size or switch to awslogs, clean unused images, and use a larger root volume.
Is "exec format error" a CannotPullContainerError?
No. The image pulled and started but was built for another CPU architecture. Build with --platform linux/amd64, or set runtimePlatform cpuArchitecture to ARM64 for Graviton.
Does a NAT gateway work for pulling from Docker Hub on private subnets?
Yes, for the network path, but every task then shares the NAT's single public IP for Docker Hub's rate limit. VPC endpoints only cover ECR, not Docker Hub.
By Saturday lunchtime the booking service was running again on a fresh deployment. By Monday, Ethan's pipeline tagged every build with its commit, the repository refused to reuse a tag, the lifecycle rule kept the last 30 images instead of deleting anything untagged after a day, and the nginx sidecar came from the pull-through cache. Jake asked whether all that was for one Saturday. "It's for the next one," said Ethan. "This error always tells you what's wrong. We just made sure the answer is never 'the image you deployed on Monday is gone.'"
📌 If you keep one line from this page
CannotPullContainerError is a startup failure with six causes, and the end of the message names the cause: timeouts mean the route, "denied" means the execution role, "not found" with a visible tag means a deleted pinned digest, and "toomanyrequests" means Docker Hub.
Public IP or NAT or endpoints · execution role, not task role · unique immutable tags · pull-through cache for Docker Hub · ephemeralStorage for big images.
Revision note. Written October 16, 2026, with the error strings in both their old and current wording and a live Docker Hub rate-limit check. "Measure twice, cut once," as the carpenters say; read the reason twice, change one thing.