ECS "Essential Container in Task Exited": Read the Exit Code

Amazon ECS "Essential container in task exited" means a container you marked (or left defaulted) as essential stopped, so ECS stopped the whole task. It is a statement about the task's lifecycle, not about what went wrong inside the container. To find the why, check two numbers and one log: the container's exitCode and reason from aws ecs describe-tasks, and the last lines it wrote to CloudWatch. 137 means the process was killed, often by memory; 127 means the command was not found; 139 is a crash in native code; 143 is a polite shutdown, usually a deployment. The surprise that catches almost everyone once: exit code 0 produces this stop reason too. A container that finishes its work and exits cleanly has still exited, and an essential container that exits ends the task, success or not. If your "server" is really a script that ran and completed, ECS will restart it forever and call every restart a failure.

Jake runs a phone repair shop. The shop's order API, which the booking page talks to, runs as an ECS service that Ethan, his friend who does the technical side, set up last year. Lately the booking page goes offline for a minute or two at a time, usually when a customer is mid-booking. Jake opened the ECS console for the first time and found a stopped task whose reason says only "Essential container in task exited."

Jake: "Is AWS telling me the container was important? I know it was important, it's my order API."

Ethan: "It is telling you that a container which must stay alive has stopped, so it stopped everything around it. The next clue is the container's exit code, not the word essential."

Jake: "What's a container, in shop terms?"

Ethan: "A sealed box with your program and everything it needs inside. A task is one or more boxes that live and die together. ECS is the manager who keeps the right number of boxes running. The manager just told you one box went dark. Now we open the box and read the note it left."

If "container", "task" and "cluster" are new words, our plain-English page on what Amazon ECS is takes five minutes and explains the manager-and-boxes picture properly. Jake skipped it and got by on the analogy; you can do either.

What "essential" means, and why exit 0 still stops the task

A task definition is the recipe: which container images, which commands, how much CPU and memory, which environment variables, where logs go. A task is one running copy of that recipe. Each container in the recipe has an essential flag. If it is true, or if you never set it (the default is true), that container stopping stops the task. Every task must have at least one essential container.

Now the part that surprises people. For an API that serves requests, the main container should run forever, so it exiting is a failure. For a one-time job, a migration or a report, the main command is supposed to finish. The same stop reason, "Essential container in task exited," describes both a broken server and a job that completed. ECS cannot tell which you meant. You can.

That distinction prevents the worst common "fix": wrapping a program that exits after startup in a sleep loop so the container stays up. The restarts stop, the service shows green, and it cannot answer a single request, because the thing that exited was the server.

Jake: "So the manager isn't wrong to shut the box. The box said it was done."

Ethan: "The manager is never wrong about that. Our job is to find out why your order API thought it was done at 11:03 on a Tuesday."

The exit-code decoder

The exit code is a lead, not a verdict. Shells, signals, runtimes and your own code can all assign meaning to the same number. But the common ones point somewhere, and that somewhere is usually right:

Exit code	Usual meaning	Check first
0	The process completed successfully	Was this a job that should finish, or a server that should not?
1	Generic application failure	The traceback: config, credentials, a missing variable
2	Usage or shell error, or an app's own code	Arguments, command syntax
126	Found the command, could not run it	Executable bit, shebang, architecture
127	Could not find the command	Entrypoint name, PATH, what the image contains
137	128 + 9: SIGKILL	Out-of-memory evidence, stop timeout, an external kill
139	128 + 11: SIGSEGV, a segmentation fault	Native dependency, binary architecture, recent image change
143	128 + 15: SIGTERM	Deployment, scale-in, a stop request, shutdown handling
255	Whatever the program decided	Last log lines, wrapper scripts, the tool's own exit mapping

There is no ECS-wide rule that 255 means permissions or that 137 always means out of memory. A shell returns 128 plus the signal number when a process dies from a signal; everything else is up to the program. Exit 1 after "database connection refused" is a database setting. Exit 127 after exec: python: not found is the image. Same stop reason, two unrelated repairs.

Jake: "Why do they add 128 to everything?"

Ethan: "So you can tell 'the program chose to exit with 9' from 'the program was killed by signal 9'. Anything above 128 means something outside the program ended it. That is useful: 137 and 143 are both the operating system or ECS doing something to your container, not your code deciding to quit."

Finding stoppedReason and exitCode before they disappear

Stopped tasks stay in the console for one hour, and the DescribeTasks API keeps them for at least that long. After that the evidence is gone unless you captured it. So, quickly:

1. Pick the right Region and cluster; the default cluster is not where your API lives.
2. ECS → Clusters → your cluster → Tasks, filter desired status to Stopped. If the service is restarting fast, note the task ID and the latest stop time.
3. Open the task. Read the task-level stopped reason, then each container's exit code and reason.
4. Work out which essential container stopped first. Others may have been stopped because the task was already shutting down.
5. Open the Logs tab, or go to the log group and stream named in the task definition.

aws ecs list-tasks --cluster shop-cluster --desired-status STOPPED --region us-east-1

aws ecs describe-tasks --cluster shop-cluster --region us-east-1 \
  --tasks arn:aws:ecs:us-east-1:123456789012:task/shop-cluster/TASK_ID \
  --query 'tasks[].{task:taskArn,stopCode:stopCode,stoppedReason:stoppedReason,definition:taskDefinitionArn,containers:containers[].{name:name,status:lastStatus,exitCode:exitCode,reason:reason}}'

The query separates task-level fields from container-level ones on purpose. stopCode is a classification, here EssentialContainerExited; stoppedReason is the readable sentence; each container has its own exitCode and optional reason. A task with three containers can have three different endings. If list-tasks returns nothing, check the service events, the cluster and Region, and whether the task aged out. For next time, capture ECS Task State Change events with EventBridge so the stop metadata survives the hour.

Jake: "One hour? A customer complained at lunch and I'm looking now at four."

Ethan: "Then today's evidence is gone and we wait for the next one, which will not be long. Tomorrow we set up the event capture so this never happens again. Today we read the one that stops while we are watching."

Which container pulled the first domino

Jake's task has three containers: the API, a proxy, and a log router. All three are stopped. Only one caused the stop. Compare each container's essential setting in the task definition with its exit code: an essential API that exits 1 can stop the task; a log router that exits 143 during that shutdown is a consequence. Unless the log router is marked essential and failed first, in which case it is the cause, and the healthy API was stopped for its sake.

aws ecs describe-task-definition --task-definition shop-api:12 --region us-east-1 \
  --query 'taskDefinition.containerDefinitions[].{name:name,essential:essential,image:image,command:command,entryPoint:entryPoint,logging:logConfiguration}'

Use the exact task definition ARN from the stopped task, not the newest revision of the family. A service may be running an older revision, or a rollback may have restored one, and reading today's newest recipe does not explain yesterday's stopped box.

Ethan: "The second container dying is not automatically your first bug. Find out who pulled the first domino."

Jake: "The log router was added last week, with the new monitoring thing."

Ethan: "Then it is the first suspect, and the API might be an innocent bystander. A sidecar added during a deployment is the classic domino."

Exit 0: the server that was really a script

The entrypoint of an image is the command that runs when the box opens. npm run build, a migration, a setup script: all correct in a pipeline or a one-off task, all wrong as the main process of a web service, because they finish:

# one-shot: fine for a finite job, fatal for a web service
CMD ["node", "scripts/migrate.js"]

# long-running: the process owns the server and stays in the foreground
CMD ["node", "server.js"]

Check CMD and ENTRYPOINT in the Dockerfile and any command or entryPoint override in the task definition; the override wins. A shell script that starts a server in the background and then exits has the same effect: the script was the main process, the script ended, the box closed. Use exec for the final foreground process so it owns the container and receives shutdown signals directly. And if the work genuinely is finite, run it as a scheduled or standalone task and treat exit 0 as the success it is.

Exit 1 and 2: let the application speak

Codes 1 and 2 are the app saying "something went wrong" without saying what. The what is the first meaningful error line above the shutdown. Node: a stack trace, a missing environment value. Python: a traceback naming a missing package, a wrong startup module, a failed connection. Compiled apps: tell the program's own codes from a launcher script returning 2.

An image that expects DATABASE_URL exits 1 when it is missing. Look at the container definition's environment, environmentFiles and secrets. Two roles matter and people swap them: the task role is what your application code uses to call AWS APIs; the execution role is what ECS itself uses to pull the image, fetch startup secrets and send logs. Then run the same image digest and arguments locally with a safe staging configuration, and keep production secret values out of your terminal history.

Exit 126 and 127: the command, the script, the wrong chip

A new image that stops almost at once with 126 or 127 is an image problem, not a resource problem. 127 means the shell could not find the command named in the entrypoint; 126 means it found a file it could not execute. Compare entryPoint and command in the task definition with ENTRYPOINT and CMD in the image, check that /app/start.sh has its executable bit and a valid interpreter on its first line, and check for Windows line endings on scripts edited on a laptop:

docker image inspect my-api:diagnostic --format '{{json .Config.Entrypoint}} {{json .Config.Cmd}}'

# in the Dockerfile, after fixing the script
# RUN chmod +x /app/start.sh
# ENTRYPOINT ["/app/start.sh"]

Two more causes wear these numbers. A task running on ARM64 with a binary compiled for x86_64 is a compatibility problem that can look like 126. And a minimal base image may not contain bash, or even sh, so a command that invokes a shell fails with 127 on an image where that shell does not exist.

Ethan: "For 127, ask where the command went. For 126, ask why the thing you found cannot run. Two targeted checks instead of another full redeploy with the same broken command."

Exit 137: killed, by memory or by the clock

137 is SIGKILL. An out-of-memory kill is the common cause, and it is not the only one: when ECS stops a container it first sends SIGTERM, waits for the stop timeout, and then sends SIGKILL to anything still running. A container that ignores the polite request dies with 137 during an ordinary deployment, and more memory will not change that.

Look for a stopped reason that mentions out-of-memory, check the memory metrics and the application's own logs, and compare the task-level and container-level memory limits in the task definition. If the exits happen only during deployments or scale-in, you are looking at termination timing: add SIGTERM handling that finishes in-flight work and closes cleanly. A container configured for 512 MiB whose workload briefly needs 650 MiB has a 138 MiB problem, and that is a reasonable hypothesis, not proof; measure before you change the definition, because every raised limit costs money and fixes nothing if the kill came from the clock. Our dedicated 137 page goes through the memory side in full.

Exit 139: a crash in native code

139 is SIGSEGV, a segmentation fault: the process touched memory it should not have. That is a native-code crash, not an application exception, and it almost always follows a change: a new native dependency, a compiled binary, a different processor architecture, a new base image. Check that the image was built for the platform in runtimePlatform and that its dynamic libraries match the image's operating system. Java, Python, Node and Go all crash this way when a native extension misbehaves, and the evidence is a fatal-signal line or a native stack trace on stderr rather than a normal exception. Rolling back to the previous image digest is sensible containment while you diff the two images; it is not the diagnosis.

Exit 143: usually a deployment doing its job

143 is SIGTERM, the polite stop. During a rollout or a scale-in, ECS asks old tasks to stop, and a well-behaved container exits 143. Correlate the stop time with service deployment history and events; if a replacement task is healthy and the old one left with 143, that is the system working. Make sure the server handles SIGTERM by finishing in-flight requests and closing listeners, set the container stop timeout to fit, and let the load balancer drain. Not every 143 is healthy, because an app can also receive SIGTERM from another source: a task that should run indefinitely and got terminated with no deployment, no scale-in and no user stop in the timeline still has something to explain.

Exit 255: the program's own number

An online answer will insist 255 means a specific AWS permission failure. It does not. 255 is whatever the program or its wrapper decided to return for "something bad". Start with the executable that actually ran, not the language it was written in: a shell wrapper can mask its child's exit code, an application can catch an exception and call exit(255) deliberately, an SSH-style tool uses it for connection failures. Read the container reason, the last stderr lines and any launcher script. In AWS Batch, read each attempt separately, because a retry can change the number you see.

Jake: "So 255 means I rewrite the permissions?"

Ethan: "255 is a request to investigate, not permission to rewrite anything. Find the process that chose 255. The number did not choose the cause for you."

Why the Logs tab is empty

No log stream is not proof the process said nothing. The container may have died before logging began, logging may not be configured, or the logging driver may have failed to start. Read the task definition's logConfiguration: the awslogs driver, the log group, the Region, the awslogs-stream-prefix, and whether the execution role may create streams and put events:

aws ecs describe-task-definition --task-definition shop-api:12 --region us-east-1 \
  --query 'taskDefinition.containerDefinitions[].{name:name,logConfiguration:logConfiguration}'

aws logs describe-log-streams --log-group-name /ecs/shop-api --log-stream-name-prefix ecs/ --region us-east-1

If the task cannot create or reach the log group, it can stop with a resource-initialization or logging-driver error before your app prints a word; in private subnets that usually means a missing VPC endpoint or route to CloudWatch Logs. If the group exists but no stream matches, you are probably in the wrong Region or the prefix is misspelled; streams are named prefix/container-name/task-id. On ECS with EC2 hosts you also have the container runtime and ECS agent logs on the host; on Fargate you do not, which is one more reason to configure logging and event capture before the next incident instead of after. Our CloudWatch page covers the log group side in plain English.

Choosing essential on purpose

Should the whole task stop when the helper container dies? That is an architecture decision expressed through one flag, not a way to make a red task green:

Container role	Usual choice	Risk of choosing badly
API or main worker	Essential	Non-essential: the task keeps running with nothing serving requests
Optional metrics exporter	Non-essential	Its death goes unnoticed, and so do the metrics
Required proxy	Usually essential	Non-essential: the app is up and unreachable
Security or policy helper	Depends on the design	The app continues without the enforcement you intended
Temporary setup helper	Use a dependsOn pattern	A short-lived essential process stops the task when it finishes

Changing the flag means a new task definition revision. dependsOn orders startup and sets conditions; it does not turn an essential process exiting into a harmless event. Keep setup steps and the server's lifecycle separate and explicit.

On Fargate

On Fargate there is no host to log in to and inspect Docker, which is the point of Fargate. Use what it gives you: task metadata, CloudWatch logs, service events. Check the launch type, platform version, task CPU and memory and each container's allocation; if the task dies during startup, check the execution role, security-group egress, subnet routes, secrets access, ECR connectivity and logging. The container inside is still a Linux process with ordinary Linux failure modes, so 1, 127, 137, 139 and 255 mean exactly what they mean elsewhere. Fix with a new task definition revision deployed through the service, or a controlled rollback to the last healthy revision while you look; a stopped standalone task does not resume, you launch a new one.

Jake: "I thought Fargate meant AWS looks after the servers so this stuff isn't my problem."

Ethan: "Fargate removes the host-maintenance job. It does not remove responsibility for the process inside your image. If your box has a bad entrypoint, it has a bad entrypoint on Fargate too."

On ECS with EC2 hosts

With your own container instances you get one more layer: the host. The task and container descriptions remain the starting point, and you can also read the container runtime and ECS agent logs on the instance. Compare failures across hosts: every task from one image failing everywhere points at the image; failures clustering on one host point at that host's memory, disk, runtime or agent connectivity. Container-level memory limits can kill a container even when the host has memory to spare, and heavy host pressure can hurt everything on it. Collect the instance ID, the container instance ARN, runtime messages and agent logs before touching a production host that other tasks share.

The AWS Batch variant

AWS Batch runs ECS-backed jobs, so the same message appears there with the same meaning; start with the job and its attempts rather than the task. AWS Batch → Jobs → the job: read status and statusReason, then each attempt's exit code, reason and log stream, because an earlier attempt can fail with one code and the retry with another:

aws batch describe-jobs --jobs JOB_ID --region us-east-1 \
  --query 'jobs[].{jobId:jobId,status:status,statusReason:statusReason,attempts:attempts[].{statusReason:statusReason,exitCode:container.exitCode,reason:container.reason,taskArn:container.taskArn,logStream:container.logStreamName},currentExit:container.exitCode}'

ECS-backed Batch jobs log to /aws/batch/job; the attempt's logStreamName is the exact stream. If an attempt has no stream, it never got far enough to make one. Batch can also run on EKS with multi-container jobs; if the description shows eksProperties instead of the container fields, look at the pod. Some Batch errors live outside the ECS console, so read the job definition too. And for a finite job, exit 0 is the expected result: "essential container exited" there is just the job finishing.

The timeline: logs plus service events

A task that restarts every few minutes is hard to catch live, and you do not need to. Combine the container log with the service's events and the task's created, started and stopped timestamps:

aws logs tail /ecs/shop-api --since 30m --region us-east-1

aws ecs describe-services --cluster shop-cluster --services shop-api-service --region us-east-1 \
  --query 'services[].{status:status,desired:desiredCount,running:runningCount,pending:pendingCount,events:events[0:8]}'

A service event saying tasks were stopped for failed health checks or deployment replacement is not the same as the process throwing. If the task's stopCode is scheduler-initiated, you are looking at planned lifecycle, not a spontaneous exit. For a permanent record, EventBridge's ECS Task State Change events carry the stop metadata and container exit codes and outlive the one-hour console window; capture and retention have their own small CloudWatch and EventBridge charges.

Worked example: 137 on Tuesday, 143 on Thursday

Two stopped tasks from Jake's order API, same week. Revision A stopped with exit 137 during a burst of image-processing jobs (customers uploading photos of cracked screens). Revision B stopped with 143 while the service was rolling out a deployment. Same stopped reason, different stories:

Evidence	Revision A	Revision B
Container exit	137	143
Signal	SIGKILL	SIGTERM
Likely story	Memory, or a forced kill after the stop timeout	Graceful shutdown ordered by the scheduler
Companion data	Memory metrics, container reason	Stop code, service events
Likely correction	Resource or workload change	Shutdown handling, deployment settings

For A, Ethan checked the memory allocation (1 GiB) against the image-processing load, found the container reason mentioned memory, and still looked for a deliberate kill before saying "OOM". For B, he matched the stop time to the deployment event and confirmed the replacement task was healthy; the only improvement was finishing in-flight requests before exit so nobody's booking was dropped mid-save. One task-hour of 1 vCPU and 2 GiB running alongside its replacement during a rollout is the compute cost of the whole episode; the real cost is in the next sentence.

Jake: "If the task dies halfway through saving a booking and restarts, does the customer get booked twice?"

Ethan: "Only if the code lets it. The retry has to be able to say 'I have already done this one'. That is called idempotency, and it matters more than any of the container minutes we have been counting. A restart loop that repeats a notification three times costs you a customer, not cents."

Health-check failures that look like exits

A service event about failed load-balancer or container health checks is a different sequence: the task was alive, judged unready, and replaced; its containers then exited. Check the target group's health path, expected codes, startup grace, port mapping and the app's readiness logic; a check on / fails if the app only answers on /health. For container health checks, review the task definition's healthCheck command, interval, retries, timeout and start period, and make sure the image contains the tool the check runs. A minimal image with no curl makes a healthy app look dead. Change one of the three, process, container health, target health, at a time, so you can see which one mattered.

Roles, secrets and network when startup fails

A container that exits while fetching a secret or opening a connection fails in one of two layers, and the stop fields tell you which. Resource-initialization reasons are ECS failing before launch: execution role, ECR reachability, logging. An application error after launch is the task role (what the app's own AWS calls use) or the network: private Fargate tasks need VPC endpoints or outbound connectivity for ECR, CloudWatch Logs, Secrets Manager and whatever the app talks to. A DNS failure shows up as an app exit 1; a logging failure prevents any output at all. While you diagnose a missing permission, keep the full secret value out of bug reports and logs, and fix the smallest permission or route that explains what you saw.

Service or one-off task: fix the right thing

The same image can be an always-on service and an occasional job, and the two have different correct lifecycles. A service keeps a desired number of tasks and replaces stopped ones; a standalone task stops when its essential containers finish. If the image runs a web server, give it a foreground entrypoint and health checks; if it runs a migration, let it exit and read the result. When you register a corrected task definition, point the service at it; a stopped standalone task is launched anew. And an updated image tag does not guarantee the running deployment uses the new digest:

aws ecs update-service --cluster shop-cluster --service shop-api-service --task-definition shop-api:13 --region us-east-1

Then watch the new tasks. If one stops again, read its stopped reason before the next broad change.

Recovering before you understand

Recovery and diagnosis can run on separate tracks. If an earlier task definition is known good, deploy that exact revision, by digest and revision number rather than a mutable tag, while you investigate. Before you do, ask whether the new release changed a database schema, a queue message format or shared storage; old code may not safely read what new code wrote. A successful UpdateService response means ECS accepted the request, not that the app is healthy, so watch desired and running counts settle and then make a real booking request. If tasks keep stopping after the rollback, the image was coincidental: look at the database, the secrets, the network. For standalone tasks and Batch jobs, run a corrected definition rather than reviving the stopped one, and if a retry could repeat a payment or a notification, make the job idempotent or reconcile what the first run did.

Hearing about the next one before a customer does

If a restart means missed orders, do not learn about it from the phone. EventBridge can match ECS task state changes with your stop code and send an alert with the task ARN:

{
  "source": ["aws.ecs"],
  "detail-type": ["ECS Task State Change"],
  "detail": {
    "lastStatus": ["STOPPED"],
    "stopCode": ["EssentialContainerExited"]
  }
}

Watch desired versus running count, error rates, latency and job success too. One task stopping during a deployment is routine; an essential container exiting 1 every five minutes is not. Tune alerts around customer impact, not around the existence of any STOPPED event, and remember the capture itself has a small cost.

Five changes that make a stopped task worse

Each of these turns the service green for an hour and leaves the real fault in place, or deletes the one thing that would have explained it:

• Raising memory before reading the reason. If the 137 came from the stop timeout during a deployment, you now pay for memory nobody uses and the container still ignores SIGTERM.
• Wrapping the entrypoint in a sleep loop. The container stays up, the task stays green, and nothing is listening. You have converted a visible failure into an invisible one.
• Marking every container non-essential. The task never stops, so the service never replaces it, so the dead API sits there receiving traffic until a customer tells you.
• Deploying by image tag. latest moved under you. The task definition says the same thing it said yesterday and runs different code. Pin digests.
• Forcing a new deployment to "clear it". The stopped task that held the exit code ages out of the console while the new one fails the same way. Read first, redeploy second.

Jake: "The monitoring vendor's guide said to set the router to non-essential and move on."

Ethan: "Which is the right call for a log router, and the wrong call for your API. The flag is a decision about what the task can live without. Make it per container, on purpose, not as a way to stop the red rows."

What to collect when the first fixes fail

1. Region, account, cluster, service, task ARN, task definition ARN and image digest, with no secret values.
2. The full describe-tasks output: stopCode, stoppedReason, every container's reason and exit code.
3. The application log stream and service events, or a note on why no stream exists.
4. Launch type, Fargate platform version or EC2 instance, CPU and memory, and each container's essential flag.
5. The last healthy task definition and image digest beside the failing ones.
6. The smallest repeatable reproduction, with the time range and a request or job ID.

Ethan: "Show the evidence in order: task reason, container code, first real log error, and the revision that changed. That is a bug report another person can act on. A screenshot of a red row is not."

The exact ECS essential-container questions people search for

Jake's domino was the log router added with the new monitoring, marked essential by default, crashing on a configuration it could not read and taking a perfectly healthy order API down with it every few minutes. Ethan marked it non-essential, fixed its configuration, and set up the EventBridge rule so the next stop would land in an inbox instead of a customer's phone call. The order API itself never had a bug. Jake keeps the decoder table printed by the counter now, which he admits he mostly uses to look knowledgeable when the monitoring vendor calls.

Revision note. Written October 9, 2026, with the stop-code fields and the Fargate and Batch variants covered. Even an elephant slips, says a Tamil proverb; a healthy container can exit 0 and still stop the whole task.