Auto Scaling Group Keeps Terminating Instances: ELB Health Check Failure, Grace Period, Launch Errors, and How to Stop the Loop
When your EC2 Auto Scaling group launches instances that are terminated a few minutes later, again and again, the group is almost always replacing them because they failed a health check. The most common cause is the ELB health check: your group uses the load balancer's target health, your application is not answering on the health check path and port yet (or ever), and the health check grace period is shorter than the time your instance needs to boot, start the app and pass enough checks. Other causes: the instance fails EC2 status checks (a broken AMI or user data), it was stopped or rebooted by hand, it never really launched (a launch failure such as a vCPU quota, no capacity, or an encrypted volume whose KMS key the group cannot use), a scaling policy or scheduled action scaled in, Spot interruptions, a Capacity Block ending, AZ rebalancing, an instance refresh or a maximum instance lifetime. Your group's Activity tab names the exact cause for every termination. Below: how to read it, how to stop the loop long enough to debug, the target group reason codes, the grace period math, launch errors, and the fix for each cause.
The shop's new web servers ran in an Auto Scaling group behind a load balancer, and on the day of the switch-over the group would not settle. Every few minutes an instance launched, ran for a while, and was terminated; a fresh one replaced it, and the cycle repeated. Nothing was crashing. The application took four minutes to warm up, the group gave it ninety seconds before trusting the load balancer's opinion, and the load balancer's opinion during those four minutes was "unhealthy". The group was doing exactly what it had been told: replace unhealthy instances.
New to Auto Scaling or load balancers? The cards below cover Auto Scaling, ALB errors, user data and EC2 capacity errors.
What an Auto Scaling group checks before it terminates
If you are still asking what an Auto Scaling group is: it is a set of EC2 instances that AWS keeps at a number you choose (the desired capacity), between a minimum and a maximum, all launched from one launch template. When an instance goes away or is judged unhealthy, the group launches a replacement so the count stays right. That self-healing is the whole point, and it is also why a small mistake turns into a loop: every replacement inherits the same mistake. Our Auto Scaling explainer covers the basics if you are setting up your first group.
Every instance starts as Healthy. The group only changes its mind when a health check source reports a problem. These are the sources it can listen to:
| Health check | What it looks at | On by default? |
|---|---|---|
| EC2 status checks and scheduled events | The instance is in the running state, its system and instance status checks are not impaired, and no scheduled retirement is pending | Yes, always on; it cannot be turned off |
| Elastic Load Balancing (ELB) | The target group's health check says the instance can serve requests | No; you turn it on with health check type ELB |
| VPC Lattice | The VPC Lattice target group reports the instance healthy | No |
| Amazon EBS | Attached volumes are reachable and pass I/O status checks | No |
| EC2 application status checks | An HTTP or HTTPS check, set up in EC2 and tied to the group's instances by tag; an impaired overall status gets the instance replaced | Only if someone associated one with the group's tag |
| Custom health checks | Your own script calls set-instance-health to mark an instance Unhealthy | Only if you built one |
Check which ones your group uses, and its grace period, with describe-auto-scaling-groups:
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names shop-web \ --query "AutoScalingGroups[].[HealthCheckType,HealthCheckGracePeriod,MinSize,DesiredCapacity,MaxSize]" --output table
If HealthCheckType is EC2, the load balancer cannot be the reason your instances die, and you can skip straight to the status check and launch sections. If it is ELB, the load balancer's opinion counts, and that is where most loops start.
Jake: "So the group isn't randomly killing things. Something told it the instance was sick."
Ethan: "Right. Our job is finding out who said it, and why."
Step one: read the Activity history
An Auto Scaling group records why it launched or terminated every instance. In the EC2 console: Auto Scaling groups, your group, the Activity tab. Each termination has a Cause line in plain words. With the CLI:
aws autoscaling describe-scaling-activities --auto-scaling-group-name shop-web \ --max-items 10 --query "Activities[].[StartTime,StatusCode,Description,Cause]" --output text
The cause text tells you which branch of this page you are on. Typical examples:
- "... an instance was taken out of service in response to an ELB system health check failure": the load balancer's target health check failed. Go to ELB failures.
- "... in response to an EC2 instance status check failure": the instance or its host is impaired. Go to status checks.
- "... in response to an EC2 health check that indicated it had been terminated or stopped": someone (or a script, or user data) stopped, rebooted or terminated it, or Spot took it back.
- "... a user request update of AutoScalingGroup constraints to min, max, desired" or "a monitor alarm ... in state ALARM triggered policy ...": a person, a deployment or a scaling policy changed the desired capacity.
- "... instance rebalancing", "... instance refresh", or "... maximum instance lifetime": planned replacements.
- A failed launch with a status message (quota, capacity, invalid parameter, Client.InternalError): the instance never got going. Go to launch failures.
You can also see the group's current verdict on each instance. An instance marked Unhealthy here is about to be replaced:
aws autoscaling describe-auto-scaling-instances \ --query "AutoScalingInstances[?AutoScalingGroupName=='shop-web'].[InstanceId,LifecycleState,HealthStatus,AvailabilityZone]" --output table
Jake: "So it writes down why it fired each one?"
Ethan: "Every single time. Most people just never open that tab."
Every cause, and its fix
| Activity says | Real cause | Fix |
|---|---|---|
| ELB system health check failure | Target unhealthy: app not listening, wrong path or port, security group blocks the load balancer, or start-up slower than the grace period | Check target health reasons; fix the check; raise the grace period |
| EC2 instance status check failure | Instance failed status checks (bad AMI, kernel, disk, host problem) | Read the console output; test the AMI on a single instance |
| EC2 health check: terminated or stopped | Manual stop or reboot, user data that shuts down, Spot interruption | Use Standby before stopping; fix user data; diversify Spot |
| Launch failed: vCPU limit, InsufficientInstanceCapacity | Quota or capacity, not a health problem | Raise the quota; add instance types and zones |
| Client.InternalError: Client error on launch | Encrypted EBS with a KMS key the group's role cannot use | Key policy or grant for the Auto Scaling service-linked role |
| Alarm triggered policy / desired capacity changed | Scale-in from a policy, schedule, or a deployment tool | Review the policy and min/desired; expected behavior |
| Instance rebalancing | Group evens out instances across zones | Expected; suspend AZRebalance if it disrupts you |
| Instance refresh / maximum instance lifetime | Planned replacement you (or a tool) configured | Expected; adjust the settings |
| Terminated near a Capacity Block end time | EC2 ends instances 30 minutes before a Capacity Block expires | Scale in yourself before the end; lower desired capacity |
| Lifecycle hook abandoned or timed out | A launch hook was not completed in time | Complete the hook from your bootstrap, or lengthen its timeout |
ELB health check failures: the most common loop
If your group's health check type is ELB, an instance is replaced when the load balancer's target group reports it unhealthy (once the grace period is over). The group checks both: EC2 status checks and the load balancer. So your instance can be perfectly fine as a machine and still be terminated because the application behind port 80 never answered. Find out why the target is unhealthy:
aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/shop-web/0123456789abcdef \ --query "TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]" --output table
The reason codes point straight at the fix:
| Reason code | What it means for you | Where to look |
|---|---|---|
Target.Timeout | The load balancer cannot reach the instance at all | Instance security group must allow the health check port from the load balancer's security group; app must listen on that port |
Target.ResponseCodeMismatch | The app answered, but not with the expected status | A 301 to HTTPS, a 401 login page, or a 404 on the wrong path; use a simple /health path or widen the matcher |
Target.FailedHealthChecks | Connections or answers failed | The app's own logs; a crashing process; a full disk |
Elb.InitialHealthChecking | Still in the first checks after registration | Normal for a minute or two; it matters only if it outlasts your grace period |
Target.NotInUse | The target's zone is not enabled on the load balancer, or no listener uses the target group | Enable the same Availability Zones on the load balancer as the group uses |
Work through it in this order; each step rules out a whole class of problems:
- Connect to an instance (Session Manager or EC2 Instance Connect; suspend replacement first, see below, if they die too fast).
- Test the health check from inside:
curl -i http://localhost:8080/healthwith your health check port and path. No answer means the app is the problem; check its service status and logs. - Compare the status code with the target group's matcher. A 200 from inside but
ResponseCodeMismatchoutside usually means a redirect or a host-header rule. - Check the network path: the instance's security group must allow the health check port from the load balancer's security group, and the subnet's network ACL must allow the return traffic.
- Check the zones: the load balancer must be enabled in every Availability Zone your group launches into.
- Only then touch timing: if the check passes eventually but too late, the grace period is your fix (next section).
To change the check itself, edit the target group, not the group:
aws elbv2 modify-target-group --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/shop-web/0123456789abcdef \ --health-check-path /health --matcher HttpCode=200-399 \ --health-check-interval-seconds 15 --healthy-threshold-count 3
Our ALB 502 vs 504 guide covers the load balancer side in more depth, including what users see while targets flap.
Jake: "Our curl from inside worked fine. Why did the load balancer still say unhealthy?"
Ethan: "Because it worked at minute four. The group checked at minute one and a half."
The health check grace period
What is the health check grace period? It is how long your group waits after an instance enters service before it acts on the optional health checks (ELB, VPC Lattice, EBS). If your instance needs four minutes to boot, run user data, pull code and start the app, and the grace period is shorter, the group sees "unhealthy" during start-up and replaces an instance that would have been fine. It applies to newly launched instances, instances coming back from Standby, and instances you attach by hand.
Two details catch people out:
- The default depends on how you created the group. When you create an Auto Scaling group in the console, the grace period starts at 300 seconds. When you create it with the AWS CLI or an SDK (and many infrastructure-as-code templates that leave it out), it starts at 0, and 0 turns the grace period off. A slow-starting app plus a group created from a script is the classic loop.
- The grace period does not protect a stopped instance. If the group sees an instance leave the
runningstate during the grace period, it marks it Unhealthy at once and replaces it.
The load balancer also needs time after your app is up. With the usual target group defaults (a check every 30 seconds, 5 healthy checks in a row to pass), a target needs about two and a half minutes of good answers before it counts as healthy. So your grace period should cover both: start-up time plus the time to pass the healthy threshold. Measure it instead of guessing:
- Launch one instance by hand from the same launch template, outside the group.
- Note the time it reaches
running. - Poll the health check path from another instance in the VPC until it returns your success code; note that time.
- Add the healthy threshold multiplied by the check interval (for example 3 × 15 seconds).
- Add a safety margin of about 30 percent, and set the grace period to the total.
aws autoscaling update-auto-scaling-group --auto-scaling-group-name shop-web \ --health-check-type ELB --health-check-grace-period 420
Setting it far too high has a cost of its own: a genuinely broken instance stays in service, taking traffic, until the grace period runs out. If your start-up time varies a lot, a launch lifecycle hook is the cleaner answer. The instance waits in Pending:Wait until your bootstrap script calls complete-lifecycle-action, so it only enters service when it is ready, and the grace period can then be short or even 0.
For the shop, measuring gave four minutes of start-up plus 45 seconds of checks. Raising the grace period from 90 to 420 seconds stopped the loop on the next launch.
Jake: "So the instances were fine, they were just being judged too early?"
Ethan: "Judged at ninety seconds for a four-minute warm-up. Of course they failed."
Windows instances need more time. A Windows AMI runs its launch agent, may apply settings on first boot and often restarts once before your app starts, so start-up of ten minutes or more is normal. Measure it the same way, and expect a grace period well above the console's 300-second default; a Windows group created from a script with a grace period of 0 will loop almost every time.
One more thing if you also run containers: an ECS service has its own ECS health check grace period (healthCheckGracePeriodSeconds), separate from the Auto Scaling group's. A task that is killed during start-up needs that one raised, not the group's; our ECS health check guide covers the task side.
Stop the loop long enough to debug
When instances are terminated before you can log in and look, pause the replacement. Each loop also costs you money: every replacement is a new instance-hour billed on your cloud hosting bill, plus a fresh EBS volume, so a group stuck cycling all night is not free.
- Suspend health checks and replacement:
aws autoscaling suspend-processes --auto-scaling-group-name shop-web --scaling-processes HealthCheck ReplaceUnhealthy. Unhealthy instances now stay running. - Connect to one with Session Manager or EC2 Instance Connect and check the app, its logs, and the user data log (
/var/log/cloud-init-output.logon Linux). - Or move one instance to Standby:
aws autoscaling enter-standby --instance-ids i-0123456789abcdef0 --auto-scaling-group-name shop-web --should-decrement-desired-capacity. It leaves the load balancer, and the group runs no health checks on it while you investigate. - Fix the cause (health check path, security group, grace period, launch template, user data).
- Resume:
aws autoscaling resume-processes --auto-scaling-group-name shop-web --scaling-processes HealthCheck ReplaceUnhealthy, and exit Standby or terminate the debug instance.
Forgetting step 5 is a real risk: with replacement suspended, a genuinely broken instance would stay in service and keep failing requests. Set a reminder. Note too that scale-in protection is the wrong tool here: it stops scale-in from choosing that instance, but a failed health check still gets it replaced. Our user data guide shows where boot scripts log their errors.
Jake: "Can I just stop the instance to look at its disk?"
Ethan: "Not while it's in the group. A stopped instance is an unhealthy instance. Standby first, then stop it."
Launch lifecycle hooks: let the instance say "ready"
Guessing a grace period works until start-up time varies: a slow package mirror one day, a cold cache the next. A launch lifecycle hook replaces the guess with a signal. The group puts each new instance in Pending:Wait, keeps it out of the load balancer and free of health checks, and only moves it into service when your bootstrap says it is ready.
Step 1: add the hook to the group:
aws autoscaling put-lifecycle-hook --auto-scaling-group-name shop-web \ --lifecycle-hook-name app-ready \ --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING \ --heartbeat-timeout 900 --default-result ABANDON
Step 2: signal from user data once the app answers its own health check. This end-of-script block uses IMDSv2 to find the instance ID:
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 300") ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id) for i in $(seq 1 60); do curl -sf http://localhost:8080/health && break sleep 10 done aws autoscaling complete-lifecycle-action --auto-scaling-group-name shop-web \ --lifecycle-hook-name app-ready --instance-id "$ID" \ --lifecycle-action-result CONTINUE --region us-east-1
How the pieces behave:
- The heartbeat timeout can be 30 to 7,200 seconds (default 3,600). If nothing signals in time, the default result applies.
ABANDONas the default result means an instance that never becomes ready is terminated and replaced, which is what you want for a broken boot.CONTINUEwould put a possibly broken instance into service.- Slow but healthy start-ups can call
record-lifecycle-action-heartbeatto reset the timer, up to an overall limit on time in the wait state. - Permissions: the instance profile needs
autoscaling:CompleteLifecycleAction(and the heartbeat action if you use it) on the group.
With the hook in place, the grace period only has to cover the load balancer's healthy threshold, because the app is already answering when the instance enters service. A failed hook shows clearly in the Activity history as an abandoned lifecycle action, which is a far clearer clue than a vague health check failure.
ASG instance refresh: what it is, and when it fails
What is an ASG instance refresh? It is the built-in way to roll a change (a new AMI, a new launch template version, a new instance type) across a running group without downtime. You start it with one command or one console button, and the group replaces every instance with one built from the current launch template, a slice at a time, waiting for each new batch to be healthy before moving on. In Terraform, the instance_refresh block on aws_autoscaling_group starts one automatically whenever the launch template changes, which is why a Terraform apply can quietly kick off replacements across your fleet. When the new configuration is broken, the refresh turns into a termination loop of its own: new instances launch, fail their health checks, and are replaced, while the refresh waits. Check its state and the reason:
aws autoscaling describe-instance-refreshes --auto-scaling-group-name shop-web \ --query "InstanceRefreshes[].[InstanceRefreshId,Status,StatusReason,PercentageComplete]" --output table
The preferences that decide how a refresh behaves:
MinHealthyPercentage: how much of the desired capacity must stay healthy during the refresh; 90 percent by default unless an instance maintenance policy says otherwise. A high value means small, slow batches.InstanceWarmup: how long a new instance must be healthy before it counts as done. Too short, and the refresh moves on before you know the new instances are good.AutoRollback: off by default. Turn it on and a failed refresh puts the previous configuration back, instead of leaving a half-replaced group.- Alarms: an alarm specification can fail the refresh (and roll it back) when, say, the 5xx rate rises during the rollout.
- Checkpoints: pause after a percentage of the group is replaced, so you can check the new instances before the rest follow.
aws autoscaling start-instance-refresh --auto-scaling-group-name shop-web \
--preferences '{"MinHealthyPercentage": 90, "InstanceWarmup": 300, "AutoRollback": true, "CheckpointPercentages": [20, 100], "CheckpointDelay": 600}'
To stop a bad rollout now, run cancel-instance-refresh; it stops new replacements but does not undo the ones already done, so fix or roll back the launch template next. If the refresh failed because the new instances never passed health checks, everything above about target health and grace periods applies to the new template version: test one instance from it by hand before you refresh again.
Jake: "The refresh has said twenty percent for an hour."
Ethan: "Then the new instances aren't getting healthy. Cancel it, launch one from the new template by hand, and watch it boot."
EC2 status check failures and boot problems
If the cause is an EC2 status check, the instance itself is unhealthy: it failed the system status check (a problem with the host underneath) or the instance status check (a problem with the instance's own configuration). The group tolerates a brief failure and waits a few minutes for a host problem to clear, but a status that stays impaired gets the instance replaced. Find out which check failed:
aws ec2 describe-instance-status --instance-ids i-0123456789abcdef0 --include-all-instances \ --query "InstanceStatuses[].[InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]" --output table
Then look at what the instance printed while it booted:
aws ec2 get-console-output --instance-id i-0123456789abcdef0 --latest --output text | tail -50
- A broken AMI (a bad
/etc/fstabentry, a kernel that will not boot on the new instance type, a full root disk) fails the instance status check every time. Launch one instance by hand, outside the group, from the same launch template, and watch it boot. - User data that shuts down or reboots the instance, or crashes before the app starts, looks like a health failure. A
shutdown -hat the end of a script copied from somewhere else is a common surprise. - Instance types the AMI does not support (for example missing network or storage drivers for newer Nitro types) can fail to boot or lose networking.
- A system status failure on one instance only is usually a host problem, and the replacement is the right result. If it hits every new instance, the cause is in your AMI or template, not the hardware.
Once a hand-launched instance boots cleanly and passes its checks, put that fixed configuration into a new launch template version and let the group use it. Then terminate your test instance so it does not keep billing you.
Network and permission problems that look like a dead app
Plenty of "the app never became healthy" loops are really the instance failing to fetch something it needs while starting. The app is fine; its start-up script is stuck waiting. Check these when the app works on a hand-launched instance in a public subnet but not inside the group:
- No route to the internet from a private subnet. User data that runs
dnf install,pip installor pulls code from GitHub needs a NAT gateway (or VPC endpoints and a private mirror). Without one, the script hangs until it times out, long after the grace period. - A missing or wrong instance profile. If start-up reads configuration from S3, Parameter Store or Secrets Manager, the role in the launch template must allow it. An access denied error in the user data log, and an app that never starts, is the signature.
- VPC endpoints without the right policy: an S3 gateway endpoint whose policy only allows certain buckets will block a download from any other bucket, silently, from inside the VPC.
- IMDSv2 required, old scripts using IMDSv1. If the launch template requires session tokens and a script calls the metadata service the old way, it gets a 401 and may never learn its own instance ID or Region. Update the script to fetch a token first, as in the lifecycle hook example above.
- A security group that allows users but not the load balancer. Opening port 80 to the internet does not help if the instance sits in a private subnet; the rule that matters is the health check port from the load balancer's security group.
The user data log (/var/log/cloud-init-output.log) usually shows exactly which command hung or was denied, which is why suspending replacement and reading it is the fastest route to the fix.
Only some instance types keep failing
Groups with a mixed instances policy run several instance types, and a loop that hits only one type has its own causes:
- Architecture mismatch: an x86 AMI cannot boot on Graviton (arm64) types, and the reverse; keep each launch template override on the right architecture, or use separate templates per architecture.
- Too little memory: an app that fits on the large type may be killed for running out of memory on a smaller type in the list. Check the system log for out-of-memory messages, and remove types below your app's real need.
- Drivers and features: older AMIs may lack the network or storage drivers that newer instance families need, so those types fail status checks while older ones pass.
The Activity history and describe-auto-scaling-instances show each instance's type next to its fate, so sort the terminations by type before you change anything else.
Launch failures that look like terminations
Sometimes the instance never really started. The Activity history shows a failed launch with a status message instead of a health cause, and the group keeps retrying, which can look like a stream of short-lived instances:
- vCPU quota exceeded: your account's On-Demand or Spot vCPU limit is reached. Our vCPU limit guide covers the increase request.
- InsufficientInstanceCapacity: EC2 has no capacity for that type in that zone right now. Use several instance types (a mixed instances policy) and several zones; our capacity guide explains the options.
- Client.InternalError: Client error on launch: almost always encrypted EBS volumes with a customer managed KMS key that the group's service-linked role cannot use. Add the key policy statements or a grant; our encrypted AMI guide has the exact commands.
- Invalid launch template parameters: a deleted security group, a subnet in the wrong VPC, an AMI that no longer exists or is not shared with your account, an instance profile the group is not allowed to pass.
Jake: "Our second group shows instances that last about ten seconds."
Ethan: "Ten seconds isn't a health check. That's a launch that never finished. Read the status message."
Scale-in and replacements that are working as designed
Not every termination is a problem. Before you change anything, check whether the group was told to do exactly this:
- Scaling policies: a target tracking policy on CPU scales in when load drops. If it scales in right after scaling out, your target is too tight or the metric too jumpy; widen the target and set a default instance warm-up.
- Scheduled actions that set desired capacity at certain times of day.
- Deployment tools (CodeDeploy, Terraform, CloudFormation) that change min, max or desired, or start instance refreshes. A pipeline that resets desired capacity on every run undoes any manual change you make.
- AZ rebalancing after a zone had capacity trouble: the group launches in the short zone and terminates in the long one.
- Spot interruptions, with their two-minute warning; our Spot guide covers handling them gracefully. More instance types in more zones, plus Capacity Rebalancing, make them rarer.
- Capacity Blocks: EC2 ends any instance still running 30 minutes before the block's end time, and the group then tries to replace it. Scale in yourself before that point.
- Maximum instance lifetime, which replaces instances older than a set age (at least one day) on purpose.
Which instance does scale-in pick? When a policy lowers desired capacity, the termination policy chooses the victim, and the default surprises people who expect "the newest one goes first". The default first picks the Availability Zone with the most instances (so the group stays balanced), then prefers instances on the oldest launch template or configuration, then those closest to the next billing hour, and finally picks at random. You can change it per group:
aws autoscaling update-auto-scaling-group --auto-scaling-group-name shop-web \ --termination-policies OldestLaunchTemplate OldestInstance
The built-in choices are Default, AllocationStrategy, OldestLaunchTemplate, OldestLaunchConfiguration, ClosestToNextInstanceHour, NewestInstance and OldestInstance, and you can also point the group at your own Lambda function when you need custom rules (for example, "never the instance running tonight's batch job"). For one instance that must survive scale-in, such as a node finishing long work, turn on instance scale-in protection with set-instance-protection --protected-from-scale-in, and turn it off when the work is done. Remember the limit from earlier: protection only affects scale-in, never health check replacement.
When many instances are unhealthy at once, the group does not wipe out the whole fleet: it replaces a slice at a time (10 percent of desired capacity by default, changeable with an instance maintenance policy), which gives you time to fix a bad health check before every instance is gone.
Make instances healthy faster
A shorter start-up shrinks the window in which any health check can catch your instance half-ready, and it makes scale-out faster when real traffic arrives:
- Bake more into the AMI: install packages and code at image build time (EC2 Image Builder or Packer) instead of in user data at every launch.
- A cheap, honest health endpoint:
/healthshould answer 200 as soon as the app can serve, without calling every downstream service. If your database blips, you want one error, not a fleet replaced. - Warm pools keep pre-initialized instances stopped, hibernated or running, ready to join the group in seconds. Two warm-pool details matter for terminations: an instance in the warm pool that fails its checks is replaced like any other, and with the instance reuse policy (
ReuseOnScaleIn) a scale-in sends instances back to the warm pool instead of terminating them, which makes the next scale-out faster still. - Lifecycle hooks make the group wait for your "ready" signal.
- Default instance warm-up stops new instances' metrics from skewing scaling decisions while they start.
The cost side matters too. A group that stabilizes in one minute instead of five can run fewer spare instances for the same safety, which shows up directly on your EC2 bill.
Get told the moment a loop starts
The shop found its loop because a person noticed. Next time, an alert should notice first. Auto Scaling sends events to EventBridge for every launch and termination, with the same cause text you see in the Activity tab:
{
"source": ["aws.autoscaling"],
"detail-type": ["EC2 Instance Terminate Successful", "EC2 Instance Launch Unsuccessful"],
"detail": {"AutoScalingGroupName": ["shop-web"]}
}
Send that rule to an SNS topic and every termination or failed launch reaches your inbox or chat with its cause. For a quieter signal, alarm on the rate instead of each event:
- Turn on group metrics:
aws autoscaling enable-metrics-collection --auto-scaling-group-name shop-web --granularity 1Minute. They are free and show in CloudWatch under the AutoScaling namespace. - Alarm on
GroupTerminatingInstancesstaying above zero for, say, 15 minutes outside your deployment windows; a healthy group rarely terminates continuously. - Alarm on
GroupInServiceInstancesfalling below desired capacity for more than a few minutes, which catches launch failures as well as health loops. - Alarm on the target group's
UnHealthyHostCount, which often rises before the group starts replacing instances.
Either way, include the group name and a link to its Activity tab in the alert text, so whoever gets paged starts in the right place.
Terraform, CloudFormation and CDK settings behind loops
Many loops come from a template, not a click, because a value left out of the code takes a default nobody chose:
- Terraform:
aws_autoscaling_groupsetshealth_check_grace_periodto 300 seconds if you leave it out, andhealth_check_typeto EC2. Switching to"ELB"without measuring start-up is the usual trap. Terraform also waits for healthy capacity (wait_for_capacity_timeout, 10 minutes by default), so a looping group shows up as an apply that fails with "timeout while waiting for state to become 'ok'". That message is the loop, seen from Terraform. - CloudFormation: set
HealthCheckTypeandHealthCheckGracePeriodexplicitly onAWS::AutoScaling::AutoScalingGrouprather than trusting defaults, and if you use a creation policy withcfn-signal, make sure the signal comes after the app is healthy, not after user data merely starts. - AWS CDK: the health check settings on the group construct are easy to miss in a long stack; search your code for where the ELB health check is enabled and check the grace period sits next to it.
- Every tool: if your pipeline sets desired capacity on each deploy, it will undo manual scale changes and can start replacements you did not expect; let scaling policies own desired capacity, and leave it out of the template after the first deploy where your tool allows it.
The habit that prevents all of these: write the health check type and grace period into the template on purpose, with a comment saying how the number was measured.
Frequently asked questions
Why does my Auto Scaling group keep terminating instances?
Usually because they fail the ELB or EC2 health check, often during start-up when the grace period is too short. The Activity tab names the exact cause.
How do I see why an Auto Scaling group terminated an instance?
Open the group's Activity tab, or run aws autoscaling describe-scaling-activities and read the Cause field.
What is an Auto Scaling group in AWS?
A set of EC2 instances kept at a desired count between a minimum and a maximum, launched from one launch template and replaced automatically when unhealthy.
What is the health check grace period?
The time the group waits after an instance enters service before acting on ELB, VPC Lattice or EBS health checks. Set it longer than start-up plus the time to pass the healthy threshold.
What is the default ASG health check grace period?
300 seconds for groups created in the console, and 0 (off) for groups created with the AWS CLI or an SDK.
What does "ELB system health check failure" mean?
The load balancer's target group reported the instance unhealthy; check describe-target-health for the reason code.
How do I stop Auto Scaling from replacing unhealthy instances while I debug?
Suspend the HealthCheck and ReplaceUnhealthy processes, or move an instance to Standby; resume when you are done.
What does Target.Timeout mean on a target group?
The load balancer could not reach the instance on the health check port, usually because of a security group or the app not listening.
What does Target.ResponseCodeMismatch mean?
The app answered with an unexpected status code, such as a redirect or 404; fix the path or the success codes.
Should my Auto Scaling group use EC2 or ELB health checks?
Behind a load balancer, use ELB: EC2 checks only catch instance-level failures, not a dead application. Pair it with a measured grace period.
Why do my instances launch and terminate immediately?
Either a failed launch (quota, capacity, KMS, invalid template) or a health check failure during start-up. Seconds points to a launch failure; minutes to a health check.
What does Client.InternalError mean in an Auto Scaling group?
The launch failed, usually because encrypted volumes use a KMS key the Auto Scaling service-linked role cannot use.
Does scale-in protection stop health check replacements?
No. It only keeps scale-in from choosing that instance. Use Standby or suspend ReplaceUnhealthy to keep an unhealthy instance.
Why is my Auto Scaling group scaling in right after scaling out?
A scaling policy's target is too tight or the metric jumps; widen the target and set a default instance warm-up.
Can I stop or reboot an instance in an Auto Scaling group?
Put it in Standby first. A stopped instance is treated as unhealthy and replaced, even during the grace period.
What is AZ rebalancing?
The group launches and terminates instances to even them out across Availability Zones; it is expected, and can be suspended.
How do I check the boot output of a terminated instance?
Run aws ec2 get-console-output with the instance ID and --latest soon after it stops, or suspend replacement and inspect a live one.
How is the ECS health check grace period different?
It is a setting on the ECS service (healthCheckGracePeriodSeconds) that protects tasks while they start; the Auto Scaling group's grace period protects instances.
What is a launch lifecycle hook?
A hook that holds each new instance in Pending:Wait until your bootstrap calls complete-lifecycle-action, so it only enters service when it is ready.
What is an ASG instance refresh?
A rolling replacement of every instance in an Auto Scaling group with instances from the current launch template, in batches that must be healthy before the next starts.
Can I disable health checks on an Auto Scaling group?
EC2 status checks are always on; you can switch the type back to EC2 to ignore the load balancer, or suspend the HealthCheck process while you debug.
How do I write a custom health check for an Auto Scaling group?
Run your own check and call aws autoscaling set-instance-health --health-status Unhealthy for a failing instance; the group then replaces it.
Why is my instance refresh stuck or failing?
New instances from the updated launch template are not passing health checks; check describe-instance-refreshes for the status reason and test the template by hand.
Which instance does Auto Scaling terminate first on scale-in?
With the default policy, one in the zone with the most instances, preferring the oldest launch template, then the one closest to the next billing hour.
How do I get alerted when Auto Scaling terminates instances?
Create an EventBridge rule for EC2 Instance Terminate Successful events from aws.autoscaling and send it to SNS, or alarm on group metrics.
Why does terraform apply time out waiting for my Auto Scaling group?
Terraform waits for healthy capacity; if instances keep failing health checks, the wait times out. Fix the health check, then apply again.
How long should the grace period be for Windows instances?
Measure it, but expect ten minutes or more, since Windows first boot runs its launch agent and often restarts before your app starts.
Why do instances fail health checks only in a private subnet?
Start-up usually needs the internet or AWS APIs; without a NAT gateway or the right VPC endpoints, user data hangs and the app never starts.
Why does only one instance type in my group keep failing?
Often an architecture mismatch (x86 AMI on Graviton), too little memory on a smaller type, or missing drivers for a newer instance family.
How do I change the Auto Scaling termination policy?
Run update-auto-scaling-group with --termination-policies, for example OldestLaunchTemplate OldestInstance, or point it at a custom Lambda policy.
Do Spot interruptions show in the Activity history?
Yes, as instances terminated by an EC2 health check after Spot reclaimed them; more instance types and Capacity Rebalancing make them rarer.
For readers preparing for an AWS certification
We have been covering AWS certifications alongside our fixes, so here is a short extra for readers studying for one. If you only came for the fix, you already have it; feel free to skip this part.
Exams that test this topic: CloudOps Engineer – Associate (SOA-C03) • Solutions Architect – Associate (SAA-C03) • Developer – Associate (DVA-C03 from December 1, 2026).
Here are the exam-style questions this topic tends to raise, with the reasoning you need to answer them:
1. New instances in an Auto Scaling group behind an ALB are terminated a few minutes after launch, before the app finishes starting. What should you change?
Increase the health check grace period to cover start-up plus the healthy checks (or use a launch lifecycle hook).
2. Where do you find the reason an Auto Scaling group terminated an instance?
The group's Activity history (describe-scaling-activities), in the Cause field.
3. An Auto Scaling group uses EC2 health checks only. The web app crashes but the instance keeps running. What happens?
Nothing; EC2 checks only see the instance. Turn on ELB health checks to replace instances whose app fails.
4. You need to troubleshoot an unhealthy instance without the group replacing it. Which two options work?
Suspend the HealthCheck and ReplaceUnhealthy processes, or put the instance in Standby.
5. Does instance scale-in protection stop an unhealthy instance from being replaced?
No; it only protects the instance from scale-in.
6. Instances fail to launch with Client.InternalError: Client error on launch. What is the usual cause?
Encrypted EBS volumes using a KMS key the Auto Scaling service-linked role is not allowed to use.
7. What is the default health check grace period for a group created with the AWS CLI?
0 seconds (off); the console default is 300 seconds.
Exam codes and dates change often. Our AWS certification changes page keeps track of what is current.
The web servers have been stable since the grace period change. Ethan also moved the code download out of user data and into the AMI, which cut start-up from four minutes to seventy seconds, and added a /health page that answers as soon as the app is ready. Jake's takeaway, written on the team board: "Before blaming the app, open the Activity tab. It already knows why."
📌 If you keep one line from this page
Read your Auto Scaling group's Activity history first: an ELB health check failure means fix the target health and make the grace period longer than start-up plus the healthy checks; a failed launch means quota, capacity, KMS or template; and suspend ReplaceUnhealthy while you debug.
describe-target-health for reasons · Standby before you stop an instance · bake the AMI to start faster.
Revision note. Written October 19, 2026. An Auto Scaling group never terminates without writing down why; the fix starts with reading it.