Handle AWS EC2 Spot Instance Termination: Build a 2-Minute Warning Pipeline
A terminated spot instance almost always means Amazon EC2 already sent two signals before it pulled the plug: an instance rebalance recommendation, then a Spot Instance interruption notice exactly two minutes before the stop or terminate happened. If nothing in your stack was listening for either one, the "warning" ran and disappeared into a server nobody was watching. The fix isn't a bigger budget — it's a pipeline: something polling the instance for the notice, an EventBridge rule catching it fleet-wide, and an Auto Scaling group configured to replace the instance before it dies instead of after.
Here's the part almost nobody expects: raising your maximum price does not protect you from this. Amazon EC2 states plainly that a Spot Instance can be reclaimed even when your max price is well above the current Spot price — because most interruptions aren't about price at all. They're about AWS needing the physical capacity back, hitting a host maintenance event, or being unable to satisfy a placement constraint you set. You can be willing to pay double and still get the two-minute notice.
Jake found out about spot terminations the annoying way. He runs a small mobile phone shop, and on the side he rents out a couple of spot instances to run overnight photo-backup jobs for customers who trade in old phones. One Tuesday morning, half a customer's photo library was missing. The instance that was supposed to finish uploading it to storage had simply vanished at 2:14 AM. No error in his script. No crash log. It just stopped existing.
What actually happens when a spot instance gets terminated
A spot instance is a regular EC2 instance — same hardware, same operating system, same everything — with one condition attached: Amazon EC2 can take it back. It's spare capacity that AWS isn't using for on-demand customers right now, sold to you at a discount, with the understanding that if that capacity is needed elsewhere, you lose the instance. That's the entire deal. While it's running, a spot instance behaves identically to an on-demand one. The difference only shows up the moment AWS decides it wants the hardware back.
When that moment comes, Amazon EC2 doesn't just vanish the instance. It issues a Spot Instance interruption notice — a warning, two minutes ahead of the stop, hibernate, or terminate action — so your application has a chance to react. That notice is delivered two ways at once: as an event sent to Amazon EventBridge (AWS's event bus, which is basically a switchboard that watches for specific things happening in your account and can trigger an action when they do), and as an entry in the instance metadata service running locally on the instance itself. AWS recommends checking for these notices every 5 seconds, because they're emitted on a best-effort basis — meaning there's no contractual guarantee one always arrives, though in practice they reliably do.
♂️ Jake's Reality Check
"So there was a warning? Where was it? Nothing on my screen said anything."
It went to two places you weren't watching. The warning was sitting in the instance's own metadata service and, separately, fired off as an event to EventBridge — but nothing on your instance was polling for it, and you hadn't wired up an EventBridge rule to catch it. AWS delivered the message. Nobody picked up the phone.
Here's the piece that trips up almost everyone building on spot for the first time: there's actually a second signal that can arrive before the two-minute notice, called an instance rebalance recommendation. It's not a countdown — it's a risk flag. AWS emits it when a spot instance is at elevated risk of interruption, giving you a window to move your workload before the hard two-minute clock even starts. Auto Scaling groups, EC2 Fleet, and Spot Fleet can all listen for this signal and proactively launch a replacement while your original instance is still healthy and running.
What changed: the recommended allocation strategy
- Before: the default and commonly-used strategy for Spot Fleet and EC2 Fleet was
lowest-price— always launch in whichever pool was cheapest at that instant, capacity risk or not. - Now: AWS's documented best practice is
price-capacity-optimized, which launches from pools that are both well-stocked with capacity and reasonably priced, specifically to reduce the odds of getting reclaimed soon after launch. - What that means for you: if your Auto Scaling group or Fleet is still set to
lowest-price, your replacement instances after a rebalance can land in a pool that's about to be reclaimed again — replacing an at-risk instance with another at-risk instance.
Jake's overnight job had neither of these signals wired up. No metadata polling, no EventBridge rule, and the instance was a lone spot instance he'd launched by hand — not inside an Auto Scaling group at all, so there was nothing watching for a rebalance recommendation either. AWS did its part. Jake's pipeline didn't exist.
The two signals: rebalance recommendation vs. interruption notice
These two signals get confused constantly because they sound like the same thing described twice. They're not. One is a warning about risk; the other is a countdown to action. Mixing them up is why a lot of "spot interruption handler" scripts online only listen for one and miss the other.
| Signal | What it means | EventBridge detail-type | Instance metadata path |
|---|---|---|---|
| Instance rebalance recommendation | Elevated risk of interruption. Not a deadline — a heads-up that can arrive before the two-minute clock starts. | EC2 Instance Rebalance Recommendation | /latest/meta-data/events/recommendations/rebalance |
| Spot Instance interruption notice | Hard deadline. Two minutes until stop, hibernate, or terminate (hibernate has no two-minute lead — it begins immediately). | EC2 Spot Instance Interruption Warning | /latest/meta-data/spot/instance-action |
AWS is upfront that the ordering isn't guaranteed. It is not always possible for Amazon EC2 to send the rebalance recommendation before the two-minute interruption notice — sometimes they land together, which means your "early warning" buys you zero extra time on that particular instance. That's not a bug in your setup; it's just how the underlying capacity signal works. A pipeline built only around the rebalance recommendation, assuming it'll always give you a head start, will occasionally get surprised.
♂️ Jake's Reality Check
"Okay but rebalance recommendations only apply to newer instances or something, right? What if my instance is old?"
Only a launch-date cutoff, not an age limit. Rebalance recommendations are supported for spot instances launched after November 5, 2020 at 00:00 UTC. If your fleet is running anything launched since then — which, at this point, is almost every spot instance in existence — you're covered.
Why your instance specifically got picked
Before you build anything, it's worth knowing what actually causes an interruption, because the popular assumption — "the spot price went above my max bid" — is the least common reason in most fleets today. AWS's own troubleshooting guidance lists the real causes plainly:
| Cause | What it looks like |
|---|---|
| Not enough capacity | Amazon EC2 reclaims the instance to repurpose capacity for other customers, regardless of your max price. |
| Host maintenance or hardware decommission | The physical host your instance sits on needs to be pulled for maintenance or retired. |
| Constraints can't be met | A launch group or Availability Zone constraint you set can no longer be satisfied. |
| Price actually did exceed your max | Rarer than people assume, but it happens — the request status reads instance-terminated-by-price. |
To see which of these applied to your dead instance, don't guess — check the record. In the EC2 console, open Spot Requests, find the request tied to your instance, and read the status description under Status. It'll show a specific reason code, and that code tells you exactly what happened: instance-terminated-no-capacity means AWS needed the hardware, instance-terminated-by-price means your max bid genuinely got beaten, and so on down a documented list of codes.
The full list of status codes, in plain English
Here are the ones you'll actually run into on a terminated or stopped spot instance, straight from the request status codes AWS documents:
- capacity-not-available — there isn't enough spare capacity for what you asked for, full stop.
- instance-terminated-by-price — the spot price rose above your maximum. If your request is persistent, it'll re-evaluate and try again automatically.
- instance-terminated-by-schedule — you set a fixed duration and it ran out.
- instance-terminated-launch-group-constraint — another instance in your launch group was terminated, breaking the "all together" requirement.
- instance-stopped-no-capacity / instance-terminated-no-capacity — pure capacity reclaim. This is the one you'll see most often on modern fleets.
- instance-stopped-by-user / instance-terminated-by-user — you (or a script acting as you) did it, not AWS.
Ethan's take, when Jake asked why his max price hadn't saved him: "You were thinking of it like an auction where the highest bidder always keeps the item. It's closer to renting a spare room — the landlord can still ask for it back if their kid moves home, and it doesn't matter how much rent you offered. Price is only one lever AWS pulls. Physical capacity is the other, and it's the one nobody can outbid."
Building the pipeline, step 1: catch the signal on the instance itself
This is the layer that runs inside the instance, watching its own local metadata service — a small internal web server every EC2 instance can reach at the fixed address 169.254.169.254, which only the instance itself can talk to. It's how an instance can ask AWS questions about itself ("what's my instance ID," "am I about to be interrupted") without needing any AWS credentials.
Modern EC2 instances use IMDSv2 (Instance Metadata Service version 2), which requires a short-lived security token before you can read anything from it — a small extra step designed to stop a class of web-request-forgery attacks that could otherwise trick a running application into leaking metadata. Here's how to grab that token and then check for a pending interruption, from a Linux instance:
- Request a session token, valid for the number of seconds you specify:
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` - Use that token to ask whether a stop or terminate action is pending:
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/spot/instance-action - If nothing is pending, you get an HTTP 404 — that's normal and expected almost all the time. If an interruption is coming, you get a small JSON reply like
{"action": "terminate", "time": "2026-09-21T02:14:00Z"}telling you exactly what's about to happen and when. - Check the rebalance recommendation path the same way, since it's a separate item:
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/events/recommendations/rebalance - Loop steps 2 and 4 every 5 seconds — AWS's own recommended polling interval — for as long as the instance is up, and trigger your shutdown routine the moment either one returns something other than a 404.
The action item is deliberately explicit about what's coming: the possible values are hibernate, stop, or terminate, and knowing which one applies changes what you should actually do. A stop means the instance's EBS root volume survives and can be restarted later — your local disk state is intact. A terminate means any instance store volumes are gone and any EBS volumes marked delete-on-termination go with the instance. Building a shutdown script that treats all three the same way is a common and avoidable mistake — check the action value before deciding whether to flush a cache to disk or ship it somewhere durable.
⚠️ What this actually breaks
There's an older metadata item called termination-time, kept around only for backward compatibility. Some very old tutorials still reference it. AWS's own documentation now tells you to use instance-action instead — termination-time doesn't tell you whether the outcome is a stop, hibernate, or terminate, which is exactly the distinction that matters for deciding what to save.
Building the pipeline, step 2: catch it fleet-wide with EventBridge
Polling metadata works great for one instance reacting to its own fate. But if you're running twenty spot instances and want a single place watching all of them — sending a Slack alert, kicking off a Lambda function, updating a dashboard — that's a job for Amazon EventBridge, not for twenty separate polling loops each acting alone.
Both signals are published as EventBridge events with a matching, memorable shape. Here's the interruption warning event, exactly as AWS documents it:
{
"version": "0",
"id": "12345678-1234-1234-1234-123456789012",
"detail-type": "EC2 Spot Instance Interruption Warning",
"source": "aws.ec2",
"account": "123456789012",
"time": "2026-09-21T02:14:00Z",
"region": "us-east-2",
"resources": ["arn:aws:ec2:us-east-2a:instance/i-1234567890abcdef0"],
"detail": {
"instance-id": "i-1234567890abcdef0",
"instance-action": "terminate"
}
}
Note that resources ARN in there looks a little unusual if you're used to reading EC2 ARNs — it's formatted as arn:aws:ec2:availability-zone:instance/instance-id, which differs from the standard EC2 resource ARN format used elsewhere in IAM policies. Don't reuse this exact string in an IAM policy expecting the normal format; it won't match.
Setting up the rule in the console:
- Open the Amazon EventBridge console and choose Create rule.
- Give it a name, choose the default event bus (this is where events from AWS services in your account always land), and pick Rule with an event pattern.
- For the event pattern, either build it through the form (Event source: AWS services, AWS Service: EC2 Spot Fleet, Event type: whichever of the two signals you're catching) or paste the pattern directly:
{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning"]}— swap in"EC2 Instance Rebalance Recommendation"for the earlier signal. - For the target, choose an AWS service — an SNS topic if you just want a notification (email, text, or mobile push, once you've created the topic in the SNS console), or a Lambda function if you want automated action like deregistering the instance from a load balancer.
- Review and create. From this point on, every matching event in the account fires the rule — no per-instance setup required.
✅ Why this is the one to use for fleets
Metadata polling only tells an instance about its own future — it has no idea what's happening to its neighbors. EventBridge gives you one rule and one place to look, no matter how many spot instances you're running today or add next month. For anything beyond a single lone instance, build the EventBridge rule first and treat per-instance metadata polling as the local backup, not the other way around.
Building the pipeline, step 3: let Auto Scaling replace the instance before it dies
Everything above is about noticing an interruption is coming. This step is about not needing to care as much when it does — by having Auto Scaling launch the replacement before the old instance is even gone, instead of scrambling to launch one after.
The feature is called Capacity Rebalancing, and it's an on/off setting on your Auto Scaling group. Without it, an Auto Scaling group doesn't replace a spot instance until after EC2 has already interrupted it and its health check has failed — meaning you're capacity-short for however long it takes the replacement to launch and pass health checks. With Capacity Rebalancing turned on, Auto Scaling watches for the rebalance recommendation signal and starts launching a replacement the moment that signal arrives, while the at-risk instance is still up and doing work.
The sequencing matters here and it's worth being precise about it: Auto Scaling waits for the new instance to pass its health check before terminating the old one. That can temporarily push you over your configured maximum group size — AWS explicitly allows the group to exceed its max by up to 10 percent of desired capacity during a rebalance, specifically so this launch-before-terminate order doesn't get stuck when you're already near your ceiling.
The three-step setup
| Step | Why it's required |
|---|---|
| Configure multiple instance types and Availability Zones | Gives Auto Scaling somewhere to put the replacement. A group locked to one instance type in one AZ has nowhere to rebalance to. |
| Add lifecycle hooks if you need graceful shutdown time | Lets the old instance finish draining SQS work, deregister from DNS, or ship logs to S3 before termination — but the custom action needs to finish inside two minutes, since that's still the hard outer limit. |
| Turn on Capacity Rebalancing itself | The setting that actually makes the group act on the rebalance recommendation instead of waiting for the interruption notice. |
There's one honest limitation worth knowing before you turn this on: Auto Scaling will only launch a replacement if the new instance offers the same or better availability than the one it's replacing. If every available pool right now looks just as risky, Auto Scaling won't launch a swap purely for the sake of swapping — it keeps watching and launches the moment a better pool shows up. If your original instance gets interrupted before that happens, Auto Scaling reacts the instant the interruption notice lands regardless of how risky the new pool looks, because at that point doing something beats doing nothing.
♂️ Jake's Reality Check
"Doesn't turning this on mean I get interrupted more often? Feels like I'd be poking the bear."
No — the interruption rate stays the same, but you get replaced more often, which is different and better. Capacity Rebalancing doesn't change how frequently AWS reclaims spot capacity. It changes how often Auto Scaling proactively swaps an at-risk instance for a healthier one before the reclaim happens. You'll see more replacement activity in your logs, but fewer moments where you're actually short on running capacity.
What to actually do inside the two-minute window
Detecting the notice is only half the job — the other half is having something worth doing with those 120 seconds. AWS's own guidance lists three categories of action worth building into your shutdown routine, and they layer on top of each other rather than replacing one another:
- Graceful shutdown. Finish or checkpoint in-flight work, upload logs to S3, shut down queue workers cleanly, complete deregistration from DNS so nothing keeps routing traffic to a host that's about to disappear.
- Stop scheduling new work. Even if you can't finish everything, you can keep the instance from picking up brand-new jobs it has no chance of completing — this alone prevents a lot of the "half-processed" mess that causes real data problems later.
- Proactively launch a replacement. This is what Capacity Rebalancing automates for you at the fleet level, but if you're running a lone instance outside an Auto Scaling group — like Jake was — this step has to be scripted by hand or it simply doesn't happen.
If your instance sits behind a load balancer, there's a subtlety that catches people out: Auto Scaling waits for the instance to fully deregister from the load balancer before it even calls your lifecycle hook. If deregistration is slow — a long connection-draining timeout, for instance — you can burn through a meaningful chunk of your two minutes before your custom shutdown code has even started running. Keep connection draining timeouts short on instances you expect to run on spot, or budget for it explicitly in your shutdown script's timing.
The hard cases: when the pipeline still doesn't save you
Every piece described so far is well-documented and works as advertised. But none of it turns spot into on-demand, and being upfront about where it falls short matters more than pretending it's bulletproof.
The signal arrives together, not ahead of time
As covered earlier, the rebalance recommendation and the two-minute interruption notice can land at the same moment. When that happens, Capacity Rebalancing still tries to launch a replacement immediately — but "immediately" isn't "before," and your old instance is still on a two-minute clock regardless. Your shutdown script needs to be able to complete its critical work inside two minutes on its own, without depending on the replacement having launched first.
The replacement can't find anywhere better to land
If every pool your Auto Scaling group is configured to use is equally squeezed for capacity, there may be nowhere for it to rebalance to. This is exactly why AWS's flexibility advice exists: a good rule of thumb is being flexible across at least 10 instance types per workload, and making sure every Availability Zone in your VPC is actually enabled for use, not just the two you happened to pick when you first set things up.
You picked the wrong allocation strategy
Using lowest-price as your allocation strategy — still the default in some legacy setups and in raw CreateFleet calls unless you override it — actively works against you here. AWS documents this directly: it launches instances in whichever pool is cheapest that instant, even when that pool is likely to get reclaimed again soon after. The documented fix is switching to price-capacity-optimized, which factors capacity depth into the decision, not just price.
You're running one instance, not a fleet
Every proactive-replacement feature described above — Capacity Rebalancing, Auto Scaling's health-check-then-terminate ordering — depends on being inside an Auto Scaling group or Fleet. A single spot instance launched with a plain RunInstances call has none of that machinery underneath it. It can still poll metadata and react locally, but there's no service watching its back to launch a replacement for it. This is precisely Jake's situation, and it's a legitimate reason to consider whether a lone spot instance is the right tool at all for something whose failure has a real cost attached.
⚠️ What this actually breaks
AWS's own guidance is direct about this: spot instances are not recommended for workloads that are inflexible, stateful, fault-intolerant, or tightly coupled between nodes. AWS also specifically discourages failing over to on-demand instances as your interruption-handling strategy, because that failover can itself drive further interruptions elsewhere, and if a particular instance-type-and-AZ combination just got reclaimed from you, on-demand capacity in that same combination may be hard to get right after.
Reducing how often this happens in the first place
Handling the interruption well is one lever. The other is simply getting interrupted less often, and that comes down to how narrowly you've defined what counts as "acceptable capacity."
A spot capacity pool is a specific combination of instance type and Availability Zone — for example, m5.large in us-east-1a is one pool, and m5.large in us-east-1b is a completely different one with its own independent supply and demand. If your fleet is only configured to use one or two pools, you're betting everything on those two pools staying healthy. AWS's guidance is to not just ask for a single instance type — if you'd be willing to run on larges from several different instance families, say so, because that gives the allocation strategy far more places to find capacity for you. AWS also notes that choosing earlier-generation instance types tends to result in fewer interruptions, simply because they're in lower demand from on-demand customers chasing the newest hardware.
Two features make this easier than manually researching every pool yourself:
- Attribute-based instance type selection — instead of naming specific instance types, you specify requirements like vCPUs, memory, and storage, and Auto Scaling or EC2 Fleet figures out which current and future instance types match. This also means you automatically pick up newly released instance types without updating your configuration by hand.
- Spot placement score — a point-in-time score from 1 to 10 for a Region or Availability Zone, indicating how likely you are to successfully get the spot capacity you're asking for there. It's a recommendation, not a guarantee, and it doesn't predict future interruption risk — it's a snapshot of right now, useful for deciding where to launch, not a crystal ball for how long you'll keep the instance.
The Spot max price decision AWS wants you to skip
Most tutorials treat setting a maximum price as step one of launching a spot instance. AWS's own documentation for mixed instances groups says the opposite, in almost exactly these words: we strongly recommend that you do not specify a maximum price at all. That's a specific, documented recommendation, not a vague suggestion — and it's worth sitting with, because it runs against nearly everything the word "bidding" implies.
The reasoning is concrete. If you don't specify a maximum price, Amazon EC2 defaults it to the on-demand price for that instance type — which is effectively no meaningful ceiling, since spot pricing sits well below on-demand by design under the current pricing model. You still only pay the actual spot price for whatever you launch; you never pay more just because the ceiling is set higher. Setting your own lower maximum, on the other hand, only adds a way for your request to go unfulfilled — your application simply might not run at all if you don't receive any instances, which is exactly what happens when your self-imposed maximum sits below the current price.
♂️ Jake's Reality Check
"So the 'safe' move — capping what I'm willing to pay — is actually the riskier one?"
For most workloads, yes. A cap you set yourself can only ever cost you availability, not money, since you're already paying the live spot price either way. AWS's documented advice is to leave the field blank and let the default on-demand ceiling do its job, unless you have a very specific reason to want a hard stop below that.
Building a real safety net: on-demand base capacity in a mixed instances group
Everything so far reduces how often you get interrupted and how gracefully you handle it when you do. None of it gives you a guaranteed floor. For that, Auto Scaling has a separate, purpose-built feature: a mixed instances group with an on-demand base capacity.
| Desired capacity | On-demand base (12) | Spot, above the base |
|---|---|---|
| 10 | 10 | 0 |
| 20 | 12 | 8 |
| 40 | 12 | 28 |
The idea is simple once you see the two settings involved. You designate a base number of on-demand instances to launch first — a fixed, non-negotiable floor. Only once that base is satisfied does Auto Scaling start filling the rest of desired capacity according to your chosen on-demand-versus-spot percentage split above that base. AWS's own worked example uses a base of 12: at that floor, the group keeps exactly 12 on-demand instances running no matter how large it scales, filling everything else with spot — so at a desired capacity of 40, you'd have 12 on-demand and 28 spot, and the 12 never disappear no matter how badly spot capacity dries up.
This is the mechanism Ethan would point Jake toward for anything that isn't purely disposable batch work: keep a small, deliberately-sized on-demand base that can absorb the worst day, and let everything above it enjoy the spot discount. It's a meaningfully different design decision from "fail over to on-demand when spot lets me down," which AWS discourages — a fixed base is planned and steady, while a reactive failover is a scramble that can itself worsen availability for your other spot instances in the same pools.
Terminate, stop, or hibernate: pick the right interruption behavior
Terminate is the default behavior when you request a spot instance, but it isn't the only option, and for a "time-flexible" workload — one that doesn't mind picking back up later rather than finishing right now — a different choice can matter a lot.
| Behavior | Gets the two-minute warning? | Notes |
|---|---|---|
| Terminate | Yes | Default. Instance store contents are lost; EBS volumes follow their delete-on-termination setting. |
| Stop | Yes | Request must be persistent (for Fleet or Spot Fleet, type must be "maintain"), and you can't use a launch group. Only Amazon EC2 can restart it once stopped. |
| Hibernate | No | The interruption notice is issued, but not two minutes ahead, because the hibernation process begins right away. |
Stop and hibernate both mean Amazon EC2 automatically resumes your instance later, once capacity is available again — you don't have to relaunch it yourself. That's a meaningful difference from terminate, where the instance is simply gone and any replacement is a brand-new launch, not a resumption of the old one.
Testing your pipeline without waiting for a real interruption
You shouldn't have to wait for AWS to reclaim real capacity to find out whether your handler actually works. AWS Fault Injection Service (FIS) — a service built specifically for deliberately triggering failure scenarios so you can see how your systems react — lets you initiate a real Spot Instance interruption on demand from the EC2 console, by selecting a Spot Instance request or Spot Fleet request and choosing to interrupt it.
When you trigger this, the sequence is identical to a real interruption: the instance receives an instance rebalance recommendation, then a Spot Instance interruption notice two minutes before the actual interruption, and then, after two minutes, it's genuinely stopped or terminated depending on what you configured. This is the honest way to find out whether your EventBridge rule fires, whether your metadata poller catches it in time, and whether your Auto Scaling group actually launches a healthy replacement — instead of hoping it'll all line up the first time it matters for real.
A note on what this article can and can't tell you: none of the steps above were run against a live AWS account as part of writing this — they're described exactly as AWS documents them, not as something "we tested and confirmed." If you're building this pipeline for something that genuinely can't afford to fail, running it through AWS FIS yourself, on your own account and your own workload, is the only way to actually know it holds up.
Running this on EKS or ECS: the container-native version
Everything above assumes you're reacting at the instance level. If your spot instances are actually Kubernetes worker nodes, reacting at the instance level isn't enough on its own — you also need the Kubernetes scheduler to stop placing pods on a node that's about to disappear, and to move the pods that are already there somewhere safer, before the two minutes run out.
The tool built for exactly this is AWS Node Termination Handler, an AWS open-source project for Kubernetes clusters running on EC2, including self-managed clusters and ones created with Amazon EKS. It runs in one of two modes. In Instance Metadata Service mode, a small pod runs on each host and monitors the same metadata paths described earlier in this piece — the spot and events paths — reacting by cordoning the node, which marks it as unschedulable so no new pods land there, and then draining it, which safely evicts the pods that are already running so they get rescheduled elsewhere before the node goes away. In Queue Processor mode, instead of polling from inside each node, the handler watches an Amazon SQS queue fed by EventBridge for the same underlying events — spot interruption notices and rebalance recommendations, alongside Auto Scaling group lifecycle events and EC2 status changes — and reacts centrally rather than per-node.
Which mode to use maps almost exactly onto the instance-versus-fleet distinction from earlier in this piece: Instance Metadata Service mode is the Kubernetes equivalent of the per-instance polling script, reacting locally and needing no extra AWS permissions beyond what's already on the node. Queue Processor mode is the Kubernetes equivalent of the EventBridge rule, reacting centrally across the whole cluster, but it needs IAM permissions to read and manage the SQS queue and to query the EC2 API. For a cluster of any real size, Queue Processor mode is the one that scales the way EventBridge scales for plain EC2 fleets — one place watching everything, rather than each node handling its own fate in isolation.
Popular advice that's wrong, and why
"Just switch to on-demand the moment a spot instance gets interrupted, as your fallback." AWS specifically discourages this as a strategy, not because it can't be done, but because it can make things worse for you: failing over to on-demand can drive interruptions for your other spot instances, and if the exact instance-type-and-AZ combination you got reclaimed from is tight on capacity, getting an on-demand instance in that same combination right afterward can be difficult too. If you need a guaranteed fallback, the documented approach is a mixed instances group with a genuine on-demand base capacity baked in from the start, not a reactive scramble after the fact.
"A higher bid always beats a lower one, so bid as high as you can afford." As covered earlier, a high maximum price only protects against one specific cause of interruption — the price genuinely exceeding your bid — and does nothing for the far more common cause, which is AWS needing the capacity back regardless of what you're willing to pay. AWS's own advice, in fact, is usually to not set a maximum price at all and let it default to the on-demand rate, since you'll never pay above the live spot price anyway.
Ethan's take: is spot even worth the hassle?
Jake asked, after all this, whether it was even worth the trouble. Ethan didn't hedge: "For your overnight photo job, absolutely — it's the textbook use case. It's stateless, it's fault-tolerant if you build it right, and it's not tightly coupled to anything else. AWS says this in as many words: spot is recommended for stateless, fault-tolerant, flexible applications — big data, containerized workloads, CI/CD, stateless web servers, HPC, rendering. Your backup job is basically a rendering job with photos instead of pixels. But if you were running the point-of-sale system that takes customer payments in your actual shop? I'd tell you no, and I wouldn't soften that. That's stateful, it's fault-intolerant in the sense that a mid-transaction interruption is a real problem, and it's exactly the kind of workload AWS itself says spot isn't suited for."
Jake pushed back a little: "So how do I know which bucket something's in?" Ethan's answer was simple: "Ask yourself what happens if this specific instance disappears in exactly two minutes, right now, with zero warning beyond that. If the honest answer is 'I lose an in-progress transaction and a customer is standing at my counter,' that's on-demand territory. If the honest answer is 'the job restarts from its last checkpoint and nobody notices,' that's exactly what spot was built for."
Frequently asked questions
Why did my spot instance terminate even though my max price was higher than the spot price?
Because most terminations aren't about price. AWS terminates spot instances for capacity reclaim, host maintenance or hardware decommission, or unmet launch constraints — a high max price only protects against the price-exceeded scenario specifically.
How do I find out exactly why my spot instance was interrupted?
Open the EC2 console, go to Spot Requests, select the request tied to your instance, and read the status description. It shows a specific reason code such as instance-terminated-no-capacity or instance-terminated-by-price.
Is the two-minute warning guaranteed to arrive?
Interruption notices are emitted on a best-effort basis, meaning there's no absolute guarantee, though in practice AWS reliably sends them. Build your handling to check every 5 seconds, which is AWS's own recommended polling interval, so you don't miss it.
What's the difference between a rebalance recommendation and an interruption notice?
A rebalance recommendation is a risk signal that can arrive before the hard deadline, giving you time to proactively move your workload. An interruption notice is the actual two-minute countdown to a stop, hibernate, or terminate action.
Does the rebalance recommendation always arrive before the interruption notice?
No. AWS states it's not always possible to send the rebalance recommendation before the two-minute interruption notice, so the two can arrive together, leaving you with only the standard two minutes on that instance.
How do I check for the interruption notice from inside my instance?
Get an IMDSv2 token from the metadata service, then request http://169.254.169.254/latest/meta-data/spot/instance-action using that token. An HTTP 404 means nothing is pending; a JSON response with an action and time means an interruption is coming.
Should I set a maximum price for my Spot Instances?
AWS's own guidance is that you generally shouldn't. Leaving the maximum price unset defaults it to the on-demand price, and you only ever pay the actual spot price regardless — setting your own lower cap only adds a way for your request to go unfulfilled.
Can I get the interruption notice as an EventBridge event instead of polling the instance?
Yes. Amazon EC2 emits an EC2 Spot Instance Interruption Warning event to EventBridge, and a separate EC2 Instance Rebalance Recommendation event for the earlier signal. You can build an EventBridge rule that matches either and sends it to a Lambda function or an SNS topic.
What is Capacity Rebalancing in Auto Scaling and should I turn it on?
It's an Auto Scaling group setting that proactively launches a replacement spot instance as soon as a rebalance recommendation is received, rather than waiting for the actual interruption. AWS recommends it for spot workloads that need to maintain availability, though it does mean more replacement activity in your logs.
What allocation strategy should I use to reduce interruptions on replacement instances?
AWS recommends price-capacity-optimized over lowest-price. Lowest-price launches replacements in whichever pool is cheapest that instant even if it's about to be reclaimed again; price-capacity-optimized weighs capacity depth alongside price.
What happens to my data if my spot instance is terminated versus stopped?
On terminate, any instance store volumes are lost and EBS volumes follow their delete-on-termination setting. On stop, the EBS root volume and its data survive, and only Amazon EC2 can restart the instance once capacity is available again.
Why doesn't hibernate get the two-minute warning?
Because hibernation begins immediately once the interruption notice is issued — there's no two-minute delay built in for it the way there is for stop or terminate, since the hibernation process itself needs to start right away.
What's an on-demand base capacity, and do I need one?
It's a fixed number of on-demand instances that launch first in a mixed instances group before any spot capacity is added, giving you a guaranteed floor that spot interruptions can't touch. It's worth setting up for any workload where dropping below a minimum running capacity is a real problem, not just an inconvenience.
How do I handle spot interruptions on EKS or a Kubernetes cluster?
Use AWS Node Termination Handler, an AWS open-source project that cordons and drains a node when it receives a spot interruption notice or rebalance recommendation, either by monitoring instance metadata directly on each node or centrally through an SQS queue fed by EventBridge.
Is a single lone spot instance protected by any of this automatically?
No. Capacity Rebalancing and Auto Scaling's launch-before-terminate behavior only apply inside an Auto Scaling group or Fleet. A standalone instance launched with RunInstances can still poll its own metadata and react locally, but nothing is watching to launch it a replacement.
Should I fail over to on-demand instances when a spot instance is interrupted?
AWS specifically discourages this as a strategy, since failing over can drive further interruptions for your remaining spot instances, and getting on-demand capacity in the exact instance-type-and-Availability-Zone combination you were just reclaimed from may also be difficult right afterward.
Revision note. Written September 2026. AWS occasionally adjusts allocation strategy defaults and console workflows, so if a menu path here has moved by the time you read this, the underlying metadata endpoints and event names are far more stable and worth checking first. If a spot instance just disappeared on you and cost you something real, that frustration is completely fair — the fix is a few hours of setup, not a lesson you have to relearn the hard way twice.