EKS Nodes NotReady and Pods Pending: Every Cause

Your Amazon EKS nodes show NotReady when Kubernetes cannot rely on those machines to run Pods, while a Pod stays Pending when it has not reached a running workload state. Start with kubectl get nodes, kubectl describe node, kubectl get pods -A -o wide, and kubectl describe pod. The surprising distinction: running out of Pod IP addresses often produces FailedCreatePodSandBox after scheduling, not FailedScheduling. Check the event reason before you buy larger EC2 instances or expand the node group.

Jake runs a phone-repair shop whose inventory dashboard uses containers in an Amazon EKS cluster. On a busy morning the dashboard stops updating, while his cloud bill continues to accrue EC2 charges. In this illustrative case, he sees two Pending Pods and a node marked NotReady, so he assumes that simply adding another server must solve both symptoms.

That intuition is understandable, but Kubernetes has several different failure points. The scheduler may have no suitable node, a node may be unable to report health, or a Pod may have been scheduled but cannot receive networking. Those cases produce different events, and they require different repairs.

Jake: "My Pods are waiting. Would doubling the number of servers fix it?"

Ethan: "Only if capacity is actually missing. Read the Pod events first; they tell us whether the scheduler, the node, or the network is blocking the next step."

Read the five kubectl outputs before changing anything

If your cluster looks unhealthy, collect a consistent snapshot before restarting services or terminating instances. A node marked NotReady is a Kubernetes health observation, not a complete root-cause diagnosis. A Pending Pod is equally broad, because it can remain Pending while unscheduled or while waiting for a sandbox or container setup after being assigned.

Run kubectl get nodes -o wide to see node readiness, version, age, and internal address. Follow with kubectl describe node NODE_NAME and read Conditions, Taints, Allocatable, and Events. A Ready condition with a recent heartbeat is different from one whose status is Unknown after communication stopped.

Then inspect Pods across every namespace with kubectl get pods -A -o wide. The NODE column is unusually revealing: an empty assignment suggests scheduling has not succeeded, while a named node combined with ContainerCreating suggests startup after scheduling. Describe one representative failing Pod and read the Events at the bottom rather than guessing from the STATUS column.

Finally, inspect cluster events in time order using kubectl get events -A --sort-by=.lastTimestamp. Events are useful but retained for a limited period and can be aggregated, so pair them with node logs and AWS service details when a failure is intermittent. Record the namespace, Pod name, node name, instance ID, node group, and time so you can correlate the same incident.

kubectl get nodes -o wide
kubectl describe node NODE_NAME
kubectl get pods -A -o wide
kubectl describe pod POD_NAME -n NAMESPACE
kubectl get events -A --sort-by=.lastTimestamp

Understand how NotReady nodes and Pending Pods connect

When a node becomes NotReady, the Kubernetes control plane can stop considering it a healthy placement target. A Deployment that needs replacement replicas may create new Pods, but the scheduler must still find eligible Ready capacity. A cluster can therefore show NotReady nodes and FailedScheduling warnings together without every Pending Pod sharing the same underlying cause.

Kubernetes scheduling checks more than a simple machine count. It evaluates requested CPU and memory, Pod count limits, taints, node labels, affinity requirements, topology constraints, and other placement rules. A node group may contain running EC2 instances that fail one of those tests, so looking only at the EC2 console gives an incomplete answer.

If the Pod was already assigned to a node, the kubelet starts preparing its runtime environment and Pod network. Failure at that stage can leave it in Pending with ContainerCreating displayed by kubectl. That is where a CNI failure frequently appears, and it should not be described as the scheduler running out of CPU.

Inspect the Pod spec and the Events together, especially the Scheduled event and the field spec.nodeName. A successful Scheduled event followed by sandbox errors proves the scheduler found a location. If no node assignment exists and the event is FailedScheduling, investigate eligibility and capacity first.

Jake: "The EC2 instance says running, so why would Kubernetes ignore it?"

Ethan: "EC2 only tells you the virtual machine is powered up. Kubernetes still needs a healthy kubelet, working networking, and enough allocatable resources before it can use the machine."

Before you touch the node group, a word of reassurance: most of what you are about to read is a lookup, not a rebuild. You read one health message, you compare it with what you see in EC2, and you fix the one thing they disagree about. You rarely need to delete anything.

Repair managed node groups whose instances never joined

If your managed node group launches EC2 instances but the cluster shows no corresponding nodes, open the EKS node group health details before retrying creation. NodeCreationFailure means launched instances failed to register with the EKS cluster. For managed node groups, the failure can appear when nodes have not connected within 15 minutes; this is not an arbitrary Kubernetes Pod startup timeout.

Use the AWS console path EKS, Clusters, your cluster, Compute, Node groups, then inspect the node group status and health issues. The CLI equivalent is aws eks describe-nodegroup --cluster-name CLUSTER --nodegroup-name GROUP. Compare its health message with the actual EC2 instances and Auto Scaling group rather than assuming no machine was launched.

Common reasons include missing node IAM permissions, a node role that Kubernetes does not recognize, API endpoint connectivity, DNS and DHCP configuration, security group filtering, unsuitable subnets, or broken bootstrap/user-data settings in custom launch templates. If an EC2 instance cannot reach the cluster API server, increasing the desired node count may merely launch more instances with the same defect.

Confirm the node group refers to the correct cluster and region, then inspect its role, subnets, launch template, AMI, and routes. Use the Systems Manager Automation runbook AWSSupport-TroubleshootEKSWorkerNode when appropriate. Fix the root cause before recycling nodes, otherwise replacement instances will repeat the registration failure.

aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name workers --query "nodegroup.{status:status,health:health,version:version,nodeRole:nodeRole,subnets:subnets}"
aws eks describe-cluster --name my-cluster --query "cluster.{status:status,endpoint:endpoint,version:version}"

Give the node role the right IAM permissions and cluster identity

If the kubelet cannot authenticate or initialize AWS-dependent components, start with the EC2 node instance role. Amazon EKS worker nodes normally need permissions for Kubernetes node operations and access to the private container images used by essential add-ons. The current documented managed policy examples are AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryPullOnly, or equivalent permissions.

The Amazon VPC CNI needs additional EC2 networking permissions, but those permissions need not reside on the node role. They can instead be supplied to aws-node using EKS Pod Identity or IAM roles for service accounts. If you place CNI permissions on the node role as a fallback, use the policy appropriate to the cluster IP family and recognize the wider credential exposure.

Check the role attached through the node instance profile and compare it with the IAM role configured on the managed node group. The role ARN is not interchangeable with the instance profile ARN, and a typo in an account number will not work. The EKS cluster IAM role is also distinct from the EC2 node role; reusing it as the node role is not a normal replacement for missing worker permissions.

Look at IAM policy attachments from the IAM console or through aws iam list-attached-role-policies. Repair only missing permissions and confirm there is no explicit deny from another policy or organization guardrail. If image pulls still fail after node registration, investigate registry network access and repository permissions separately from the original NotReady symptom.

aws iam list-attached-role-policies --role-name myAmazonEKSNodeRole
aws iam get-role --role-name myAmazonEKSNodeRole

Choose aws-auth or access entries without breaking node registration

If your instance can contact the cluster but is rejected, inspect how the node IAM principal is authorized in Kubernetes. Older EKS setups use the aws-auth ConfigMap, while newer authentication modes support EKS access entries. The configured cluster authentication mode determines which method is available, so copying an old aws-auth repair command into every cluster is risky.

Use aws eks describe-cluster to inspect accessConfig.authenticationMode. For an API-capable cluster, list access entries and check the node IAM role principal. For legacy ConfigMap mappings, inspect kubectl -n kube-system get configmap aws-auth -o yaml, but be careful not to overwrite unrelated role mappings while editing.

Self-managed Linux EC2 nodes can use an EC2_LINUX access entry, while Windows nodes use the corresponding Windows entry type. Managed node groups ordinarily receive the necessary registration mapping automatically through EKS. If someone removed or altered the mapping, correct that specific registration problem rather than creating multiple conflicting identities.

Where both APIs and the ConfigMap are enabled, access entries take precedence for the same IAM principal. This matters when a role appears correct in aws-auth but continues to authenticate through a different access entry. Preserve the existing cluster administration access while making changes, and use the node role ARN rather than the instance profile ARN.

aws eks describe-cluster --name my-cluster --query "cluster.accessConfig.authenticationMode"
aws eks list-access-entries --cluster-name my-cluster
kubectl -n kube-system get configmap aws-auth -o yaml
aws eks create-access-entry --cluster-name my-cluster --principal-arn arn:aws:iam::111122223333:role/mySelfManagedNodeRole --type EC2_LINUX

Check subnet routes, security groups, and the API endpoint

If a freshly launched EC2 node fails to register, trace the path from its subnet to the EKS cluster endpoint. Private-subnet nodes usually need working routes to required AWS services, through appropriate VPC endpoints or another supported egress arrangement. A route table that cannot reach necessary APIs is not repaired by adding IAM policies to the role.

Inspect the cluster endpoint public/private access configuration. With private endpoint access, your node must resolve and reach the private endpoint within the VPC network. With public endpoint access, check the source restrictions and the node's egress route. Security groups and network ACLs must permit the required traffic, including communication between control plane and worker nodes.

Also inspect VPC DNS support, DNS hostnames, and the DHCP options set. Incorrect private DNS behavior can prevent the kubelet from registering under the expected node name. If you see node "" not found or repeated Unauthorized messages in the kubelet journal, connect those messages to the identity and DNS checks instead of treating them as ordinary CPU pressure.

Use the VPC console to inspect Subnets, Route tables, Security groups, and Endpoints, then compare the affected node subnet with a working node subnet if one exists. A subnet can have free IP addresses yet still be unusable for node registration because its routes or security rules differ. Keep that distinction in mind before expanding CIDR blocks.

If your nodes are Ready and your Pod is still Pending, you are now in the scheduler’s territory, and the scheduler is honest with you. Every reason it refuses a Pod is printed in the event. Your job is to read that reason literally, because “Insufficient cpu” means your requests, not your bill.

Fix FailedScheduling from insufficient CPU and memory requests

If the Pod event contains 0/3 nodes are available: 3 Insufficient cpu, you have a scheduling capacity problem, not necessarily an AWS instance shortage. The scheduler compares the Pod's CPU requests with the CPU still allocatable on each eligible node. Actual CPU utilization could be low and the Pod can still fail to schedule because other Pods have already reserved their requested shares.

The same applies to Insufficient memory. Kubernetes compares requested memory with allocatable memory, accounting for reservations and existing scheduled workloads. A Pod requesting 3 GiB cannot fit on a node with only 2 GiB unallocated for scheduling even if a dashboard temporarily shows less than 2 GiB being used.

Inspect kubectl describe pod for each container's Requests, including any init-container impact, and inspect kubectl describe node for Allocatable and Allocated resources. kubectl top nodes is valuable for actual utilization where Metrics Server is available, but usage numbers are not the scheduler's reservation ledger. Reconcile requests before reducing them, because overly small requests can shift the failure from Pending into OOM kills or contention.

The durable fixes are to right-size requests using observed workload needs, scale the node group, or choose an appropriate instance family. Autoscaling can provision capacity where configured, but constraints such as taints, strict affinity, subnet IPs, and EC2 service quotas may still block the new nodes. A scaling configuration is not a substitute for reading the event reason.

kubectl describe pod web-abc -n shop
kubectl describe node ip-10-0-1-20.ec2.internal
kubectl top nodes
aws eks update-nodegroup-config --cluster-name my-cluster --nodegroup-name workers --scaling-config minSize=2,maxSize=6,desiredSize=3

Match node taints, Pod tolerations, selectors, and affinity

If scheduling events mention an untolerated taint, investigate placement rules before buying capacity. A taint tells Kubernetes to repel Pods unless they have a matching toleration. Nodes may carry taints because they are dedicated to a particular workload or because Kubernetes detected a health condition.

The NoSchedule effect prevents new Pods without a matching toleration from landing on that node. A toleration only permits scheduling past that taint; it does not force the Pod to land on that node or override CPU and memory requirements. That distinction explains why adding one toleration sometimes changes the event but leaves the Pod Pending.

Inspect kubectl describe node for the Taints line and compare it with the Pod's tolerations and nodeSelector, nodeAffinity, topology spread, and other constraints. A strict nodeSelector for a label that no healthy node carries produces a placement failure even when plenty of general-purpose nodes are Ready. Persistent-volume topology may introduce an additional zone constraint.

Prefer correcting a mistaken selector, adding a narrowly scoped toleration, or creating capacity that matches the workload. Removing all node taints indiscriminately can put ordinary application Pods on infrastructure reserved for critical components. If the taint is health-related, investigate the underlying NotReady or pressure condition rather than masking it.

kubectl describe node NODE_NAME
kubectl get nodes --show-labels
kubectl get pod POD_NAME -n NAMESPACE -o yaml

Understand the max-pods limit and why small nodes fit few Pods

If a node has free CPU and memory but your Pod still cannot schedule, inspect its allocatable pods count. Kubernetes treats Pod slots as another schedulable resource. Amazon EKS networking also constrains practical Pod density because each EC2 Pod needs an address and its underlying instance type has elastic network interface and address limits.

There is no single universally correct Pod maximum for every EKS node. It depends on instance networking capabilities, bootstrap settings, IP addressing strategy, and the node's configured maximum. EKS managed node groups without custom AMIs inject maxPods settings with caps of 110 or 250 depending on instance size; these are ceilings in that configuration, not a promise that every node can run that many useful workloads.

Check kubectl describe node NODE_NAME for Capacity and Allocatable Pod counts, then count assigned Pods including system DaemonSets. A small instance may hit network-address or Pod-density constraints long before its RAM is full. Scheduling an additional Pod can then fail even though a CPU chart appears quiet.

Prefix delegation can increase available Pod addresses on eligible EC2 instances, but it is not an automatic universal fix. The subnet must supply the needed contiguous blocks and your CNI and node configuration must support the mode. If you change Pod density, preserve room for kubelet, networking agents, and application resource needs instead of chasing the largest advertised number.

kubectl describe node NODE_NAME
kubectl get pods -A --field-selector spec.nodeName=NODE_NAME
kubectl get daemonset -n kube-system

This next section is the one you want if your Pod was scheduled, got a node, and then stalled anyway. You will see a different event name, and the fix lives in your subnet rather than your deployment. Give yourself a minute with the aws-node logs before you resize anything.

Separate VPC CNI IP exhaustion from scheduler failures

If the Pod has a NODE assignment and its Events show FailedCreatePodSandBox, read the network error carefully. One characteristic message is Network plugin cni failed to set up pod network: add cmd: failed to assign an IP address to container. The surrounding wording varies with kubelet and runtime versions, but that core phrase points you toward IP allocation rather than CPU requests.

The Amazon VPC CNI provides network interfaces and VPC IP addresses for Pods running on EC2 nodes. It may fail because a subnet lacks free addresses, the instance reached an ENI or address limit, the aws-node agent is unhealthy, or necessary EC2 API permissions are missing. Subnet fragmentation can also obstruct prefix delegation even when the free-address count is nonzero.

Start with kubectl -n kube-system get pods -l k8s-app=aws-node -o wide and inspect CNI DaemonSet readiness. Check the affected subnet with aws ec2 describe-subnets --subnet-ids ... --query .... If addresses appear available, inspect the node's IPAMD logs, current ENIs, and whether contiguous /28 blocks are available when prefix delegation is enabled.

Remedies include providing adequate subnet address space, using eligible subnets for Pod allocation, correcting the CNI identity and permissions, resolving IP fragmentation, or choosing instances with sufficient network interfaces. Enhanced subnet discovery and prefix delegation have version and configuration requirements; verify those before enabling them. Simply scaling EC2 may worsen shortage in a depleted subnet because new nodes also consume addresses.

kubectl -n kube-system get pods -l k8s-app=aws-node -o wide
kubectl -n kube-system describe daemonset aws-node
kubectl -n kube-system logs daemonset/aws-node -c aws-node --tail=150
aws ec2 describe-subnets --subnet-ids subnet-0123456789abcdef0 --query "Subnets[0].{Subnet:SubnetId,FreeIPs:AvailableIpAddressCount,CIDR:CidrBlock}"

Jake: "The node has spare memory. Why does it say it cannot create the Pod?"

Ethan: "Think of your shop having free repair benches but no shipping labels. The Pod needs an IP address too; CPU availability does not create one."

Read DiskPressure and MemoryPressure before restarting nodes

If a previously healthy node turns NotReady or starts evicting Pods, inspect node Conditions before assuming a networking outage. DiskPressure describes pressure on node storage resources, while MemoryPressure indicates insufficient available memory under kubelet thresholds. These conditions can produce taints, evictions, or scheduling changes, and they have different corrective actions.

Container image layers, container logs, writable container layers, and other ephemeral data can consume node storage. A node with ample EBS volume capacity on paper can still experience pressure if the relevant filesystem is full or an inode threshold is crossed. Inspect filesystem usage and kubelet eviction messages on the instance to see which storage resource is actually constrained.

On supported Linux nodes, use df -h, df -i, and journalctl -u kubelet to identify the pressure source. Clean up unnecessary files using the supported container runtime practices, adjust log retention, or replace nodes with appropriate storage. Deleting runtime-managed files blindly while workloads are active is how a disk problem becomes an outage.

For MemoryPressure, compare application requests, limits, real usage, and kernel or kubelet events. Repeated process termination or resource starvation may require more memory or different workload sizing. Clearing a taint without clearing its underlying node condition does not create reliable production capacity.

kubectl describe node NODE_NAME
sudo df -h
sudo df -i
sudo journalctl -u kubelet -n 150 --no-pager

Inspect kubelet, containerd, and CNI logs on the node

If Kubernetes events remain vague, investigate directly on the affected EC2 node using an authorized access method such as Systems Manager Session Manager or approved SSH. The kubelet reports node health and starts Pods, while containerd is the container runtime commonly used on current EKS Linux worker nodes. Their journals often show why a previously Ready machine stopped progressing.

On Amazon Linux based node images with systemd, inspect sudo systemctl status kubelet and sudo journalctl -u kubelet -n 200 --no-pager. For container runtime issues, use the corresponding containerd service logs. Search around the actual failure timestamp rather than relying on unrelated warnings from hours earlier.

When the symptom points at networking, inspect the aws-node Pod and IPAMD log file at /var/log/aws-routed-eni/ipamd.log where applicable. Logs mentioning failed EC2 API calls, missing addresses, or subnet allocations help distinguish IAM mistakes from address exhaustion. A CNI Pod in CrashLoopBackOff warrants checking its own restart reason and logs.

Node image choice matters: Bottlerocket has a different management environment, and Windows nodes require Windows-specific tools. AWS provides an EKS log collector script on supported nodes to gather diagnostic information. Capture logs before recycling a node, because termination can destroy the simplest evidence of an intermittent fault.

sudo systemctl status kubelet
sudo journalctl -u kubelet -n 200 --no-pager
sudo systemctl status containerd
sudo journalctl -u containerd -n 200 --no-pager
sudo tail -n 200 /var/log/aws-routed-eni/ipamd.log
sudo bash /etc/eks/log-collector-script/eks-log-collector.sh

Check cluster, kubelet, AMI, and add-on version compatibility

If nodes started failing after an EKS upgrade, compare the control-plane Kubernetes version with the node group and kubelet versions. EKS supports specific version-skew relationships, but the safest routine operational state is to keep nodes updated to the cluster minor version. A version mismatch is a clue to investigate, not automatic proof of the NotReady condition.

Use aws eks describe-cluster for control-plane version and aws eks describe-nodegroup for managed node group version and release. The node listing also shows kubelet versions. Inspect CNI, CoreDNS, and kube-proxy add-on versions for compatibility with the cluster and its networking setup.

When updating managed node groups, EKS can replace instances and drain Pods as part of the update process. Custom AMIs and launch templates have additional requirements, so a generic version-update command is the wrong tool for a group with customized bootstrap settings. Confirm that your preferred rollout has spare capacity and respects application availability requirements.

Some version changes alter available capabilities or defaults, but unrelated configuration defects can surface during the same maintenance window. Compare logs and events before and after the upgrade, then identify the precise failing component. If only one node image release fails while others work, review its bootstrap settings and associated role and network configuration.

aws eks describe-cluster --name my-cluster --query "cluster.version"
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name workers --query "nodegroup.{version:version,release:releaseVersion}"
kubectl get nodes -o wide
aws eks list-addons --cluster-name my-cluster

Find why nodes repeatedly alternate between Ready and NotReady

If nodes join successfully and then repeatedly disappear from service, investigate fluctuations instead of treating it as a single bootstrap failure. Possible causes include unstable network connectivity to the API server, container runtime crashes, kubelet restarts, resource starvation, storage pressure, and EC2 underlying health issues. The timing pattern often identifies the responsible layer.

Collect node Conditions and their transition times with kubectl describe node. Compare that record with EC2 status checks, kubelet restarts, containerd logs, and any instance replacement events from the Auto Scaling group. If the node is being replaced regularly, distinguish an infrastructure lifecycle action from a process that becomes unhealthy on the same instance.

Check whether readiness changes line up with spikes in memory, ephemeral storage, image pulls, or the number of newly started Pods. If a CNI DaemonSet restart precedes network failures, inspect its logs and add-on health. A pattern affecting every node simultaneously calls for different scrutiny than one faulty machine.

Once the fault is identified, correct its durable configuration and use controlled node replacement if necessary. Repeatedly running kubectl uncordon is not a cure for resource starvation or broken networking. For managed node groups, consider the supported node auto-repair capability where appropriate, while understanding its replacement thresholds and disruption implications.

You have the pieces now, so let us walk one real repair end to end with you at the keyboard. Follow the same order you would use at 2 a.m.: look, confirm, change one thing, watch. If your situation differs, you can still borrow the sequence.

Worked example: restore one NotReady worker and schedule the waiting Pod

Imagine an illustrative three-node managed group named workers in my-cluster. Two nodes are Ready, one is NotReady, and a shop API Pod requests 500 millicores of CPU plus 512 MiB of memory. The Pod remains Pending and emits a FailedScheduling event because the two healthy nodes lack sufficient unallocated CPU, while the third node is not an eligible target.

The first action is to describe the bad node and the Pending Pod. The node shows Ready=Unknown with recent kubelet heartbeat failure, and the Pod event reports insufficient CPU on the other two nodes. This narrows the issue to restoring healthy node capacity or adding a replacement, rather than changing the VPC CNI merely because the application uses AWS networking.

An authorized node session reveals that the kubelet service is stopped after a node-specific configuration problem. In this example, the operator corrects that local configuration through the approved node build and restarts kubelet, then inspects the service logs. If the defect is systemic across the launch template, fix that template instead of repeating a manual edit after every new instance.

After the kubelet registers and the Ready condition becomes True, the scheduler can reevaluate the Pending Pod. Confirm the Pod obtains a NODE value and moves to Running, then inspect its Events for successful scheduling and container startup. If readiness does not recover, drain and replace the unhealthy node through the managed group rather than forcing the node to appear Ready.

kubectl get nodes -o wide
kubectl describe node ip-10-0-1-30.ec2.internal
kubectl describe pod shop-api-abc -n shop
# On the affected Linux node, after correcting the cause:
sudo systemctl restart kubelet
sudo systemctl status kubelet
# Back at your admin terminal:
kubectl get nodes --watch
kubectl get pods -n shop -o wide

Jake: "So the Pod did not need a bigger CPU request?"

Ethan: "Correct. Its request was fine; the eligible node supply was temporarily too small. Recover the worker, then confirm the Pod actually becomes Running."

Drain and replace a bad node without casually disrupting Pods

If a worker cannot be restored reliably, plan its replacement around your applications rather than deleting the EC2 instance first. A drain marks the node unschedulable and requests eviction of eligible Pods so their controllers can recreate them elsewhere. Availability depends on spare capacity, replica counts, PodDisruptionBudgets, and workload storage behavior.

Begin with kubectl cordon NODE_NAME and inspect workloads assigned there. Then use kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data only when you understand the implications for node-local ephemeral data and the ability of Pods to move. PodDisruptionBudgets can block evictions intentionally to protect minimum availability.

After workloads migrate, replace the EC2 instance through the managed node group lifecycle or approved Auto Scaling process. Simply deleting a node object in Kubernetes does not automatically repair or terminate the corresponding virtual machine. Likewise, terminating an instance before safely evacuating it can cause avoidable application interruption.

Managed node groups automatically handle draining during supported updates and scaling operations, and can provide optional node repair. Verify a replacement joins as Ready and that application replicas recover before changing capacity again. Treat forceful eviction options as exceptional because they can bypass protections that were added for a reason.

kubectl cordon NODE_NAME
kubectl get pods -A --field-selector spec.nodeName=NODE_NAME
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data
kubectl get nodes -o wide

Fix Fargate Pods Pending when no profile matches their labels

If your workloads use EKS Fargate, an EC2 node-group repair may not apply at all. Fargate selects eligible Pods based on profiles with namespace and optional label selectors. A Pod that does not match an appropriate profile can remain Pending even when the EKS control plane is healthy.

Inspect the Pod namespace and labels using kubectl get pod ... --show-labels, then compare those values with the Fargate profile selectors. Every label specified in a selector must match the Pod, while a selector that only specifies a namespace matches that namespace without an additional label requirement. A different spelling or missing label can be enough to prevent Fargate placement.

Open the EKS console, select your cluster, then Compute and Fargate profiles. Alternatively run aws eks list-fargate-profiles and aws eks describe-fargate-profile. Confirm profile status, selector, Pod execution role, private subnet selection, and required networking to AWS services. Fargate runs on private subnets, and it does not use an arbitrary EC2 managed node group as its backing capacity.

Create an appropriate profile or correct the workload namespace and labels, then recreate a Pending Pod controlled by a Deployment or other controller when necessary to trigger fresh matching. Fargate removes your responsibility for managing EC2 worker nodes but does not eliminate IAM, subnet IP, compatibility, or Pod scheduling constraints. DaemonSets and privileged containers are not supported on Fargate, so those workloads need another design.

kubectl get pod POD_NAME -n shop --show-labels
aws eks list-fargate-profiles --cluster-name my-cluster
aws eks describe-fargate-profile --cluster-name my-cluster --fargate-profile-name shop-profile
eksctl create fargateprofile --cluster my-cluster --name shop-profile --namespace shop --labels infrastructure=fargate

Choose a capacity fix without paying for the wrong resource

If your first instinct is to increase the node-group size, compare the cost of more capacity with the actual error. Adding EC2 instances can address insufficient allocatable CPU or Pod slots, but it cannot repair a broken aws-auth mapping, dead kubelet, or exhausted subnet. New instances may also fail to join if the underlying launch template or routing issue remains.

For an illustrative calculation, suppose three worker instances each cost $0.10 per instance-hour in your selected pricing arrangement. Their compute expense is 3 × $0.10 = $0.30 per hour, excluding EBS, data transfer, cluster charges, and other resources. Increasing to five instances raises that illustrative compute component to $0.50 per hour, or $0.20 more every hour.

That example is arithmetic, not a quoted EC2 market rate. Real rates vary by instance type, purchase model, region, operating system, and date, and EKS cluster fees and networking charges are separate. Check the actual instance price in your account and region before estimating a production change. The point is to stop paying for capacity that cannot fix the failure.

When CPU reservations are the problem, right-sizing Pod requests may be the lowest-cost remedy, followed by scaling suitable nodes. When addresses are exhausted, subnet planning and CNI allocation are the likely investment. When an unhealthy node is the sole defect, replacing or repairing it can recover capacity without permanently increasing the fleet.

Monitor node health, subnet capacity, and scheduling separately

If this is your second EKS scheduling incident, build alerts around the distinct failure domains rather than one generic Pod Pending count. Node readiness, unschedulable Pods, subnet available IPs, CNI failures, and node group health can each lead you to a different repair. Separate signals make on-call decisions faster and reduce destructive guesswork.

Use Kubernetes state metrics and your chosen monitoring stack to identify NotReady nodes and Pending Pods, and preserve a way to inspect Events. Watch CloudWatch or compatible observability for EC2 and node resource saturation, container runtime issues, and unexpected instance replacement. Examine CNI IP allocation metrics and VPC subnet free-address counts where supported.

For high-impact services, maintain enough replicas and disruption budgets to tolerate ordinary node updates. Validate new launch templates and node roles in a controlled environment before broad rollout, and confirm cluster/node add-on compatibility during upgrades. A node replacement policy is useful only when the next node can launch and register correctly.

Finally, prepare a runbook that starts with five kubectl commands and branches by event reason. Keep console and CLI paths for node group health, AWS Identity and Access Management, subnets, and Fargate profiles. The best operational improvement is remembering that Pending and NotReady are symptoms that demand classification before a remedy.

Gather the right evidence if none of the fixes works

If you still cannot find the cause, gather one complete incident record before opening a support case. Include the AWS Region, EKS cluster name, node group or Fargate profile, affected instance IDs, approximate incident times, and relevant Kubernetes object names. These are more useful than a screenshot showing only Pending or NotReady.

Capture kubectl describe node, kubectl describe pod, the relevant Kubernetes Events, and the node group health issues. If the failure occurs inside a node, include kubelet and containerd logs and the appropriate CNI diagnostics. EKS optimized nodes may provide the log collector script; confirm its availability on your node image.

Remove credentials, tokens, sensitive Pod environment values, and private customer data before sharing diagnostics. Give the exact event reason and full message, not merely a paraphrase of the status. If a failure began after a version change, provide the before-and-after versions and launch template details.

AWS Support may help isolate AWS-managed service or configuration issues, but cannot make an exhausted subnet hold unlimited addresses or force a workload onto a node that fails its scheduling requirements. If the application requires more resources than any eligible node offers, the durable solution is capacity or workload redesign. Keep the remediation connected to the evidence.

Frequently asked questions about EKS NotReady nodes and Pending Pods

Why are my EKS nodes NotReady?

A node is NotReady when Kubernetes does not consider it healthy for normal Pod scheduling. Inspect node Conditions, kubelet connectivity, container runtime, memory, storage, and networking. An EC2 instance running does not guarantee a Ready Kubernetes node.

Why are EKS Pods stuck Pending with FailedScheduling?

FailedScheduling means the scheduler did not find an eligible placement for the Pod. Inspect CPU and memory requests, taints, selectors, affinity, and Pod limits. The precise event message usually identifies which requirement blocked scheduling.

How do I fix NodeCreationFailure in an EKS managed node group?

Inspect the node group health issue, node role permissions, cluster access mapping, bootstrap configuration, and routes to the cluster API. EKS reports NodeCreationFailure when launched managed instances fail to register. Correct the shared defect before attempting node replacement.

What does failed to assign an IP address to container mean?

The VPC CNI could not allocate the Pod networking address needed during sandbox creation. Investigate subnet free IPs, ENI limits, aws-node health, and CNI permissions. This often follows successful scheduling rather than a FailedScheduling event.

How do I tell no CPU capacity from no IP addresses in EKS?

An Insufficient cpu event under FailedScheduling points to Pod requests exceeding eligible allocatable CPU. A FailedCreatePodSandBox event containing failed to assign an IP address points to CNI setup after scheduling. Use kubectl describe pod and check whether a NODE has been assigned.

How many Pods can one EKS node run?

The limit depends on instance networking capabilities, configuration, and resource availability. Managed node groups without a custom AMI use injected maxPods values capped at 110 or 250 depending on size, but many small nodes have lower practical limits. Check node Capacity and Allocatable rather than assuming a universal figure.

Why is an EKS node running in EC2 but missing from kubectl get nodes?

The instance may not have registered with the cluster. Check its instance IAM role, aws-auth or access entry, node bootstrap, DNS, security groups, subnets, and cluster endpoint connectivity. EC2 running state alone does not prove Kubernetes registration.

Does aws-auth still matter when EKS uses access entries?

It matters in clusters whose authentication mode includes ConfigMap authentication. In API_AND_CONFIGMAP mode, an access entry takes precedence over a ConfigMap mapping for the same IAM principal. Inspect the cluster authentication mode before modifying either.

Can an untolerated taint keep a Pod Pending?

Yes. A NoSchedule taint blocks Pods that lack a matching toleration. Adding a toleration only removes that particular restriction; the Pod must still satisfy its other scheduling constraints.

How do I fix DiskPressure on an EKS node?

Inspect node Conditions and node filesystem usage, including space and inodes. Identify excessive container logs, images, or ephemeral data, then use supported cleanup, storage sizing, or replacement procedures. Removing the taint alone does not clear the actual pressure.

Where can I read EKS kubelet and containerd logs?

On supported systemd-based Linux nodes, use journalctl -u kubelet and journalctl -u containerd. The supported commands differ for Bottlerocket and Windows nodes. For CNI allocation failures, inspect aws-node logs and IPAMD diagnostics.

Why do EKS nodes become Ready and NotReady repeatedly?

Intermittent API connectivity, kubelet or container runtime failures, storage or memory pressure, and EC2 health problems can cause flapping. Correlate node condition transitions with service logs and instance health events. Correct the underlying condition instead of repeatedly uncordoning the node.

How do I safely drain and replace an EKS node?

Cordon the node, inspect workloads, then drain it with suitable flags and spare capacity while respecting PodDisruptionBudgets. Replace the instance through the managed node group or approved lifecycle process. Confirm replacement readiness and workload recovery afterward.

Why is an EKS Fargate Pod stuck Pending?

A Pod intended for Fargate may lack a matching profile for its namespace and labels. Inspect the Fargate profile selectors, its status, Pod execution role, and supported private subnets. After correcting the selection, a controlled Pod recreation can trigger matching.

Does EKS Fargate eliminate all Pending Pod problems?

No. Fargate removes much of the EC2 node management burden but Pods still need matching profiles, appropriate execution roles, sufficient subnet networking, and supported Pod configurations. Profile mismatch and unsupported features can still prevent scheduling.

Should I scale my EKS node group to fix all Pending Pods?

Scale only when the events identify a legitimate shortage of eligible capacity. More nodes cannot repair broken registration mappings, missing Fargate profiles, or a shared depleted subnet. Diagnose the event reason first, so your spend goes up only when it restores service.

Jake’s dashboard is useful again only when the repaired or replaced node is Ready, the API Pods are Running, and their endpoints respond to real requests. Your cluster deserves the same end-to-end confirmation: distinguish scheduling from container startup, identify the precise failing layer, and make the smallest correction that restores capacity. If your event text differs from an example on this page, use that difference as a clue rather than trying to force the wrong repair.

Revision note. Written October 15, 2026, with access entries and the VPC CNI prefix-delegation caps as EKS documents them. A chain is only as strong as its weakest link, and a node group is only as Ready as its least healthy node.