Fix EKS error not authorized to perform: sts:AssumeRoleWithWebIdentity in AWS

The EKS error not authorized to perform: sts:AssumeRoleWithWebIdentity means AWS Security Token Service rejected the pod's attempt to exchange its Kubernetes service-account OIDC token for temporary IAM role credentials. It does not ordinarily mean that an S3 permission is missing. First check that the EKS cluster's OIDC issuer is registered as an IAM identity provider, then compare the role trust policy's provider ARN, sub, and aud conditions with the pod's actual namespace, service account, and token. The most overlooked fix is recreating an already-running pod after annotating its service account: the EKS admission webhook injects the IRSA token mount and environment variables when your pod is created, not retroactively into existing pods.

aws eks describe-cluster --name repair-prod --region us-east-1 \
  --query 'cluster.identity.oidc.issuer' --output text
aws iam list-open-id-connect-providers
kubectl -n bookings get sa booking-reader -o yaml
kubectl -n bookings get deploy booking-api -o jsonpath='{.spec.template.spec.serviceAccountName}'
aws iam get-role --role-name BookingS3ReadRole \
  --query Role.AssumeRolePolicyDocument

Jake runs a phone repair shop. His booking API runs inside an Amazon EKS cluster and needs to read product manuals from a private S3 bucket. The deployment works locally when a developer uses AWS credentials, but the pod fails in production with AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity.

The team adds s3:GetObject to the workload role. Nothing changes. The role already has the required bucket permission, but the pod is not authorized to assume the role in the first place. The permission failure occurs at STS before S3 receives a signed request.

A closer look reveals that the trust policy allows the service account in the default namespace, while the deployment runs in bookings. This small string mismatch is enough for STS to deny the exchange. The troubleshooting process below shows how to establish exactly where that identity chain breaks.

Understand IRSA in plain English: how the pod becomes an AWS role

IAM Roles for Service Accounts, or IRSA, connects a Kubernetes service account to an AWS IAM role. Kubernetes service accounts identify workloads inside the cluster; IAM roles define what AWS APIs those workloads can call. IRSA bridges the two identities without distributing a static AWS access key to every pod.

Every EKS cluster has an OIDC issuer URL. Kubernetes issues a signed projected service-account token to an eligible pod. The token has claims such as an issuer, an audience, and a subject naming the namespace and service account. IAM can validate this issuer when you register the corresponding OIDC provider.

When your pod uses an AWS SDK, the SDK reads AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, loads the signed token, and requests temporary credentials with AssumeRoleWithWebIdentity. STS validates the token and the role trust conditions. If they match, STS issues temporary role credentials.

Only after the exchange succeeds does the SDK use those credentials to call S3, DynamoDB, or another AWS service. A rejection from STS therefore points first to federated trust, token identity, and endpoint access—not to the downstream S3 bucket policy. The two stages need separate diagnosis.

Jake: "I already gave the role permission to read S3. Why does access still fail?"

Ethan: "S3 permission is step two. First, the pod must prove to STS that it is the service account trusted by that role."

Read the precise error before modifying IAM policies

The full error might include AccessDenied, InvalidIdentityToken, IDPRejectedClaim, or a failure to reach STS. These conditions are related but not identical. An access denial may mean the trust statement does not match; an invalid identity token may indicate issuer, audience, signature, or token validity issues.

Check the complete error text, the pod name, namespace, service account, role ARN, AWS Region, and relevant timestamp. Confirm whether the application used the intended AWS SDK credential provider. Static AWS environment credentials can take precedence over web identity credentials and confuse the picture.

The cluster and IAM role can also be in unexpected AWS accounts. The IAM OIDC provider must exist in the account where the role being assumed is defined. A role in another account needs a trust policy referencing the correct issuer provider registered in that role-owning account.

Use a consistent diagnostic order: issuer and provider, trust principal, trust conditions, annotation, pod creation time, token content, network, then downstream permissions. This sequence avoids repeatedly editing permission policies that cannot correct an OIDC mismatch.

Your first job is to prove that IAM knows your cluster’s token issuer. If it does not, nothing else in this guide can work, so start here even if you are sure it is set up.

Find the cluster OIDC issuer and IAM identity provider

The first prerequisite for IRSA is registering the EKS cluster’s OIDC issuer as an IAM identity provider. Creating an EKS cluster does not guarantee that someone also created this IAM provider. The cluster displays an issuer URL, but the separate IAM trust relationship must exist in the role-owning account.

Retrieve the issuer URL and note its trailing identifier. For example, the issuer may look like https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE. Your real issuer will have a different identifier.

CLUSTER=repair-prod
REGION=us-east-1
ISSUER=$(aws eks describe-cluster --name "$CLUSTER" \
  --region "$REGION" --query 'cluster.identity.oidc.issuer' --output text)
echo "$ISSUER"
aws sts get-caller-identity
aws iam list-open-id-connect-providers

The IAM list shows provider ARNs. Match the complete issuer host and path, not merely the AWS Region. A new cluster in the same Region normally has a different issuer identifier, so an old provider left over from a deleted cluster is not an equivalent replacement.

To inspect a specific provider after discovering its ARN, run aws iam get-open-id-connect-provider --open-id-connect-provider-arn "$PROVIDER_ARN" and inspect its URL and client ID list. The usual audience client ID for IRSA is sts.amazonaws.com.

If the provider is absent and you have authorization, associate it with the cluster using eksctl utils associate-iam-oidc-provider --cluster repair-prod --region us-east-1 --approve. In restricted-network environments, ensure the tool can resolve and reach the issuer endpoint; private DNS may require additional configuration.

Match the federated principal to the exact issuer

A role for IRSA trusts a federated identity provider rather than an EC2 instance role or the ordinary EKS cluster service principal. The trust policy’s Principal.Federated field names the IAM OIDC provider ARN belonging to the EKS issuer.

A common error is copying a trust policy from another cluster. Both clusters might be in the same account and Region, and both roles may have the same friendly name, but the issuer identifiers differ. STS validates the actual token issuer, so the old trust relationship will not match.

Another mistake is using the HTTPS prefix in the policy condition key. The Federated ARN takes the form arn:aws:iam::ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/ID. The condition-key prefix is the issuer hostname and path without https://.

Retrieve the trust document with aws iam get-role --role-name BookingS3ReadRole --query Role.AssumeRolePolicyDocument. Compare the complete federated principal against the registered IAM provider and the iss claim in the token.

Use the exact IRSA trust policy with sub and aud conditions

For one workload, the least-privilege trust policy matches a single namespace and service-account name. The sub claim follows the form system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT. The normal STS audience is sts.amazonaws.com.

The following example trusts the booking-reader service account in the bookings namespace. Replace your account ID and OIDC issuer ID everywhere they appear. Both the provider ARN and condition keys must refer to the same real issuer.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:bookings:booking-reader"
        }
      }
    }
  ]
}

Use StringEquals for an exact match. Broad wildcards make a role easier to assume from unintended service accounts. If you intentionally trust multiple service accounts, prefer an explicit list of subjects or carefully constrained patterns rather than granting access to every service account in the cluster.

The trust policy is separate from the role’s permissions policy. This trust document allows the specific pod identity to receive credentials; a separate permissions policy grants s3:GetObject to the intended bucket objects. Both policies are needed for the worked example.

Jake: "Can I leave the namespace out and match only the service account name?"

Ethan: "Not with an exact trust condition. The subject includes both values, so changing the namespace changes the identity STS sees."

If your trust policy now looks right, turn to your pod’s side of the handshake: the service account it actually runs as, which is not always the one you think.

Check the service account annotation and deployment identity

Under IRSA, the Kubernetes service account commonly has the annotation eks.amazonaws.com/role-arn. The EKS pod identity webhook sees that annotation when a pod is created and mutates the pod to include the web identity environment variables and projected token volume.

Check your service account first, then inspect the workload’s serviceAccountName. Your deployment must name the annotated service account; simply creating the right service account does not make every pod in the namespace use it.

kubectl -n bookings get serviceaccount booking-reader -o yaml
kubectl -n bookings get deployment booking-api \
  -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'
kubectl -n bookings get pods -l app=booking-api \
  -o custom-columns=NAME:.metadata.name,SA:.spec.serviceAccountName

To annotate an existing service account, replace the role ARN with the actual role and use --overwrite if the annotation already exists. This Kubernetes operation changes the service account resource; it does not by itself update the environment of running pods.

kubectl -n bookings annotate serviceaccount booking-reader \
  eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/BookingS3ReadRole \
  --overwrite

If your workload uses the default service account, either update the deployment to use the intended dedicated service account or deliberately secure the default account. A dedicated account per workload is usually easier to audit and limits accidental sharing of AWS permissions.

Jake: "I annotated the service account ten minutes ago. Why does the pod still act like nothing changed?"

Ethan: "Because it was already running. The token and the variables only get added when a pod is created, so this one needs replacing."

Restart pods after changing IRSA annotation

A frequent source of confusion is adding the correct annotation and seeing no immediate improvement. The webhook performs pod mutation at pod admission. It does not go back and insert a new projected token volume or change environment variables on a pod that already exists.

For a Deployment, a rollout restart creates replacement pods from the current pod template. Their admission process can then apply the current service-account annotation. Use the restart only after confirming the trust policy and annotation are correct.

kubectl -n bookings rollout restart deployment/booking-api
kubectl -n bookings rollout status deployment/booking-api
kubectl -n bookings get pods -l app=booking-api

After replacement, inspect a newly created pod. Verify the identity-related variables and projected token mount. If they remain absent, confirm the pod references the correct service account and that the EKS webhook mutation is functioning.

A running process might also cache credentials or use a credential provider that precedes web identity. Even after pod replacement, review the SDK credential chain if your role identity is still unexpected. Before you delete any healthy pods, confirm that mutation is really what is blocking you.

Inspect the environment variables injected inside a pod

An IRSA-enabled pod normally receives AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE. The token is commonly mounted under /var/run/secrets/eks.amazonaws.com/serviceaccount/token. A regional STS endpoint setting may also be configured.

Use kubectl exec to inspect a container that has a shell and the relevant tools installed. Some minimal application images omit both shell utilities and AWS CLI. In that case, inspect the pod spec or use a suitably controlled diagnostic container.

POD=$(kubectl -n bookings get pod -l app=booking-api \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n bookings exec "$POD" -- sh -c \
  'printf "role=%s\ntoken=%s\n" "$AWS_ROLE_ARN" "$AWS_WEB_IDENTITY_TOKEN_FILE"; test -r "$AWS_WEB_IDENTITY_TOKEN_FILE" && echo readable'

The command intentionally prints only the path, not the bearer token itself. A projected OIDC token is a credential; copying it into logs, ticket comments, or a public terminal transcript risks exposing a credential that can be exchanged during its validity period.

When your container includes the AWS CLI, run aws sts get-caller-identity. A successful result should identify an assumed-role identity associated with the intended IAM role. If it identifies a worker-node role instead, inspect the provider chain and whether IRSA mutation occurred.

Jake: "The service account looks correct, but the pod has no AWS_WEB_IDENTITY_TOKEN_FILE. What does that tell us?"

Ethan: "The pod probably was not mutated for IRSA. Check which service account it used at creation and recreate it after fixing the annotation."

Check token iss, sub, aud, and expiration without leaking the token

The projected service-account token is a signed JWT. Its claims can be decoded for diagnostics without treating the unsigned, decoded content as independently trustworthy. STS performs the necessary cryptographic verification; your diagnostic inspection is only a way to compare values.

Inspect the iss, sub, aud, and expiration fields using a script inside the pod or an authorized diagnostic environment. Keep the raw token on your machine: decode it locally rather than pasting it into an online JWT decoder.

python - <<'PY'
import os, json, base64
path = os.environ['AWS_WEB_IDENTITY_TOKEN_FILE']
with open(path) as f:
    token = f.read().strip()
_, segment, _ = token.split('.')
pad = '=' * (-len(segment) % 4)
claims = json.loads(base64.urlsafe_b64decode(segment + pad))
for key in ('iss','sub','aud','exp'):
    print(key, claims.get(key))
PY

A healthy IRSA token might show sub=system:serviceaccount:bookings:booking-reader and an audience that includes sts.amazonaws.com. The issuer must match the one registered in IAM and used in the role trust policy.

If the subject differs, fix the workload’s namespace or service account or adjust the trust policy intentionally. If the audience differs, ensure the token was projected for STS. If the token is expired, investigate token rotation, filesystem access, SDK compatibility, and the application’s handling of refreshed tokens.

What the audience must be and why a wrong value fails

The standard audience for EKS IRSA is sts.amazonaws.com. It identifies AWS STS as the intended recipient of the web identity token. The IAM OIDC provider should list that client ID, and the role trust condition ordinarily matches it exactly.

If a token was projected for another audience, such as the Kubernetes API server or a custom internal service, STS should not accept it as proof of the intended AWS role identity. Reusing the wrong projected token can lead to an InvalidIdentityToken or an access denial.

The IAM provider’s client ID list, token claim, and trust policy must be consistent. Inspect all three independently: provider configuration through IAM, projected token through pod diagnostics, and role trust using get-role.

The tempting shortcut is deleting the audience condition; fix the value instead, so the role stays locked to STS. Restricting both aud and sub helps ensure that the role can only be assumed through the intended workload identity. Preserve those constraints unless a carefully reviewed alternate design requires different trusted claims.

Once your pod can assume the role, you can finally ask the question you started with: can it read your bucket?

Give the role access to S3 only after STS trust succeeds

In Jake’s case, successful role assumption is necessary but not sufficient. The role also needs authorization to read the specific S3 objects. For a private bucket, the identity policy, bucket policy, and possibly a KMS key policy may all influence the final request.

Use a narrowly scoped identity policy allowing s3:GetObject only for the intended object prefix. If your application needs to list keys, add s3:ListBucket to the bucket ARN separately, with an appropriate prefix condition where possible.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadRepairManuals",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::repair-manuals-private/manuals/*"
    }
  ]
}

When your bucket uses SSE-KMS with a customer managed key, the role may also need kms:Decrypt and key-policy authorization. That is a separate failure from AssumeRoleWithWebIdentity, and it should be investigated only after credentials are acquired successfully.

Use aws sts get-caller-identity inside the pod before testing S3. If the identity is correct but S3 denies access, you have isolated the issue to resource authorization rather than federation trust.

Worked example: repair a pod that cannot read a private S3 object

Suppose the EKS cluster is repair-prod, the namespace is bookings, the service account is booking-reader, and the role is BookingS3ReadRole. The bucket is repair-manuals-private. These are illustrative names and account IDs; substitute your actual values.

Step one: check the pod. Its deployment explicitly uses booking-reader, but its AWS SDK logs show AccessDenied on AssumeRoleWithWebIdentity. Step two: inspect the IAM role trust policy. Its sub condition mistakenly reads system:serviceaccount:default:booking-reader.

Step three: correct the trust condition to the actual subject and make sure the OIDC provider and audience also match. If you maintain infrastructure as code, update that source and deploy through the approved pipeline; otherwise, an authorized administrator can update the role’s trust policy with the AWS CLI.

# Save the corrected trust JSON as trust-policy.json first.
aws iam update-assume-role-policy \
  --role-name BookingS3ReadRole \
  --policy-document file://trust-policy.json

kubectl -n bookings annotate sa booking-reader \
  eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/BookingS3ReadRole \
  --overwrite
kubectl -n bookings rollout restart deployment/booking-api
kubectl -n bookings rollout status deployment/booking-api

Step four: inspect the new pod. The projected token contains the intended sub and aud values, and the SDK obtains temporary credentials for the role. Step five: confirm the role has the appropriate S3 object permission and perform a read-only test using the real bucket and key.

# From a pod with AWS CLI installed:
aws sts get-caller-identity
aws s3api head-object \
  --bucket repair-manuals-private \
  --key manuals/screen-replacement.pdf

A successful GetCallerIdentity result proves that the AWS SDK or CLI obtained usable credentials. A successful HeadObject test establishes access to the specified object, subject to the API’s own required permissions and the object’s existence. The root cause was the namespace mismatch in the trust policy, not the lack of S3 permission.

Jake: "So adding more S3 permissions would never have fixed the original STS error?"

Ethan: "Correct. The token subject and trusted namespace had to match before the role’s S3 permissions could be used."

Share one IAM role across several service accounts carefully

One IAM role can trust more than one Kubernetes service account. For a small set of workloads requiring identical AWS permissions, list the exact accepted subjects under the trust policy’s sub condition. Keep the audience restriction unchanged.

For example, two service accounts in different namespaces can both be trusted if your role’s access scope is appropriate. This is more explicit than trusting every service account in every namespace.

"Condition": {
  "StringEquals": {
    "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com",
    "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": [
      "system:serviceaccount:bookings:booking-reader",
      "system:serviceaccount:inventory:inventory-reader"
    ]
  }
}

When workloads span distinct clusters, their issuer URLs normally differ. A role shared across clusters needs appropriate trust statements for each registered IAM OIDC provider. The same service-account name in another cluster is not the same federated identity.

Give each workload its own role when it needs different permissions or a separate audit trail. One role per workload often gives clearer least-privilege enforcement and better attribution, even when maintaining a shared role would require fewer IAM objects.

Check network, STS endpoint, and SDK credential precedence

If your OIDC trust configuration looks correct but obtaining credentials still fails, inspect network connectivity from the workload environment. Your pod must be able to reach the relevant AWS STS endpoint, through public internet access or an appropriately configured VPC interface endpoint.

Use a regional STS endpoint for IRSA. A service-account annotation such as eks.amazonaws.com/sts-regional-endpoints: "true" can help configure supported injected behavior; also verify the SDK’s actual endpoint selection and environment settings.

Credential precedence is another important source of misleading results. Static AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables might take priority over web identity credentials. Older SDK versions might not support the web identity provider correctly.

If your pod can still access an EC2 node instance profile through metadata paths, the SDK may obtain node credentials if IRSA is absent or unsupported. Check the resolved identity rather than assuming the projected token is being used merely because a service-account annotation exists.

Understand cross-account IRSA and role ownership

A pod in one AWS account can assume a role in another account when your destination account establishes the required OIDC provider and trust relationship for the cluster issuer, or when using an appropriately designed two-role assumption pattern.

In the direct OIDC trust pattern, the role-owning account must register the cluster issuer as an IAM OIDC provider in that account. The destination role’s trust policy names that provider, and its sub and aud conditions match the workload token.

In a chained pattern, the pod obtains credentials for a local IRSA role first, then assumes a destination role using sts:AssumeRole. The second role trusts the first AWS role rather than the Kubernetes token directly, and both roles require the relevant permissions.

In a different account’s role trust policy, use that account’s own IAM OIDC provider ARN, not the one from the cluster account. IAM identity provider objects are account-specific. Confirm your active account with STS before creating or updating the provider and trust document.

If you are setting up a new workload rather than fixing an old one, you have a second option worth knowing before you copy the IRSA pattern again.

Compare IRSA with the newer EKS Pod Identity approach

EKS Pod Identity is an alternative for granting AWS permissions to supported EKS workloads. It avoids per-cluster IAM OIDC provider setup and does not require the eks.amazonaws.com/role-arn annotation on the Kubernetes service account. Instead, an EKS Pod Identity association maps the cluster, namespace, and service account to a role.

Pod Identity uses the EKS Auth service and a node-side Pod Identity Agent where required. The agent obtains credentials using AssumeRoleForPodIdentity and makes them available to compatible AWS SDKs through the container credential provider.

For supported new EKS workloads, Pod Identity is often simpler to administer because IAM trust can be reused across clusters and Kubernetes account mappings live in EKS associations. IRSA remains useful when it is already functioning well or when workloads run in environments Pod Identity does not support, including some Fargate and Windows scenarios.

These are distinct credential mechanisms. Keep the two systems apart: an IRSA OIDC trust policy will not work on a Pod Identity role, and a Pod Identity association will not create the IRSA token variables. Choose one clearly supported credential chain for the workload.

Use the correct trust policy for EKS Pod Identity

A Pod Identity role trusts the AWS service principal pods.eks.amazonaws.com and allows two operations: sts:AssumeRole and sts:TagSession. It does not use the IRSA OIDC federated principal or the sts:AssumeRoleWithWebIdentity action.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

Create the required EKS Pod Identity association after configuring the role and ensuring the agent is available for your node environment. The following command associates the existing role with one service account. It does not require a Kubernetes annotation.

aws eks create-pod-identity-association \
  --cluster-name repair-prod \
  --namespace bookings \
  --service-account booking-reader \
  --role-arn arn:aws:iam::111122223333:role/BookingS3ReadRole \
  --region us-east-1

Pod Identity session tags can carry cluster, namespace, and service-account context. Role trust policies can use request-tag conditions to limit allowed workloads; the exact keys should follow EKS’s published tag names. These are not equivalent to the OIDC sub and aud conditions.

For node groups where an agent is required, install or confirm the eks-pod-identity-agent add-on. EKS Auto Mode incorporates Pod Identity without requiring that add-on separately. Verify SDK support and restart workloads as appropriate after changing their credential association.

Secure IRSA and keep workload identity auditable

Give each workload the smallest AWS permission set it needs. For a pod that reads manuals, s3:GetObject over a specific prefix is safer than granting s3:* over every bucket. Use a distinct service account to prevent unrelated pods from acquiring the same rights.

Restrict who can edit Kubernetes service accounts, deploy pods under sensitive service accounts, and mutate the IRSA annotation. Kubernetes RBAC is part of the security boundary: an actor who can create pods with a privileged service account might be able to use that service account’s AWS role.

Treat IRSA as a way to hand out credentials, not as a complete isolation boundary between containers that share a node. Protect access to node instance credentials and other credential paths. Use current AWS SDK versions that support token refresh, and monitor STS authentication failures without logging bearer tokens.

Document the cluster issuer, role ARN, namespace, service account, application owner, and intended AWS resources. When rotating or recreating clusters, update trust providers and run a known-good STS and resource access test before decommissioning the old configuration.

Still stuck? Work through this list in order. Each line rules out one cause, so you always know where you are and what you have already proved.

Troubleshooting checklist for stubborn AssumeRoleWithWebIdentity failures

When the simple fix does not work, capture the source of truth at each boundary. From EKS, get the exact issuer and namespace. From IAM, get the registered provider, its audiences, and the role trust policy. From Kubernetes, get the service-account annotation and the pod’s current service-account identity.

From the running pod, verify the injected variables, token file readability, JWT claims, SDK version, and resolved STS caller. From the network, verify access to the regional STS endpoint. Each result should confirm one link of the chain rather than generate more guesses.

If the error changes from STS access denied to S3 access denied, that is progress: authentication now succeeds, and the remaining problem concerns permissions on the bucket, object, KMS key, or IAM identity. Keep those investigations separate, so a correct trust repair stays in place.

For a production support case, include sanitized trust-policy conditions, the role ARN, OIDC issuer, affected namespace and service account, pod creation time, SDK version, and request timestamps. Leave out live projected tokens, static AWS keys and application secrets.

Diagnose webhook mutation and admission timing precisely

If the annotation exists but a new pod still lacks IRSA variables, check admission rather than retrying STS. The EKS pod identity mutating webhook evaluates the pod when Kubernetes creates it. Your pod must reference the annotated service account at that moment; changing the service account resource after your pod exists does not rewrite the pod specification.

Compare the creation time of the pod with the last modification of its service account. Inspect your Deployment template, ReplicaSet, and current pod rather than checking only the Kubernetes object you intended to edit. Helm releases, GitOps controllers, and rollout automation can restore old annotations or serviceAccountName values after a manual fix.

Run kubectl describe pod on a newly created pod to examine events, mounts, and configuration. Check your projected service-account token volume in your pod spec. The IRSA token path should come from a projected token with the STS audience, not from the ordinary Kubernetes API token mount used for cluster authentication.

If your infrastructure is managed declaratively, record the annotation in the repository configuration. Otherwise, a successful emergency kubectl annotate operation may be undone during the next deployment. Reconcile the source of truth, deploy it, then restart and confirm a new pod received the intended mutation.

Test the AWS identity from application code as well as AWS CLI

A CLI test is helpful but not always conclusive when your application uses a different SDK version, a custom credential provider, or cached session objects. The SDK should normally rely on its default credential chain and use the projected web identity automatically. Hard-coded profiles or credentials can prevent that behavior.

For Python applications, boto3 uses botocore credential providers that support web identity. Run a minimal read-only identity check in the same environment with the same Python dependencies that the application uses. Keep the token and any temporary access keys out of your output. The returned assumed-role ARN is enough to confirm which role was selected.

import boto3
sts = boto3.client("sts", region_name="us-east-1")
identity = sts.get_caller_identity()
print("Account:", identity["Account"])
print("Caller ARN:", identity["Arn"])

If Python resolves an unexpected role, check AWS_PROFILE, AWS_SHARED_CREDENTIALS_FILE, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and any custom session construction. Environment-provided static credentials can take priority over web identity. Remove unintended credential sources instead of broadening the trust policy.

When the same pod sometimes works and sometimes fails after a token ages, investigate how the SDK reloads rotated projected tokens. Kubernetes rotates those tokens, and compatible SDKs request new temporary credentials. Keeping an old token string in application memory indefinitely can make a previously working authentication path fail later.

Validate error recovery, deployment rollout, and safe rollback

A role-assumption repair should be tested in a controlled rollout. Update a test service account or a small workload replica first, obtain the expected AWS role identity, and perform a narrowly scoped call against the required AWS resource. Then expand the rollout only after confirming the trust and application permissions both work.

For business-critical services, restart Deployments one at a time unless you have verified readiness probes and spare capacity. A rollout restart can temporarily affect availability if replacement pods cannot schedule or become healthy. Monitor deployment status, Kubernetes events, and application logs as each replica turns over.

If authentication still fails, gather one clean example of the pod name, namespace, service account, issuer, role ARN, decoded JWT claims, error name, timestamp, and request identifier. This evidence usually isolates whether the issue sits in pod mutation, OIDC provider registration, trust conditions, or service networking.

Preserve a reversible change plan. Role trust policies and Kubernetes annotations should be updated through managed configuration where possible, and a rollback should restore only the previous known-good trust rules. Resist the quick fix of trusting every service account or allowing unrestricted role assumption, even for a few minutes; it turns one broken pod into an open door.

Know when the failure is not an IRSA trust-policy error

Not all credential problems mentioning STS can be corrected by editing OIDC trust. Network timeouts, DNS failures, expired projected tokens, or unsupported SDK credential chains may prevent the request from reaching the right service or presenting the right proof. Read the full exception class and request context.

If get-caller-identity already shows the intended IRSA role, the federation step has succeeded at least for that test. A later AccessDenied from S3, KMS, DynamoDB, or another service is a distinct authorization issue. Inspect those resources and identity policies rather than changing the OIDC subject again.

Similarly, the presence of AWS_CONTAINER_CREDENTIALS_FULL_URI and an EKS Pod Identity association points to the newer Pod Identity path, not ordinary web identity. Its debugging process examines the EKS Auth credential agent and the pods.eks.amazonaws.com trust service principal rather than the IRSA federated principal.

Keeping those boundaries straight is the fastest way to stay clear of accidental over-permission. Identify which credential mechanism the pod actually uses, which AWS principal results, and which API call is denied. A specific fix at that boundary is both easier to validate and safer to retain.

Frequently asked questions about EKS IRSA and STS access denied

What does not authorized to perform sts:AssumeRoleWithWebIdentity mean in EKS?

It means STS rejected the pod’s attempt to exchange a signed OIDC service-account token for temporary IAM role credentials. Check the IAM OIDC provider, role trust policy, token subject and audience, and the actual pod identity before troubleshooting permissions on S3 or other services.

How do I check whether my EKS OIDC provider exists?

Retrieve cluster.identity.oidc.issuer with aws eks describe-cluster and compare its complete URL with the provider ARNs returned by aws iam list-open-id-connect-providers. Inspect a matching provider with aws iam get-open-id-connect-provider.

What must the aud value be for EKS IRSA?

For the standard EKS IRSA trust relationship, the projected service-account token audience and the provider client ID must include sts.amazonaws.com. The IAM role trust policy normally uses a StringEquals condition on the issuer-specific aud key.

What should the sub claim contain for a service account?

The typical subject is system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT. For a pod using booking-reader in bookings, the exact subject is system:serviceaccount:bookings:booking-reader.

Do I need to restart a pod after annotating its service account?

Yes, normally recreate the existing pod or restart its Deployment so the admission webhook can inject the IRSA environment variables and token volume at pod creation. Editing the annotation does not automatically mutate already-running pods.

What annotation connects a Kubernetes service account to an IAM role?

For IRSA, set eks.amazonaws.com/role-arn to the IAM role ARN on the Kubernetes service account. Confirm that the pod actually uses that service account and that the trusted namespace and subject match.

How do I inspect IRSA credentials from inside a pod?

Check AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, confirm that the token file is readable, inspect its nonsecret claims locally, and run aws sts get-caller-identity if AWS CLI is installed. Never publish the raw token.

Why does the pod use the node IAM role instead of the IRSA role?

The pod might not have been mutated, use the wrong service account, run an incompatible SDK, or resolve credentials earlier in the provider chain. Inspect its environment and actual STS caller rather than assuming the annotation guarantees role selection.

Can one IAM role trust several EKS service accounts?

Yes. Put an explicit list of allowed sub claim values in the trust policy and retain the aud condition. For multiple clusters, account for each cluster’s distinct OIDC issuer provider.

Can IRSA work across AWS accounts?

Yes. The role-owning account can register the cluster’s OIDC issuer and trust that provider and specific service-account subject. Another pattern is to assume a local IRSA role first and then chain into a destination role with sts:AssumeRole.

Why does STS still deny access when the role has s3:GetObject?

The permissions policy determines what a role can do after assumption, while the trust policy determines who may assume it. An incorrect OIDC provider, namespace, service-account subject, or audience can block the credential exchange before S3 is called.

What is the difference between IRSA and EKS Pod Identity?

IRSA uses an IAM OIDC provider and sts:AssumeRoleWithWebIdentity with a service-account annotation. EKS Pod Identity uses an EKS association and agent-based credential delivery, with a role trusting pods.eks.amazonaws.com.

What is the EKS Pod Identity trust policy principal?

The principal is the AWS service pods.eks.amazonaws.com and the action list includes sts:AssumeRole and sts:TagSession. This differs from the federated OIDC principal and AssumeRoleWithWebIdentity action used by IRSA.

Does EKS Pod Identity require an OIDC provider or role annotation?

No. The association between cluster, namespace, service account, and IAM role is managed in EKS. Supported workloads use injected container credential-provider settings rather than the IRSA role annotation.

What happens if my service account namespace changes?

The IRSA JWT sub claim changes, so an exact trust condition for the previous namespace no longer matches. Update the role trust policy deliberately and recreate affected pods after verifying their service-account configuration.

How can I test an IRSA fix without exposing credentials?

Restart one affected workload, verify its projected token claims without printing the token, run aws sts get-caller-identity, and perform a narrow read-only call to the required AWS resource. Keep raw tokens and secrets out of logs and support tickets.

For Jake’s repair-booking workload, the fix was a precise trust-policy correction: the service account belonged to bookings, not default. Once the issuer, subject, and audience matched, STS could issue the role credentials and the existing S3 permissions finally mattered.

If you encounter AssumeRoleWithWebIdentity access denied, first prove which Kubernetes identity your pod presents and which identity the IAM role trusts. Restore that match, recreate the pod when required, and test the actual AWS principal before changing downstream resource policies. A small correction to the right boundary is safer than a broad permission grant.

Revision note. Written Nov 1 2026. Always read the token's claims before you touch the role.