API Gateway 500 Internal Server Error With Lambda: "Invalid Permissions on Lambda Function", Execution Logs, 500 vs 502 vs 504, Aliases, Mapping Templates and the One-Line add-permission Fix

API Gateway 500 Internal Server Error from a Lambda integration often means the invocation was rejected or API Gateway failed to configure or reach its integration. Check the API Gateway logs first: REST API execution logs or HTTP API access logs with $context.integrationErrorMessage. If Lambda actually ran, examine its function logs next. The counterintuitive part is that a Lambda handler that throws an exception commonly produces 502 Bad Gateway, not 500; the client can still receive the identical generic {"message":"Internal server error"} body.

Jake runs a small phone-repair shop with a booking form backed by an API Gateway endpoint and one Lambda function. His customers can submit the form, but some requests come back with an unhelpful server error. A staff member then has to telephone the customer to determine whether the booking exists. The API problem has become a business problem because an ambiguous failure can cause duplicate repair tickets.

A direct test in the Lambda console succeeds with the sample event Jake saved last week. His developer wonders whether to increase memory, redeploy everything, or change the booking form. None of those steps identifies which boundary rejected the real browser request. A reproducible request ID and the integration error are much cheaper evidence.

Read 500, 502, and 504 as Different Failures

When your endpoint returns an error, write down the HTTP status before reading the generic JSON body. The status separates gateway-side invocation problems from function execution errors and timeout problems. AWS Lambda integrations have a particularly useful distinction: rejection by the Lambda invocation API can yield 500, while an invoked function that fails can yield 502. Yet a consumer might see an identical error message in both cases.

A 500 can mean the integration cannot invoke its target, the target ARN is wrong, or API Gateway has a configuration error. A 502 frequently means the handler raised an error or a Lambda proxy response violated the expected format. A 504 indicates an integration timeout; it does not prove that backend work stopped at that instant. Match these patterns with your actual request logs rather than assuming one status has just one possible cause.

Other settings can affect the response: a REST API custom integration can map a Lambda error according to its integration response, and your Lambda function can deliberately return an application-level HTTP 500. That handled application 500 is not an invocation rejection. HTTP status alone should therefore guide, not replace, your diagnosis. The precise failure log tells you where to work next.

A timeout and a function exception can look the same from your frontend, but they are different failures. API Gateway might stop waiting while Lambda continues its asynchronous work or finishes later, depending on timing. A retried booking can then be processed twice unless your business operation is designed to handle duplicate attempts. Record both the HTTP result and the expected business side effect before retrying sensitive operations.

Jake: "So 500 means my shop counter could not pass the job to the technician?"

Ethan: "Often, yes. A 502 can mean the technician took the job but returned something the counter could not use. Read the actual error line before deciding."

Start with the Request That Actually Failed

Your first job is to capture one failure without changing the system. Note the full URL, HTTP method, request time, Region, API ID, and stage. Save the status code, response headers, request ID, and an appropriately redacted request body. This context makes a CloudWatch search about one request rather than every request your API ever handled.

Begin with the deployed URL, not a remembered configuration screenshot. Custom domain mappings can lead to a different API or stage from the one currently open in the console. A newly edited REST API method may not be deployed to the stage receiving requests. Check the real endpoint mapping before investigating the wrong function.

1. Send one harmless request to the deployed URL and capture the response and timestamp.
2. Identify the API as REST or HTTP and record its API ID, stage, and route.
3. Locate the gateway access or execution log for that request ID.
4. Read the integration error and determine whether Lambda invocation happened.
5. If Lambda ran, correlate its log entries and inspect exception or response details.
6. Make the smallest targeted correction, then repeat the same request.
7. Confirm the browser or production client works, including response headers and business outcome.

Read the error in order. If the gateway says Invalid permissions on Lambda function, the problem is not solved by increasing handler memory. If the invocation succeeded and logs show an uncaught exception, changing invocation permissions adds risk without helping. If the function ran cleanly but the proxy response format was invalid, inspect the response object. Every branch has its own best next test.

This approach also avoids hiding the original failure with later changes. A broad wildcard policy can mask a missing-permission diagnosis but leave unnecessary access afterward. A new deployment can introduce unrelated configuration differences. Keep one failing request, one logged cause, one change, and one new request as your troubleshooting unit.

Turn On REST API Execution Logging with the CloudWatch Role

If your API is REST and the client only shows a generic 500, execution logs are where gateway processing becomes visible. Open API Gateway, select the REST API, and go to Stages. Select the affected stage and open Logs and tracing to configure execution logging. Choose ERROR for ongoing diagnostic visibility or temporarily INFO for a more detailed investigation.

REST execution logging needs an account-level IAM role for CloudWatch Logs. That IAM role trusts the API Gateway service and has permission to create log streams and publish events. The managed policy AmazonAPIGatewayPushToCloudWatchLogs supplies the common logging permissions. The role setting is account-level within a Region, so a role configured in a different Region does not finish this setup.

Create the following trust policy locally as apigw-trust.json. A trust policy controls which service can assume a role, whereas the permissions policy controls what that role can do after assumption. Keeping these two functions separate prevents the common mistake of attaching CloudWatch permissions but forgetting the API Gateway principal. Use an IAM identity authorized to create and pass or associate the role.

{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"apigateway.amazonaws.com"},"Action":"sts:AssumeRole"}]}

aws iam create-role --role-name ApiGatewayCloudWatchRole --assume-role-policy-document file://apigw-trust.json
aws iam attach-role-policy --role-name ApiGatewayCloudWatchRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs
aws apigateway update-account --region us-east-1 --patch-operations op=replace,path=/cloudwatchRoleArn,value=arn:aws:iam::123456789012:role/ApiGatewayCloudWatchRole

After the role exists, enable stage logging and invoke a request. API Gateway manages the REST execution log group using the naming pattern API-Gateway-Execution-Logs_{rest-api-id}/{stage}. In CloudWatch Logs, select that group and inspect streams near the failed request time. Log delivery depends on the service role and regional STS configuration, so a missing group can be a logging setup problem rather than proof that the API never ran.

For CLI stage settings, the following command uses the REST API update-stage operation and the escaped wildcard method-settings path. The /*/*/ part means every resource and every method, and the API ID and stage must already exist. Use get-stage afterward to inspect whether loggingLevel was applied. The example enables error-level logs rather than full data tracing.

aws apigateway update-stage --rest-api-id abc123def4 --stage-name prod --patch-operations 'op=replace,path=/*/*/logging/loglevel,value=ERROR'
aws apigateway get-stage --rest-api-id abc123def4 --stage-name prod

Execution logs can disclose details about the request path and integration processing, so protect the log group with appropriate IAM permissions and retention. Data tracing can include payload data and deserves additional caution. REST access logs are separately configurable and valuable for long-term aggregation, but the execution log is especially useful during a live integration failure. Your goal is the error line, not an indefinitely verbose audit of customer submissions.

Jake: "I enabled a Lambda log group, but the gateway error still has no explanation there."

Ethan: "That may mean Lambda never ran. Your REST API execution log records the attempt to call it, including a denied invocation."

Capture the HTTP API Integration Error Message

If your API is an HTTP API, do not search for the REST execution log group just described. HTTP APIs support stage access logging instead of the same execution-log mechanism. Select the HTTP API, open Monitor and Logging, choose the stage, and enable access logging. Set a CloudWatch Logs destination and include $context.integrationErrorMessage in the format.

The log destination must be a CloudWatch log group you create or select, and the identity enabling logging needs suitable CloudWatch Logs permissions. Give the stage the destination ARN, which identifies the log group precisely. A JSON log format is useful because you can filter fields with CloudWatch Logs Insights. Include the request ID so a single browser report can be correlated with the gateway record.

{"requestId":"$context.requestId","status":"$context.status","routeKey":"$context.routeKey","integrationError":"$context.integrationErrorMessage","errorMessage":"$context.error.message","responseType":"$context.error.responseType"}

For a command-line configuration, create the destination log group and update the HTTP API stage with the appropriate apigatewayv2 command. The example below shows the access logging settings; for repeatable deployment, store these settings in a JSON file to avoid difficult shell escaping. The same settings are available in the console, which is often easier when you only need to inspect one stage. Replace the placeholder identifiers and save the file before invoking the CLI.

aws logs create-log-group --log-group-name /aws/apigateway/bookings-http --region us-east-1
aws apigatewayv2 update-stage --api-id abc123def4 --stage-name '$default' --access-log-settings 'DestinationArn=arn:aws:logs:us-east-1:123456789012:log-group:/aws/apigateway/bookings-http,Format={"requestId":"$context.requestId","status":"$context.status","integrationError":"$context.integrationErrorMessage"}' 

The example command is written for a Unix-like shell; Windows PowerShell and Command Prompt require their own quoting adjustments. If your log format command is difficult to quote, use a JSON file supplied through the appropriate CLI input mechanism or choose the console. The crucial requirement is the variable names, not the shell style. Once logging is on, send another request and open the latest CloudWatch log stream.

If the integration error is empty, add $context.error.message and $context.error.responseType to get more gateway-level context. If a Lambda authorizer is involved, include $context.authorizer.error as well. These variables apply to the relevant API type and failure layer; REST execution-log syntax is not interchangeable with HTTP API access-log syntax. Use the exact stage that receives the failed route.

Read Lambda Logs Only After Invocation Reaches the Function

When the gateway log indicates the Lambda invocation proceeded, open the function in the Lambda console and select Monitor and its CloudWatch log view. The conventional log group is /aws/lambda/function-name, unless the logging configuration uses a custom group. Locate entries around the failed request and inspect the stack trace, timeout report, and request ID. This establishes whether the handler really started and what happened during execution.

A handler can fail during initialization before it reaches your first application log statement. Missing dependencies, incompatible deployment artifacts, syntax errors, and configuration mistakes can all prevent normal handler execution. Logs may contain an initialization error rather than your expected application output. Treat that error as part of Lambda execution diagnosis, not as an API Gateway permission problem.

A function can also begin successfully and throw because the real API event differs from the saved test event. With Lambda proxy requests, event.body is commonly a string containing JSON rather than a nested object. A handler that accesses event.body.customer.name without parsing can raise a JavaScript error. Verify your payload format version and inspect the actual incoming event shape safely, without exposing customer data in logs.

export const handler = async (event) => {
  let data;
  try {
    data = typeof event.body === "string" ? JSON.parse(event.body) : (event.body || {});
  } catch {
    return { statusCode: 400, headers: { "Content-Type": "application/json" }, body: JSON.stringify({ message: "Invalid JSON body" }) };
  }
  if (!data.customer?.name) {
    return { statusCode: 400, headers: { "Content-Type": "application/json" }, body: JSON.stringify({ message: "Customer name required" }) };
  }
  return { statusCode: 200, headers: { "Content-Type": "application/json" }, body: JSON.stringify({ bookingAccepted: true }) };
};

That handler demonstrates controlled input errors rather than allowing a missing field to become an unhandled exception. A production booking handler should also handle database writes, idempotency, and application failure responses. Log unexpected errors with a correlation ID and omit secrets or raw sensitive bodies. The client should receive a useful but non-sensitive failure message rather than a runtime stack trace.

A Lambda timeout is different from a JavaScript exception. Review the configured Lambda timeout, function duration, and the API integration timeout to determine which boundary expired first. If the integration times out before Lambda finishes, the client may get 504 while the function may still complete later. A canceled or retried client request is not a safe signal that the business operation was undone.

Jake: "Lambda Test says green, so is the handler innocent?"

Ethan: "Only for that test event and invocation path. The real API can send a different body, use a different alias, or fail before the handler starts."

Apply the Resource-Based Policy That Lets API Gateway Invoke Lambda

If REST execution logs report Invalid permissions on Lambda function, focus on who is allowed to invoke the function. Lambda has a resource-based policy that can authorize the API Gateway service principal. That is separate from the Lambda execution role, which controls calls the running function makes to other AWS services. Granting the execution role more database permissions will not authorize an API Gateway invocation.

Open Lambda, select Configuration and Permissions, and inspect the resource-based policy. Look for lambda:InvokeFunction with principal apigateway.amazonaws.com and a source ARN matching your actual API, stage, method, and route. The permission might have been removed during deployment or added to another function name. Also inspect a qualified alias if the integration invokes one.

The one-line CLI correction is aws lambda add-permission. This example covers methods and paths within the specified API and is appropriate as a clearly scoped diagnostic starting point, not the narrowest final production policy. Replace the sample function, Region, account, and API ID. Choose a unique statement ID so the operation does not collide with an existing statement.

aws lambda add-permission --function-name booking-handler --statement-id AllowBookingApi --action lambda:InvokeFunction --principal apigateway.amazonaws.com --source-arn 'arn:aws:execute-api:us-east-1:123456789012:abc123def4/*/*/*' 

Once the failure is understood, restrict the scope to the intended stage and method where your deployment pattern allows it. A specific REST route may use an execution ARN such as arn:aws:execute-api:us-east-1:123456789012:abc123def4/prod/POST/bookings. The exact ARN depends on API type and route structure. An excessively narrow or mismatched ARN causes the same denied-invocation symptom as a missing statement.

Inspect the result using aws lambda get-policy --function-name booking-handler. If the integration points to booking-handler:prod, check the qualified resource policy with --qualifier prod as needed. Lambda aliases can carry invocation permissions relevant to that qualified target. A broad permission on a different function will not repair a stale integration URI.

An integration configured to assume an IAM credentials role has another authorization path to inspect: API Gateway must be allowed to assume that role and the role must permit the Lambda invocation. The common resource-based service-principal model and the role-based model should not be confused. Check the integration configuration before adding statements. Unnecessary permission changes can expand access without fixing the call path.

Jake: "If I can open and test the function myself, why would API Gateway need another permission?"

Ethan: "Your console test uses your access path, not the gateway service principal. Allow the correct API to invoke its backend; do not hand that privilege to every API."

Detect a Deleted Function, Incorrect Region, or Wrong Alias

When an integration breaks after a rename, rollback, or alias update, confirm the target ARN before changing code. The API method may still refer to the old function even though a newer function appears in your Lambda console. The Region and account in the integration URI matter as much as the function name. A successful direct test of another target cannot establish that the gateway target exists.

For a REST API, open Resources, select the method, and inspect its Integration request settings. The Lambda URI encodes the intended function invocation endpoint. For an HTTP API, inspect the integration attached to the route and its payload format version. Write down the exact target rather than relying on a similar display name.

aws apigateway get-integration --rest-api-id abc123def4 --resource-id a1b2c3 --http-method POST
aws apigatewayv2 get-integrations --api-id abc123def4
aws lambda get-function --function-name booking-handler --qualifier prod

An alias is a movable name pointing to a Lambda function version. If the alias was deleted, the qualified function ARN can become invalid even though the unqualified function still exists. If the alias points to an older version, a direct console test of the latest unpublished code may not reflect the deployed API. Inspect get-alias, the function version, and the policy applicable to the actual qualified invocation.

aws lambda get-alias --function-name booking-handler --name prod
aws lambda get-policy --function-name booking-handler --qualifier prod

Stage variables on REST APIs can select different function aliases by stage. Your development stage might resolve correctly while production points to an obsolete alias. Inspect the deployed stage variables and integration expression together. Correct the integration or variable, grant permission for the effective target, then deploy the REST API stage if its integration configuration changed.

Cross-account invokes add a further identity boundary. Verify that the function policy authorizes the correct invoking API, and inspect any credentials role configured for the integration. A wrong account number in a source ARN can look like a missing permission, while the actual cause is mismatched identity. Confirm both account IDs before editing policies or assuming a cross-account service defect.

Separate REST Custom Mapping Errors from Handler Failures

If your REST method uses a Lambda custom integration rather than a proxy integration, request and response mapping are additional failure points. An integration request mapping template can reshape incoming headers and JSON before Lambda sees them. An integration response mapping can choose status codes and body formats after Lambda returns. The two mappings solve different problems and leave different diagnostic evidence.

Request templates use Velocity Template Language, or VTL, to construct integration payloads. A template can produce an unexpected or incomplete request for your handler, leading to a subsequent function exception. By contrast, a broken integration response mapping can fail after the function finishes. Determine whether your failure happened before invocation, during Lambda execution, or while forming the method response.

Open the REST method and compare Integration request, Integration response, and Method response. One recognizable configuration failure is Output mapping refers to an invalid method response. That means an integration response is configured to target a method status response that is not declared as required. Repair that relationship and deploy the updated method rather than granting Lambda broader permissions.

A custom Lambda integration also has special error-mapping behavior. Lambda can report a function error while API Gateway returns HTTP 200 by default unless you configure an integration response selection pattern and corresponding method response. This is not the same as Lambda proxy behavior. Treat it as a response mapping problem and define how your API should present the backend error to consumers.

Mapping errors are easier to isolate with a fixed input that can be repeated safely. Compare the transformed payload with the shape the function expects, then inspect the configured response status and templates. A response template can accidentally hide a successful backend result or turn an error into a confusing status. Keep error messages concise and avoid copying stack traces into public responses.

If your API uses binary media types, review the content handling and transformation settings as well. An incompatible content encoding or unsupported mapping can become an API configuration failure rather than a business-logic exception. Check the relevant gateway error type and integration configuration for the specific content type. This diagnostic branch matters particularly when JSON requests work but another body format fails.

Jake: "Could the function finish the booking but the counter still fail to print the receipt?"

Ethan: "Yes, in a custom REST integration. The response mapping is another step after Lambda, so inspect it when the function ran but the method failed."

Understand Why Request Validation Usually Returns 400, Not 500

If the submitted JSON is missing a required field, a REST API request validator normally rejects it with HTTP 400. That validation occurs before Lambda integration and is not inherently a gateway 500. This matters because a malformed customer payload does not call for the same repair as a missing invocation policy. Check the actual status and the method request settings first.

REST request validation can require request parameters, validate the body against a model, or perform both checks. A model defines the JSON structure expected for a content type. Open the REST method, inspect Method request, the selected request validator, and any associated models. If the request violates that schema, correct the client input or the intended schema according to the API contract.

A Lambda handler can implement its own validation without using API Gateway request validation. If the handler throws on invalid JSON or a missing property, API Gateway may expose a 502 because it is a function error. Return a controlled application 400 instead so the frontend can distinguish bad input from an infrastructure fault. Log the validation decision without storing complete customer submissions unnecessarily.

Another trap is assuming an HTTP API has the same request-validator settings as a REST API. Their configuration models differ, and the REST method validator instructions do not translate directly. For HTTP APIs, inspect route authorization, integration payloads, and the application's own validation behavior. Use API-type-specific settings whenever you troubleshoot an input failure.

If a valid input fixes a 400 but the next request now yields a 500 or 502, that progression can be legitimate. The validator was the first blocker, and the integration error was previously hidden behind it. Continue by reading the new error line rather than revisiting the already corrected schema. One request can fail at only the first relevant gate it reaches.

Repair Malformed Proxy Responses and Payload Versions

If Lambda runs and completes but API Gateway returns 502, inspect the returned object before editing IAM. REST Lambda proxy integrations require a response envelope containing an HTTP status code and string body, with optional headers and Base64 information. Returning a raw object as the response body may violate the expected REST proxy contract. API Gateway then cannot turn the function output into the intended HTTP response.

The following structure is a valid ordinary JSON response shape for a REST API Lambda proxy integration. Notice that the outer object is a Lambda return value and body contains a serialized JSON string. The function did not have to throw an exception to generate a server error if it returned an invalid shape. Review the complete object rather than focusing only on the status field.

{
  "statusCode": 200,
  "headers": { "Content-Type": "application/json" },
  "isBase64Encoded": false,
  "body": "{\"message\":\"Booking accepted\"}"
}

HTTP API Lambda proxy integrations support payload format versions 1.0 and 2.0. Their incoming event fields and interpretation of returned responses differ. In version 2.0, API Gateway can infer a response from valid JSON output without every field used by a REST proxy envelope. Read the integration payload version before changing a handler shared by multiple API types.

A normally returned application error such as statusCode: 500 is distinct from malformed output. It can be a deliberate, well-formed response carrying a sanitized error message. That returned status may count as an API 5XX but not a Lambda execution Errors event if the function completed normally. Your monitoring must recognize this difference to avoid false reassurance from a zero Lambda Errors graph.

The next test should capture the exact return object in a controlled environment and compare it against the appropriate integration response contract. Pay attention to headers, serialization, body type, cookies, and Base64 handling as relevant to your payload version. The correct fix is to return a supported structure, not to add permissions to a function that already executed. Once fixed, test the deployed API rather than only the Lambda console.

Use the REST Console Test Without Mistaking It for Production

If Lambda passes its own console test, the next useful test is through API Gateway itself. For a REST API, select Resources, the method, and its Test tab. Provide the relevant headers, query parameters, and JSON request body, then choose Test. The result shows the method status, response body, headers, latency, and a simulated execution log.

The simulated logs are useful, but the method invocation can have real backend side effects. A test POST that creates a booking can still create a real booking. Use a harmless fixture or a safe staging destination and avoid repeatedly submitting payments or irreversible actions. This distinction matters because console output is simulated logging, not a harmless simulated Lambda invocation.

You can run the same REST method test with the CLI. Supply the API ID, resource ID, HTTP method, and body that match the method's contract. This tests the selected method, but it is not identical to sending a request through the actual deployed endpoint. It therefore belongs before a final real-client test, not in place of one.

aws apigateway test-invoke-method --rest-api-id abc123def4 --resource-id a1b2c3 --http-method POST --headers '{"Content-Type":"application/json"}' --body '{"customer":{"name":"Jake"}}' 

For HTTP APIs, call the deployed invoke URL using curl or a browser-compatible HTTP client. You may also use curl on REST APIs to confirm the deployed stage and any custom domain mapping. Include a real Origin header when diagnosing browser CORS behavior. A successful server-side curl request and a browser CORS failure are separate observations that deserve separate fixes.

curl -i -X POST 'https://abc123def4.execute-api.us-east-1.amazonaws.com/prod/bookings' -H 'Content-Type: application/json' -d '{"customer":{"name":"Jake"}}' 

If the test succeeds while production fails, compare stage deployment IDs, route configuration, authorizers, Lambda aliases, and environment-specific variables. The test can have a different request context from a live request. A deployment discrepancy is especially likely after a REST integration change that was saved but not deployed. Preserve both test outputs and diagnose only the configuration difference relevant to the failing route.

Jake: "If the method Test button shows 200, may I tell customers the website is fixed?"

Ethan: "Not yet. Send one request through the real deployed URL and check the business result. The console test is useful evidence, but it does not replace the customer path."

Worked Case: Trace a Blank 500 to Its One-Line Policy Fix

Consider a hypothetical booking endpoint with REST API ID abc123def4, stage prod, and route POST /bookings. Its configured function is booking-handler in account 123456789012, Region us-east-1. These identifiers are examples, not live resources. Jake sends a request containing a valid customer name and receives an HTTP 500 with a generic internal server error message.

The first check is the deployed integration, not the current Lambda console test. The integration URI points to the expected function, but no Lambda execution log appears for the failing request. The REST API execution log includes Execution failed due to configuration error: Invalid permissions on Lambda function. That error line narrows the case to invocation authorization, rather than a handler exception or response body problem.

Execution failed due to configuration error:
Invalid permissions on Lambda function
Method completed with status: 500

Next inspect the Lambda resource policy with get-policy and the actual integration target with get-integration. If the function policy does not allow this API to invoke the target, add the intended permission. Use a statement ID that is not already present. A simple explicit source ARN for this particular REST method is shown below.

aws lambda add-permission --function-name booking-handler --statement-id AllowProdBookingPost --action lambda:InvokeFunction --principal apigateway.amazonaws.com --source-arn 'arn:aws:execute-api:us-east-1:123456789012:abc123def4/prod/POST/bookings' 

The diagnostic loop is now precise: repeat the identical POST request and compare the new log record. If Lambda is invoked, the original permission boundary no longer blocks this path. If the function now throws, the result can change to 502, exposing a separate handler problem. If the API still reports invalid permissions, compare the effective function alias, source ARN, and integration invocation role before expanding the policy.

1. Capture the 500 response and request ID for the real deployed stage.
2. Read the REST execution log and find the invalid-permissions line.
3. Inspect the effective Lambda target and resource-based policy.
4. Add the matching invoke permission, scoped to the method or approved routes.
5. Repeat the same request and confirm the gateway can now invoke the function.
6. If a new error appears, classify it independently using its current log record.

An additional measurable result can be kept without claiming a live test occurred. Suppose this example had 200 attempts, of which 12 were blocked before Lambda; the API-visible failure rate would be 12 / 200 = 6%. If all 12 were invocation rejections, Lambda handler-error metrics would not count them as execution errors. That is why the API's 5XX metric and the Lambda Errors metric belong side by side.

The final check must include the business action. A successful HTTP response should correspond to the intended booking record, and an error response should not silently conceal a duplicate side effect. For write operations, use a request identifier or idempotency mechanism appropriate to your application. Fixing the invocation permission is the infrastructure repair, while confirming consistent business outcomes completes the customer-facing repair.

Explain Why API Gateway 500 Does Not Always Raise Lambda Errors

When your API 5XX chart rises but Lambda Errors remains flat, the metrics may both be accurate. API Gateway counts results at the HTTP endpoint boundary, while Lambda Errors tracks function invocations that resulted in execution errors. A gateway failure before the handler starts is not the same event as a running handler throwing. This is a measurement distinction, not evidence that one dashboard is broken.

A missing invoke permission, stale integration target, or gateway mapping failure can prevent a successful invocation. Such errors can reach the API 5XX metric without creating a Lambda function execution error. A handler that catches an exception and returns a valid HTTP 500 is also different: it may complete successfully at Lambda's runtime level. Your application should log or meter handled 500 responses separately if they matter operationally.

Use API Gateway 4XX and 5XX metrics for the endpoint, and Lambda Errors, Duration, Invocations, and Throttles for the function where applicable. If available, correlate per-route or per-stage dimensions to avoid combining healthy and unhealthy routes. CloudWatch metric periods can make a low-volume issue appear intermittent. Log timestamps and request IDs provide higher-resolution diagnostic evidence for a particular request.

A gateway 504 is not always a Lambda timeout. Gateway integration timeout and Lambda function timeout are separately configured boundaries. The gateway may stop waiting while Lambda still completes the invocation later, or Lambda may time out first and return an execution error. Compare the duration and log events to see which happened, especially when booking writes might continue after the client disconnects.

For an API used by customers, monitor the actual HTTP results together with the application outcome. A booking API that deliberately returns a well-formed 500 is still failing customers, even if Lambda execution errors remain at zero. Conversely, a handler may log a recoverable internal warning but return a correct 200. Design alerts around what customers see and whether the intended transaction completed.

Make Gateway and Lambda Error Responses Readable in Browsers

If curl reveals a 500 but the browser only reports a CORS failure, the backend problem and the browser access check are overlapping. Cross-Origin Resource Sharing controls whether scripts on a different origin may read an API response. A missing Access-Control-Allow-Origin header can hide an otherwise visible HTTP error from frontend code. Fix the underlying integration failure and the CORS behavior separately.

For REST APIs, gateway responses can customize errors produced by API Gateway itself. Open Gateway responses for the REST API and choose an appropriate response type such as DEFAULT_5XX. Add the required gatewayresponse.header.Access-Control-Allow-Origin mapping. Choose the specific origin allowed by your application instead of reflexively granting every origin access.

An easy-to-quote approach is to create a small file called gateway-cors.json containing the response header parameter map, then give it to the CLI. In the parameter map, the single quotation marks around the origin are part of the mapping expression value and must be preserved. This protects you from multiple layers of shell quoting. The example origin is illustrative and should be replaced.

{"gatewayresponse.header.Access-Control-Allow-Origin":"'https://shop.example.com'","gatewayresponse.header.Access-Control-Allow-Headers":"'Content-Type,Authorization'","gatewayresponse.header.Access-Control-Allow-Methods":"'POST,OPTIONS'"}

aws apigateway put-gateway-response --rest-api-id abc123def4 --response-type DEFAULT_5XX --response-parameters file://gateway-cors.json

For a REST Lambda proxy response that your handler constructs, include CORS headers in both success and handled-error responses as needed. A gateway response setting does not automatically rewrite every normal Lambda proxy response. Handle OPTIONS preflight and the requested methods consistently. Test with the actual browser Origin and the headers your frontend sends.

For HTTP APIs, use the API's built-in CORS configuration for allowed origins, headers, and methods. With that configuration enabled, API Gateway can add the configured CORS headers to integration responses and respond to preflight requests. It also ignores the backend's own CORS headers for those configured requests. Include an Origin header when testing, because that is needed to observe the configured CORS behavior.

Credentials change the design of your CORS configuration. If the frontend sends cookies or other credentials, an unrestricted wildcard origin is not the correct match for a credentialed browser request. Prefer a specific allowed origin and only the methods and headers your browser client uses. Keep your CORS change narrow, so debugging a gateway error does not leave you with a broad-access policy.

Resolve Throttling, Authorizers, Cross-Account Access, and Stale Stages

When the same route sometimes works and sometimes returns server errors, check for traffic-sensitive conditions before hunting for random syntax bugs. Lambda concurrency capacity can cause invocation throttling even when the handler code is correct. Inspect Lambda Throttles alongside API Gateway integration failures during the affected window. Increasing concurrency is appropriate only when the metrics and service quotas support that diagnosis.

A Lambda authorizer is another function that can fail before your integration handler starts. If API Gateway cannot invoke an HTTP API Lambda authorizer or receives an invalid authorizer response, the client can receive 500. Include $context.authorizer.error in HTTP API access logs and inspect the authorizer's Lambda resource policy and function logs. The failing function may be your authorizer, not your booking handler.

For cross-account arrangements, identify the API owner's account and Lambda function owner's account separately. The Lambda resource policy or the configured integration credentials role must authorize the actual access path. An ARN with the wrong account or Region may yield a symptom resembling a missing permission. Check service-specific cross-Region permission requirements rather than casually reusing a same-Region source ARN.

REST API deployments can preserve an earlier integration even after you edit its configuration. The stage's deployed snapshot and current editor state are not always the same. Review the deployment attached to the affected stage, then deploy approved changes to the correct environment. If development works but production does not, compare stage variables, aliases, authorizers, and permission scopes before merging unrelated configuration.

If you use a custom domain, confirm its API mapping and stage destination. A request may land on a different API than the one you inspected in the console. Test the direct execute-api URL and the custom domain with the same method and body, accounting for differences in authorization and routing. This quickly distinguishes domain mapping confusion from Lambda handler behavior.

Private networking may be relevant to what your Lambda function does after invocation, but it is not a generic explanation for an API Gateway invoke-permission error. If Lambda runs and then hangs on a dependency, inspect the function logs, timeout, and outbound connectivity. Keep backend network diagnosis separate from gateway-to-Lambda IAM authorization. The request ID and observed invocation status tell you which boundary to inspect first.

Calculate Failure Rates Without Inventing a Cloud Bill

If requests fail during a sales period, you need both an operational failure rate and a sensible cost view. API Gateway requests, Lambda execution time, and CloudWatch log storage are different usage categories. A rejected invocation does not automatically create billed handler execution, while a handler that actually runs can consume compute even when its response fails. An HTTP status alone cannot tell you the dollar charge.

Take an illustrative day with 2,000 booking requests, 50 API-level failures, and 30 Lambda execution errors. The endpoint failure rate is 50 / 2000 = 2.5%. If the other 20 errors were rejected before function execution, those 20 should be investigated in the gateway or invocation layer. This arithmetic illustrates why the application metric and function metric measure different populations.

For actual dollars, consult the applicable US East (N. Virginia) pricing pages and your account's billing and usage records for the specific API type, Lambda configuration, and CloudWatch usage. A REST API and an HTTP API do not necessarily have the same pricing model. Free usage and account-specific credits can also affect what is owed. Leaving an unverified dollar rate out is safer than creating a precise-looking but stale cost claim.

Detailed logging can add cost and expose more diagnostic data. Use enough verbosity to solve the issue, set retention appropriate to your environment, and avoid recording sensitive request bodies unnecessarily. Short-lived INFO logging during an investigation can be justified, but permanent verbose tracing of every customer payload is rarely required to diagnose a permission problem. Restore your intended production logging policy after the immediate issue is addressed.

The larger business cost can be uncertainty, not just AWS usage. A client seeing a 504 may retry even when a booking write later succeeds, and a handled HTTP 500 may represent either a rejected or partially committed operation. Use idempotency keys or equivalent application controls when repeated requests could create duplicate records. Monitor both customer-visible failures and final transaction outcomes.

Know What to Collect When the First Fix Does Not Work

If the gateway still returns 500 after a permission change, use the next request to find the new error, rather than declaring the change unsuccessful. The permission boundary might be gone and a handler exception could now be visible as 502. A different route might also be using an alias with a separate policy. Compare the latest request ID and error line with the original evidence.

Collect the AWS Region, API ID, API type, stage, route, timestamp in UTC, gateway request ID, returned status, and sanitized integration error. Include the effective Lambda ARN with alias or version, and the relevant policy statement or integration credentials role. If Lambda executed, capture its request ID and log excerpt. This collection gives another engineer a reproducible path through the failure without a risky production change.

1. Reproduce one failing request and capture its request ID and timestamp.
2. Inspect the correct API-type log format and confirm that the request reached the expected API and stage.
3. Compare the effective Lambda integration target and invocation permissions.
4. If Lambda ran, inspect the runtime logs, timeout, and proxy output.
5. Check deployment state, custom domains, authorizers, and cross-account roles only where evidence points to them.
6. Prepare sanitized request evidence and open a support case if a service behavior remains unexplained.

When logs appear blank, investigate permissions to publish logs and whether the correct stage was enabled. REST execution logging needs the API Gateway CloudWatch role, while HTTP API access logging needs a valid destination and suitable activation permissions. Some gateway failures may not yield every expected log or metric. In those cases, the request ID and response headers remain useful evidence for support.

AWS Support can help interpret service errors and investigate the gateway's behavior when you provide identifiers and timestamps. Support cannot automatically correct your application business logic, missing database data, or a deliberately restrictive IAM policy without appropriate configuration changes. Keep credentials, customer personal data, and unredacted authorization tokens out of ordinary diagnostic excerpts. A focused support case is easier to act on than a broad description that every request sometimes breaks.

After recovery, make a small runbook containing the exact log locations, authorized Lambda targets, and rollback steps for your environment. Add a test that invokes the deployed route, not only the Lambda console. A simple permission regression check or integration smoke test can catch future stale alias problems before customers see them. The aim is not to memorize every HTTP status but to identify the failing boundary quickly.

Jake: "If I fix the policy and then see 502, did I make things worse?"

Ethan: "Not necessarily. The new error can mean the gateway finally reached Lambda and uncovered a separate response problem. Read the newest log and solve that next."

Frequently Asked Questions About API Gateway Lambda 500 Errors

What causes API Gateway 500 Internal Server Error with Lambda?

Common causes include rejected Lambda invocation, missing invoke permission, invalid integration targets, gateway configuration failures, and REST integration response mapping problems. A handler exception is more commonly surfaced as 502 in a proxy integration, so first inspect the integration error.

How can I tell an API Gateway 500 from a 502?

An invocation rejected by Lambda can appear as 500, while an invoked Lambda function that fails or returns an invalid proxy result commonly produces 502. The client body may look identical, so correlate the status with gateway and function logs.

How is an API Gateway 504 different from 500 and 502?

A 504 indicates the integration did not provide a response before the gateway timeout. Examine integration latency and Lambda duration; backend work may continue even after the gateway stops waiting.

Why does the Lambda console test work while API Gateway fails?

A console test uses a different caller and potentially a different event, alias, or version. API Gateway also depends on its deployed integration, invocation permission, request mapping, and authorization configuration.

Where is the real API Gateway Internal server error log line?

REST API execution logs expose configuration and integration execution failures. HTTP API access logs should include $context.integrationErrorMessage, and Lambda function logs contain runtime errors when the handler actually runs.

How do I turn on execution logging for a REST API?

Configure the regional API Gateway account CloudWatch role and enable execution logging in the REST API stage under Logs and tracing. The managed execution log group then uses the API-Gateway-Execution-Logs API and stage naming pattern.

Why do HTTP API logs not have REST execution traces?

HTTP APIs provide access logging instead of the same REST execution logging facility. Include $context.integrationErrorMessage and appropriate request identifiers in the access log format to diagnose the integration.

What is the one-line fix for Invalid permissions on Lambda function?

Use aws lambda add-permission with lambda:InvokeFunction, principal apigateway.amazonaws.com, and a source ARN that matches your API. If the integration assumes a credentials role or invokes an alias, inspect that exact authorization path too.

Does an API Gateway 500 increment the Lambda Errors metric?

Not always. A rejected invocation does not count as a function execution error, and a normally returned application HTTP 500 need not increment Lambda Errors. Monitor gateway 5XX, Lambda Errors, and Throttles together.

How do I test a REST API Lambda integration from the console?

Open Resources, select the method, and use its Test tab with the appropriate request body and headers. The displayed logs are simulated but the backend invocation can have real side effects, so confirm with a safe real deployed-stage request afterward.

Can an old Lambda alias or deleted function cause 500?

Yes, the gateway may point to a function or qualified alias that no longer exists or is not authorized. Compare the full effective integration target ARN with the Lambda function and alias configuration.

Can a REST API mapping template cause an internal server error?

Yes, a broken integration mapping or undefined method response can cause a gateway configuration error. A request template can also send malformed or unexpected data to Lambda, producing a separate handler error.

Does an API Gateway request validator return 500 for invalid JSON?

A normal REST request validation failure returns 400 before calling the integration. If invalid JSON instead reaches Lambda and causes an uncaught exception, that is a different execution failure.

Why does API Gateway say Malformed Lambda proxy response?

A REST Lambda proxy integration expects a valid status and serialized body response envelope. HTTP API integrations have different payload format versions, so inspect the configured version before changing the returned object.

How do I add CORS headers to API Gateway error responses?

For REST APIs, customize gateway responses for gateway-generated errors and include headers in Lambda proxy responses for handled errors. For HTTP APIs, configure built-in CORS and test using an Origin request header.

Why is there no Lambda log entry for my failed API call?

The failure may have occurred before execution because invocation was denied, the target was invalid, or an authorizer failed. Check gateway logs, the effective function target, and logging configuration before assuming the handler ran.

If your booking endpoint or production integration fails again, start with one request ID and its actual integration error before making another deployment. A readable gateway log, a narrowly scoped invoke policy, and a real deployed-route test are more useful than a dozen speculative code changes. I hope your next error message gives you a specific fix rather than another afternoon of guesswork.

Revision note. Written October 17, 2026. "The devil is in the details," the proverb says; with API Gateway, the details are in the execution log.