CloudWatch Logs Subscription Filter to Lambda: Setup, the Permission That Breaks It, Filter Patterns, the 5-Filter Limit, the 10-Minute Disable, Recursion Loops and Every "Not Triggering" Cause

A CloudWatch Logs subscription filter sends matching log events from a log group to a Lambda function, a Kinesis data stream, a Firehose stream or an OpenSearch domain within about three minutes of the line being written. For Lambda it takes two commands: aws lambda add-permission so CloudWatch Logs may invoke the function, then aws logs put-subscription-filter with the log group, a filter pattern and the function ARN. The payload is gzip-compressed and base64-encoded under awslogs.data, and your handler has to decode it before it sees a single log line. Here is the part nobody tells you until it costs money: when the destination returns a non-retryable error such as AccessDeniedException, CloudWatch Logs switches the filter off for up to 10 minutes and skips every log event in that window, with no alarm unless you built one on the DeliveryErrors metric. This page sets the filter up properly, decodes the payload with a handler we ran locally, explains the five-filter limit, the account-level alternative, the recursion loop that doubles your ingestion bill, and every reason a filter sits there and never triggers.

Jake runs a phone repair shop with a small booking app on AWS, and he wanted a Slack message whenever the app logged payment declined. His friend Ethan, a developer, said that's a subscription filter and a ten-line Lambda function, and it was, except for the hour in between: the filter said it was created, the function never ran, the test button said the pattern matched, and the only clue was a metric neither of them had heard of. This page is the version of that hour with the detours removed. If you need a count or an alarm rather than the log lines themselves, you want a metric filter instead; our metric filter guide covers that side.

What a CloudWatch Logs subscription filter is, and how it differs from a metric filter

A log group in CloudWatch Logs holds log events; a subscription filter attached to it watches every new event, compares it with a filter pattern, and forwards the matches to one destination: a Lambda function, a Kinesis Data Streams stream, a Firehose stream, or an OpenSearch Service domain. Delivery is near real time, usually under three minutes after ingestion, at least once (duplicates are possible), and CloudWatch batches several events into one delivery when it can. The forwarded data is always gzip-compressed and base64-encoded, whichever destination you choose.

A metric filter uses the same pattern syntax but produces a number, a CloudWatch metric you can graph and alarm on; it never hands you the log lines. The rule of thumb: count it with a metric filter, process it with a subscription filter. Many teams run both on the same log group, which is fine because the two don't share a quota.

Two constraints decide early whether this feature is available to you at all. Subscriptions work only on log groups in the Standard log class; a group created in the Infrequent Access class to save money can't have one, and the console simply doesn't offer it. And if the log group has a log transformer, the pattern is matched against the transformed event, not the raw line your application wrote.

Jake: "So the thing that counts errors and the thing that sends me the error are two different filters?"

Ethan: "Same pattern language, different jobs. You'll end up with both: a metric filter for the alarm, a subscription filter for the message with the details."

Set up a subscription filter to Lambda in four commands

The sequence is: function, permission, filter, test. Use the handler below as the function; it's the one we ran locally while writing this page.

1. Create the function. Python 3.13, the handler below, an execution role with the basic Lambda logging policy. Deploy it as you normally would; nothing about it is special.
2. Let CloudWatch Logs invoke it. This is the step people skip, because for Lambda there's no IAM role to create; the permission lives on the function itself:

   aws lambda add-permission \
     --function-name log-forwarder \
     --statement-id cwl-orders \
     --principal logs.amazonaws.com \
     --action lambda:InvokeFunction \
     --source-arn "arn:aws:logs:us-east-1:111122223333:log-group:/aws/lambda/orders:*" \
     --source-account 111122223333

   The :* at the end of the source ARN is required: the log group's ARN ends with :*, and a source ARN without it never matches.
3. Create the filter.

   aws logs put-subscription-filter \
     --log-group-name /aws/lambda/orders \
     --filter-name errors-to-lambda \
     --filter-pattern '{ $.level = "ERROR" }' \
     --destination-arn arn:aws:lambda:us-east-1:111122223333:function:log-forwarder

   An empty pattern ("") forwards every event. There's no --role-arn for a Lambda destination; you'll need one for Kinesis and Firehose.
4. Test with a real event, not the console's test button. Write one line into the log group and watch the function's own log group:

   aws logs put-log-events --log-group-name /aws/lambda/orders --log-stream-name manual-test \
     --log-events timestamp=$(date +%s000),message='{"level":"ERROR","msg":"payment declined","order":1001}'
   aws logs tail /aws/lambda/log-forwarder --since 5m

   Create the manual-test stream first with aws logs create-log-stream if it doesn't exist. Expect a delay of a few seconds to a couple of minutes before the forwarder logs anything.

In the console the same thing is Log groups, your group, the Subscription filters tab, Create, Create Lambda subscription filter; the console adds the resource policy for you. CloudFormation and Terraform versions are in the infrastructure-as-code section.

What arrives at the function

The event is a single key, awslogs, whose data is base64 text that decompresses to this JSON:

{
  "owner": "111122223333",
  "logGroup": "/aws/lambda/orders",
  "logStream": "2026/10/24/[$LATEST]abc",
  "subscriptionFilters": ["errors-to-lambda"],
  "messageType": "DATA_MESSAGE",
  "logEvents": [
    {"id": "36000000000000000000000000000000000000000000000000000000", "timestamp": 1792832400000,
     "message": "{\"level\": \"ERROR\", \"msg\": \"payment declined\", \"order\": 1001}"}
  ]
}

Each message is the raw log line as a string; if your application logs JSON, you parse it a second time. messageType is DATA_MESSAGE for real data and CONTROL_MESSAGE for the health check CloudWatch sends to see whether the destination is reachable. The compression is not cosmetic: when we packed 201 JSON log events into one event, 36,543 bytes of JSON became 2,656 bytes on the wire, about 7 percent, so a batch that looks tiny in the payload can hold hundreds of lines.

"Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function."

This InvalidParameterException comes back from put-subscription-filter itself, which is good news: it means CloudWatch Logs tried to invoke the function during creation and was refused, so the problem is entirely in the function's resource-based policy. Check what's there:

aws lambda get-policy --function-name log-forwarder --query Policy --output text | python3 -m json.tool

You need a statement with Principal logs.amazonaws.com (the Regional form logs.us-east-1.amazonaws.com also works), Action lambda:InvokeFunction, and a SourceArn condition that covers the log group. The failures, in the order we see them:

• Source ARN without :*. arn:aws:logs:us-east-1:111122223333:log-group:/aws/lambda/orders looks right and never matches, because the resource that invokes is ...:log-group:/aws/lambda/orders:*. In Terraform, use "${aws_cloudwatch_log_group.orders.arn}:*", since the group's arn attribute omits the suffix.
• One permission for several log groups. Each log group needs its own statement, or one statement whose source ARN is a wildcard that covers them all (log-group:/aws/lambda/*:*). Frameworks that generate one permission per function and then attach several log groups hit this; the first group works and the second returns this error.
• Different Region. The log group and the function must be in the same Region. The destination ARN tells you where the function is; the log group is wherever you're running the command.
• Wrong account in --source-account. It must be the account that owns the log group; a cross-account forward to Lambda isn't supported directly, so this is always your own account.
• Function doesn't exist yet. In CloudFormation, the filter must depend on the permission, which must depend on the function; without DependsOn the stack sometimes creates the filter first and fails with this message.

Delete and recreate the permission after fixing it (remove-permission with the statement ID, then add-permission), then run put-subscription-filter again; it's idempotent on the filter name.

Jake: "My permission had the log group name exactly right. The only thing missing was a colon and a star."

Ethan: "That colon-star has cost more afternoons than any other two characters in CloudWatch."

Subscription filter not triggering Lambda: the checklist, in order

The filter exists, no error was returned, and the function shows zero invocations. Work down this list; each step takes a minute.

1. Check the forwarding metrics first. CloudWatch Logs publishes three metrics per filter in the AWS/Logs namespace: ForwardedLogEvents, DeliveryErrors and DeliveryThrottling, with dimensions LogGroupName, DestinationType and FilterName. If ForwardedLogEvents is zero, nothing matched or nothing arrived; if it's positive and the function didn't run, look at DeliveryErrors.

   aws cloudwatch get-metric-statistics --namespace AWS/Logs --metric-name DeliveryErrors \
     --dimensions Name=LogGroupName,Value=/aws/lambda/orders Name=DestinationType,Value=Lambda Name=FilterName,Value=errors-to-lambda \
     --start-time $(date -u -d '-1 hour' +%FT%TZ) --end-time $(date -u +%FT%TZ) --period 60 --statistics Sum

2. The 10-minute disable. A non-retryable error from the destination (AccessDeniedException, ResourceNotFoundException, a deleted function, a revoked permission) makes CloudWatch Logs disable the filter for up to 10 minutes, then try again. Log events during that window are skipped, not queued. A permission that was fixed five minutes ago can still look broken; wait out the window, then test again. Retryable errors (throttling, 5xx) are different: those are retried for up to 24 hours.
3. Pattern mismatch. JSON patterns need the event to be valid JSON from the first character; a Lambda log line such as 2026-10-24T10:00:00Z abc-123 INFO {"level":...} starts with a timestamp, so { $.level = "ERROR" } never matches it. Use a text term (ERROR) or structured logging that writes pure JSON. The Test pattern box in the console runs your pattern against real recent events; believe it over your eyes.
4. Wrong log group. A function named orders logs to /aws/lambda/orders; an ECS task logs wherever its log configuration says; API Gateway access logs go to the group you named in the stage. Open the group you subscribed and confirm new events are arriving at all.
5. Infrequent Access log class. If the group was created with the IA class, subscriptions aren't available. There's no in-place conversion; create a Standard group and point the producer at it.
6. The function is throttled or has reserved concurrency 0. CloudWatch Logs invokes Lambda asynchronously, so Lambda queues the event and retries on throttles; a function with reserved concurrency set to zero accepts nothing, and the events age out of the queue. Check the function's Throttles metric.
7. The handler crashes on the control message. The first thing a new filter sends is a CONTROL_MESSAGE with no useful logEvents; a handler that assumes logEvents[0] is JSON throws, the invocation errors, and nothing you see explains it. The handler below skips it.
8. Transformed logs. With a log transformer on the group, the pattern is matched against the transformed event. A transformer that renames level to severity breaks a pattern that still says $.level.
9. Delay, not failure. Delivery is "usually under three minutes," and the first delivery after creating a filter can take longer. Test with a burst of events, wait five minutes, and only then conclude it's broken.

Set an alarm on DeliveryErrors (Sum greater than 0 over 5 minutes) for every production filter. It's the only signal for the disable window, and our alarm-to-email guide covers the SNS side.

Filter pattern examples that work for Lambda, and the regex limits

Subscription filters use the same pattern syntax as metric filters. The patterns people need most:

Regex has quotas that bite on busy groups: at most 5 filter patterns containing regex per log group, counted across metric filters and subscription filters together, and at most 2 regex expressions inside one pattern. The regex dialect is small (no lookarounds, a short list of symbols), and a dot in an IP address must be escaped. A reader who wanted to forward only the lines of a Node.js Lambda that say info and carry an object with an id and a projectId asked for the right pattern: if the function logs JSON, { $.level = "info" && $.id = * && $.projectId = * } does it; if it logs with console.log and a prefix, the event isn't JSON and a term pattern (info projectId, both words required) is the honest option until the logging is structured.

Another reader wanted to know which part of the event matched, so an alert could say why. The payload doesn't tell you; it lists the subscriptionFilters that matched, not the term. The pattern is to create one narrow filter per reason (root-login, failed-mfa) within the five-filter limit, or one broad filter and re-match in the handler with your own regexes, which is also where you get the matched text.

The 5-filter limit per log group (it was 2), LimitExceededException, and the account-level filter

For years a log group could have two subscription filters, and the error on the third was LimitExceededException: Resource limit exceeded. The quota is now five per log group and is not adjustable; the sixth still returns that error. A team that fans out one CloudTrail log group to Slack, a SIEM and an archive used to hit the wall immediately; five covers most fan-outs, and when it doesn't, the pattern is one subscription to a Kinesis data stream, which can have many consumers, or one Lambda that re-publishes to an SNS topic with several subscriptions.

The other route is an account-level subscription filter: one policy per Region that applies to every log group in the account (or a selection), with the same destinations:

aws logs put-account-policy \
  --policy-name all-errors-to-lambda \
  --policy-type SUBSCRIPTION_FILTER_POLICY \
  --policy-document '{"DestinationArn":"arn:aws:lambda:us-east-1:111122223333:function:log-forwarder","FilterPattern":"ERROR"}' \
  --selection-criteria 'LogGroupName NOT IN ["/aws/lambda/log-forwarder", "/aws/vpc/flowlogs"]' \
  --scope ALL

aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY

Points that matter: one account-level filter per Region; the account-level filter doesn't count against the five on each group; selection-criteria is optional but is the only thing standing between you and the recursion loop in the next section, and it has a 25 KB size limit; and the metrics for it carry a PolicyLevel=AccountPolicy dimension instead of LogGroupName. The Lambda still needs the resource-based permission, this time with a source ARN that covers every group (log-group:*).

The recursion loop: how a subscription filter doubles your ingestion bill

A subscription filter on the wrong log group creates an infinite loop: the function processes a batch, logs a line about it, the line lands in the function's own log group, the filter forwards it, the function runs again, logs again, and so on. Every pass bills CloudWatch Logs ingestion, Lambda invocations and whatever the destination charges, and it runs until someone notices the bill or the concurrency alarm. The same loop happens through a Firehose to S3 destination when a Lambda processes the S3 objects and logs to a subscribed group, and through VPC flow logs when the processing function sits in a VPC whose flow-log group is subscribed.

The rules that prevent it:

• Never subscribe the log group of the function, task or state machine that is part of the delivery path. For log-group-level filters this is entirely on you; CloudWatch Logs won't refuse the filter.
• For account-level filters, exclude those groups with selection-criteria; the services to think about are Lambda, ECS on Fargate, Step Functions and VPC flow logs. The one built-in safeguard: an account-level policy doesn't forward the destination Lambda's own log group back to it, so that one exclusion isn't needed there.
• Give the forwarder a short log retention and a reserved concurrency ceiling, so a mistake is bounded by the ceiling instead of the account limit.
• Put a billing alarm on IncomingBytes for the forwarder's log group. A loop shows up there within minutes.

If it's already happening, delete the subscription filter (aws logs delete-subscription-filter --log-group-name ... --filter-name ...), then set the function's reserved concurrency to zero while the queued async invocations drain. Our S3 recursive loop post walks through the cleanup and the bill conversation with AWS Support.

Jake: "So the forwarder must never watch itself."

Ethan: "And nothing that watches the forwarder. The loop can go through S3 or a VPC flow log and still come back to the same function."

The Lambda handler: decode, skip control messages, expect duplicates

This is the Python handler we tested locally with the Lambda Runtime Interface Emulator; it decodes the payload, skips the control message, parses JSON lines and keeps plain-text lines as text:

import base64, gzip, json

def handler(event, context):
    payload = json.loads(gzip.decompress(base64.b64decode(event['awslogs']['data'])))
    if payload.get('messageType') != 'DATA_MESSAGE':
        return {'skipped': payload.get('messageType')}          # health check, nothing to process
    out = []
    for e in payload['logEvents']:
        try:
            msg = json.loads(e['message'])                       # JSON log lines
        except ValueError:
            msg = {'text': e['message']}                         # plain-text log lines
        out.append({'ts': e['timestamp'], 'group': payload['logGroup'], **msg})
    print(json.dumps({'events': len(out), 'filters': payload['subscriptionFilters']}))
    return {'processed': len(out), 'first': out[0] if out else None}

Fed a 201-event data message, it returned {"processed": 201, "first": {"ts": 1792832400000, "group": "/aws/lambda/orders", "level": "ERROR", "msg": "payment declined", "order": 1001}}; fed a control message, it returned {"skipped": "CONTROL_MESSAGE"} instead of crashing. To run the same test on your machine, build the event with a few lines of Python (JSON, gzip, base64, wrap in awslogs.data) and post it to the emulator; our local Lambda testing guide has the container commands.

Four things to build in before production:

• Asynchronous invocation. CloudWatch Logs invokes the function asynchronously: Lambda queues the event, returns success to CloudWatch immediately, and retries the function twice on error by default. Configure an on-failure destination (SQS or SNS) on the function so batches that fail three times aren't lost; CloudWatch Logs itself won't resend a batch Lambda accepted. Our Lambda destinations guide covers the setup.
• Duplicates. Delivery is at least once. Key your downstream writes on the event id, which is unique per log event, so a retried batch doesn't create two Slack messages or two rows.
• Batches, not lines. One invocation can carry one event or hundreds; a handler that calls an external API once per event will be throttled on a busy group. Aggregate per batch, then send.
• Log less than you receive. The forwarder's own log lines are ingestion you pay for a second time. One summary line per batch, as above, is plenty.

CloudFormation, Terraform and Serverless: the subscription filter as code

The same three resources in each tool, with the dependency order that avoids the permission error.

CloudFormation

Resources:
  ForwarderPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref LogForwarder
      Action: lambda:InvokeFunction
      Principal: logs.amazonaws.com
      SourceAccount: !Ref AWS::AccountId
      SourceArn: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/orders:*
  ErrorsToLambda:
    Type: AWS::Logs::SubscriptionFilter
    DependsOn: ForwarderPermission
    Properties:
      LogGroupName: /aws/lambda/orders
      FilterName: errors-to-lambda
      FilterPattern: '{ $.level = "ERROR" }'
      DestinationArn: !GetAtt LogForwarder.Arn

Terraform

resource "aws_lambda_permission" "cwl" {
  statement_id   = "cwl-orders"
  action         = "lambda:InvokeFunction"
  function_name  = aws_lambda_function.forwarder.function_name
  principal      = "logs.amazonaws.com"
  source_account = data.aws_caller_identity.current.account_id
  source_arn     = "${aws_cloudwatch_log_group.orders.arn}:*"
}

resource "aws_cloudwatch_log_subscription_filter" "errors" {
  name            = "errors-to-lambda"
  log_group_name  = aws_cloudwatch_log_group.orders.name
  filter_pattern  = "{ $.level = \"ERROR\" }"
  destination_arn = aws_lambda_function.forwarder.arn
  depends_on      = [aws_lambda_permission.cwl]
}

For a Kinesis or Firehose destination, add role_arn (Terraform) or RoleArn (CloudFormation): the subscription filter IAM role, one that trusts logs.amazonaws.com and may kinesis:PutRecord or firehose:PutRecord on the stream, and set distribution = "Random" for Kinesis if one log stream dominates. In the Serverless Framework, a cloudwatchLog event on the function creates the filter and the permission; declare one event per log group and check the generated template, because older versions produced a single permission statement that covered only the first group.

Kinesis, Firehose and OpenSearch destinations: when Lambda is the wrong choice

Lambda is the right destination for processing: enrich, alert, call an API. For moving logs, pick by volume and target:

• Firehose when the target is S3, OpenSearch, Splunk or an HTTP endpoint and you want buffering, batching and compression done for you. One role for CloudWatch Logs to write to Firehose, another for Firehose to write to S3. The S3 objects are gzip-compressed and hold the same JSON structure as the Lambda payload, one record per batch.
• Kinesis Data Streams when several consumers need the same feed, when you need replay, or when you've hit the five-filter limit. Size the shards for the log volume; throttled deliveries are retried for 24 hours and then dropped, and DeliveryThrottling tells you it's happening. Our Kinesis pricing guide has the shard math.
• OpenSearch Service through the console's "Subscription filter to OpenSearch" wizard, which creates a Lambda behind the scenes to do the indexing. It's convenient and it's still a Lambda, with the same permission and recursion rules; our OpenSearch guide covers sizing the domain.
• Cross-account. A Lambda in another account can't be a destination directly. Create a CloudWatch Logs destination (a wrapper around a Kinesis or Firehose stream) in the receiving account, with a destination policy that allows the sending account, in the same Region as the source log group; the stream behind it can live anywhere. Account-level filters support the same arrangement.

What a subscription filter costs

There's no per-filter fee and no per-event forwarding charge on the CloudWatch Logs side; you've already paid ingestion for the events. The bill arrives at the destination: Lambda requests and duration, Kinesis shard hours or on-demand throughput, Firehose per gigabyte, OpenSearch instance hours. A rare-event filter to Lambda, the case a reader asked about, costs effectively nothing: a few invocations a day inside the free tier. The expensive mistakes are forwarding everything from a chatty group into Lambda (one invocation per batch, all day) and the recursion loop above, which bills ingestion twice.

CloudWatch Logs subscription filter to Lambda: frequently asked questions

Jake's Slack message arrived forty minutes after the colon-star went into the permission, carrying the order number and the decline reason, and the second message five minutes later was identical, which is how he learned about at-least-once delivery before Ethan could explain it. The forwarder now keys on the event ID, logs one line per batch to a group with a three-day retention, and has a DeliveryErrors alarm that has fired once, on the day someone renamed the function. Ethan wrote the line below in the runbook.

Revision note. Written October 24, 2026, for everyone whose filter said "created" and whose function said nothing. "A chain is only as strong as its weakest link," says the old proverb; here the link is two characters long.