What Is AWS Lake Formation? How It Works, Pricing, vs Glue and S3, LF-Tags, and "Insufficient Lake Formation Permission(s)" Fixed

Logeshwaran
—

AWS Lake Formation is the permissions layer for a data lake on AWS. Your data stays in Amazon S3 and its tables are described in the AWS Glue Data Catalog; Lake Formation decides who can see which databases, tables, columns and even rows, and enforces it in Athena, Redshift Spectrum, EMR, Glue and QuickSight, with one set of grants instead of a tangle of S3 bucket policies. Managing permissions in Lake Formation has no separate charge: you pay for S3, the Glue Data Catalog and the query services. The error everyone meets is "Insufficient Lake Formation permission(s)": the principal has IAM permission but no Lake Formation grant on that database or table (Required Describe, Select, Create Table, Drop), or no data location permission on the S3 path, or a crawler's role cannot create tables. Fix it by granting the missing permission in Lake Formation to that role (as a data lake administrator), or, for tables not yet under Lake Formation control, by keeping the IAMAllowedPrincipals grant (hybrid access). Below: how Lake Formation works, setup step by step, LF-Tags, row and column security, cross-account sharing, Terraform, Lake Formation vs Glue vs S3, and every permission error with its fix.

The shop's sales data lived in S3 and Athena for two years, with access controlled by bucket policies that only Ethan understood. When a new analyst needed sales totals but not customers' email addresses, the bucket policy approach ran out of road: S3 thinks in files, not columns. Ethan turned on Lake Formation, granted the analyst every column except two, and the next morning the analyst's Athena query failed anyway, with "Insufficient Lake Formation permission(s): Required Describe on sales". The grant was right; the database permission was missing. Lake Formation had done exactly what it promised: nothing that was not granted.

⚡ Quick Answer

• What it is: database, table, column and row permissions for data in S3. How it works.

• Cost: no charge for permissions; you pay for S3, Glue and query services. Pricing.

• "Insufficient Lake Formation permission(s)": grant the named permission, or data location access. Every variant.

• Getting started: admin, register the S3 location, grant. Steps.

IAMAllowedPrincipals · LF-Tags · Row and column security · Cross-account · vs Glue and S3.

New to data lakes on AWS? The cards below cover S3, querying it with Athena, IAM policies and what Athena costs.

🧭 NEW HERE? READ THESE FIRST

New to data lakes, Athena or IAM? These five pages give you the background this one builds on:

📌 Bookmark this; the permissions table and the error list are the parts you will come back for.

What AWS Lake Formation is, and how it works

A data lake on AWS has three layers. The files (CSV, Parquet, JSON) sit in S3. The Glue Data Catalog holds the table definitions: which S3 folder is the "sales" table, what its columns are, how it is partitioned. And the query services (Athena, Redshift Spectrum, EMR, Glue jobs, QuickSight) read the catalog to know what to scan. Our guide to querying S3 with Athena covers those first two layers.

Without Lake Formation, access is controlled with IAM and S3 policies: anyone allowed to read the S3 files and the catalog entry can read the whole table. Lake Formation adds a permissions layer on the catalog:

  • You register an S3 location with Lake Formation. From then on, Lake Formation hands the query services temporary credentials to read it on behalf of users who have been granted access.
  • You grant permissions on catalog objects to IAM users, roles, Identity Center users, or other accounts: DESCRIBE and CREATE_TABLE on databases; SELECT, INSERT, DELETE, ALTER and DROP on tables; with optional column lists and row filters.
  • The query services enforce those grants. An analyst granted three columns sees three columns in Athena, whatever their S3 permissions say.

The result is one place to answer "who can see what" for the whole lake, at the level analysts and auditors think in: tables, columns and rows rather than buckets and prefixes.

Jake: "So S3 is the warehouse, the catalog is the index card, and Lake Formation is the person at the door?"

Ethan: "And the person at the door can let someone read only some of the lines on the card."

Lake Formation architecture: what happens when someone runs a query

Architecture diagrams of Lake Formation look busy, but the path of a single query is short. When the analyst runs SELECT * FROM sales.orders in Athena:

  1. Athena asks the Glue Data Catalog for the table's definition: its columns and the S3 location of its files.
  2. Lake Formation checks the grants for the analyst's role on that database and table, including any column list and row filter.
  3. Lake Formation hands Athena temporary credentials scoped to the table's registered S3 location. This "credential vending" is why the analyst's own role needs no S3 permissions at all, only lakeformation:GetDataAccess.
  4. Athena reads the files with those credentials and applies the column and row limits before returning results.
  5. The access is logged in CloudTrail, including which principal was given data access to which table.

Two consequences follow. First, if a table's S3 location is not registered with Lake Formation, there is nothing to vend credentials for, and the query service falls back to the caller's own IAM and S3 permissions. Second, the services that enforce Lake Formation are the ones that ask it: Athena, Redshift Spectrum, EMR, Glue and QuickSight. Someone with direct S3 permissions on the bucket can still download the raw files with the AWS CLI, which is why a governed lake keeps bucket access for people narrow and lets Lake Formation's registered role do the reading.

The permissions you will use

PermissionOnLets the principal
DESCRIBEDatabase, tableSee that it exists and read its definition (needed before anything else)
SELECTTable (all or some columns, optional row filter)Query the data
CREATE_TABLEDatabaseCreate tables in it (crawlers and ETL jobs need this)
ALTERDatabase, tableChange the definition (new partitions, columns)
INSERT, DELETETableWrite or remove data through integrated services
DROPDatabase, tableDelete it from the catalog
DATA_LOCATION_ACCESSRegistered S3 locationCreate or alter tables that point at that S3 path
SUPER / ALLDatabase, tableEvery permission on that object

Any of these can be given with grant option, which lets the receiver pass the permission on, the way you share with another account's data lake administrator. Your users still need IAM permissions to call the services (Athena, Glue's catalog API, lakeformation:GetDataAccess); Lake Formation decides what data they get once they are allowed in. Our IAM policy explainer covers that side.

Setting up Lake Formation, step by step

  1. Make yourself a data lake administrator. Open Lake Formation in the console; the first visit asks you to add administrators. Admins can grant any permission and see every resource. You can also add read-only administrators, who see all permissions but cannot change them, which suits auditors.
  2. Check the default settings. Under Data Catalog settings, decide whether new databases and tables should keep "Use only IAM access control" (the old behavior) or be governed by Lake Formation from the start.
  3. Register the S3 location that holds the data (Data lake locations, Register location), using the Lake Formation service-linked role or a role of your own that can read the bucket.
  4. Create or crawl the tables into a Glue database, if they are not catalogued yet. The crawler's role needs CREATE_TABLE on the database and data location access on the S3 path.
  5. Grant permissions: Data lake permissions, Grant, choose the principal, the database or table, the columns, and the permissions (for example DESCRIBE on the database plus SELECT on the table).
  6. Move existing tables to Lake Formation control by removing the IAMAllowedPrincipals grant from them once the new grants are in place (next section).
  7. Test as the user: run their Athena query with their role before telling them it is ready.

The same grant from the CLI:

# Let the analyst role see the database
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst \
  --permissions DESCRIBE \
  --resource '{"Database": {"Name": "sales"}}'

# Let it query every column except two
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst \
  --permissions SELECT \
  --resource '{"TableWithColumns": {"DatabaseName": "sales", "Name": "orders",
               "ColumnWildcard": {"ExcludedColumnNames": ["customer_email", "customer_phone"]}}}'

# What does this role have?
aws lakeformation list-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst

Tutorial: one governed table, start to finish

The quickest way to understand Lake Formation is to put one small table under it and watch the grants work. This walk-through uses a CSV of orders and an analyst role; it takes about twenty minutes and costs a few cents in Athena scans.

  1. Put the data in S3. Upload orders.csv to s3://shop-lake/sales/orders/, with columns such as order_id, store_id, total, customer_email.
  2. Create the database and table in Athena as an administrator: CREATE DATABASE sales, then a CREATE EXTERNAL TABLE pointing at that folder.
  3. Register the location so Lake Formation can vend access to it.
  4. Take the table out of IAM-only mode by revoking the IAMAllowedPrincipals grant on it.
  5. Grant the analyst DESCRIBE on the database and SELECT on every column except the email.
  6. Test as the analyst: SELECT * returns three columns; selecting customer_email by name fails with a permission error.

Steps 3 and 4 from the CLI, run as a data lake administrator:

# 3. Register the S3 path with the Lake Formation service-linked role
aws lakeformation register-resource \
  --resource-arn arn:aws:s3:::shop-lake/sales \
  --use-service-linked-role

# 4. Remove the "IAM decides" grant from this one table
aws lakeformation revoke-permissions \
  --principal DataLakePrincipalIdentifier=IAM_ALLOWED_PRINCIPALS \
  --permissions ALL \
  --resource '{"Table": {"DatabaseName": "sales", "Name": "orders"}}'

Step 5 is the pair of grants shown in the previous section. The analyst's IAM role needs only the Athena permissions to run queries and write results, Glue catalog read permissions, and lakeformation:GetDataAccess; it needs no permission on the shop-lake bucket at all. If the test in step 6 shows the email column, step 4 did not happen on the right table, which is the single most common slip in this tutorial.

Once this works for one table, the rest of the lake is the same pattern repeated, or, better, the LF-Tags approach below, which covers new tables without new grants.

IAMAllowedPrincipals and hybrid access: the setting behind most surprises

When Lake Formation was introduced, AWS needed existing data lakes to keep working. The answer is a special group called IAMAllowedPrincipals: by default, every new database and table in the catalog gets a grant of all permissions to this group, which means "anyone IAM allows can use this, as before". As long as that grant exists on a table, Lake Formation is effectively not restricting it.

That explains two opposite complaints:

  • "I granted only three columns, but the analyst sees everything." The table still has the IAMAllowedPrincipals grant, so IAM permissions win. Revoke that grant from the table (and change the default for new tables) to put it under Lake Formation's control.
  • "I removed IAMAllowedPrincipals and now everyone is locked out." Exactly what is meant to happen: now only explicit Lake Formation grants work. Grant DESCRIBE and SELECT to each role that should still read it, ideally before you revoke.

Hybrid access mode is the gentler path: it lets you put specific principals under Lake Formation permissions on a table while others keep using IAM, so you can move one team at a time instead of switching a whole table overnight. And one rule that catches people at sharing time: a resource that still has an IAMAllowedPrincipals grant cannot be shared with another account; the attempt fails with "Insufficient Lake Formation permissions: Illegal combination".

"Insufficient Lake Formation permission(s)": every variant and its fix

ErrorCauseFix
"Insufficient Lake Formation permission(s): Required Describe on DATABASE"The principal cannot even see the databaseGrant DESCRIBE on the database
"... Required Select on TABLE" (or an Athena "permission denied" on the table)No SELECT grant, or not on the columns queriedGrant SELECT on the table or those columns
"... Required Create Table on DATABASE"A crawler, Glue job or user creating tables lacks the grantGrant CREATE_TABLE on the database to that role
"Insufficient Lake Formation permissions on s3://..."Creating or altering a table that points at a registered location without data location accessGrant DATA_LOCATION_ACCESS on that location
"... Required Drop on TABLE"Deleting a table (often by a CloudFormation or Terraform role) without the grantGrant DROP, or make the deploy role a data lake admin
"Required create tag on catalog"Creating LF-Tags without being an administratorOnly data lake administrators can create LF-Tags
"Insufficient Lake Formation permissions: Illegal combination"Sharing a resource that still has an IAMAllowedPrincipals grantRevoke IAMAllowedPrincipals from it, then share
"Insufficient encryption key permissions for Glue API"The Data Catalog is encrypted and the granter lacks KMS permissionGive the granter IAM permission on the catalog's KMS key
Crawler: "Resource does not exist or requester is not authorized..."The crawler's role cannot create tables in the target databaseGrant CREATE_TABLE on the database
Crawler: AccessDeniedException on CreateTableNo data location permission on the target pathGrant DATA_LOCATION_ACCESS to the crawler's role
Athena or Redshift query with manifests failsLake Formation does not support queries that use manifestsQuery without manifests for governed tables

To see exactly what a principal has, run aws lakeformation list-permissions for it (shown above), or open Data lake permissions in the console and filter by principal. The error message names the permission and the object it was missing, so the fix is usually one grant. Ethan's analyst needed exactly that: one DESCRIBE on the sales database, on top of the column-limited SELECT he had already granted.

If the error is a plain IAM Access Denied rather than a Lake Formation one, the principal is missing an IAM permission such as athena:StartQueryExecution, glue:GetTable or lakeformation:GetDataAccess; our guide to reading Access Denied messages helps tell the two apart. And if the query runs but fails on the data itself, our Athena HIVE_BAD_DATA guide covers file problems.

Debugging any Lake Formation permission problem in five minutes

When a query fails and the message is less clear than the ones above, this sequence finds the cause almost every time:

  • Confirm who actually ran it. A Glue job, crawler or scheduled query runs as its own role, not as the person who started it. Grants to the person do nothing for the job.
  • List that principal's grants: aws lakeformation list-permissions --principal DataLakePrincipalIdentifier=ROLE_ARN. Look for DESCRIBE on the database and SELECT on the table or its columns.
  • List the table's grants: the same command with --resource and the table. If IAM_ALLOWED_PRINCIPALS appears, the table is still in IAM mode, and your Lake Formation grants are not what is deciding.
  • Check the location: aws lakeformation list-resources shows registered paths. Creating or changing tables on a registered path needs DATA_LOCATION_ACCESS.
  • Check the IAM side: the role still needs Athena, Glue catalog and lakeformation:GetDataAccess permissions. A plain IAM AccessDenied is not a Lake Formation problem at all.
  • For shared tables, check both halves: in the consumer account, the user needs DESCRIBE on the resource link and SELECT on the shared target table. Granting only the link is a classic miss.

Jake: "So when the analyst says 'it's broken', you just run those?"

Ethan: "Top to bottom. It's nearly always the second or third line."

"Insufficient Lake Formation permission(s) on default"

A puzzling variant names a database called default, which you may never have used. Spark-based services, including Glue ETL jobs and EMR, look up the default database when a session starts, whatever database the job actually works on. Once Lake Formation governs the catalog, that lookup needs a grant like any other, and the job fails before reading a single row of your real table.

The fix is a single grant: DESCRIBE on the default database for the job's role. If the database does not exist in your catalog at all, create an empty one named default and grant DESCRIBE on it. Nothing needs to be stored in it; the role just has to be allowed to see it.

Similar messages name "catalog" or a Glue table rather than a database. "On catalog" usually means the action needs a catalog-level permission, such as creating a database, which a data lake administrator can grant with CREATE_DATABASE on the catalog. "On glue table" points at a specific table the job tried to read or update without SELECT, ALTER or INSERT. In every case the message tells you the object; the fix is the matching grant to the role that ran the job, not the person who launched it.

Moving an existing data lake to Lake Formation without breaking it

The risk in adopting Lake Formation is not the new grants; it is the moment the old IAM-based access stops working. Doing it in this order keeps every report running:

  1. Inventory who reads what. CloudTrail's Athena and Glue events, plus your IAM policies, show which roles query which databases today. Write it down as a table of role, database and tables.
  2. Register the S3 locations and set yourself and the deploy role as data lake administrators. Nothing changes for users yet, because every table still carries the IAMAllowedPrincipals grant.
  3. Grant the equivalent Lake Formation permissions for every role in your inventory, preferably with LF-Tags. Grants added now are harmless, since IAM still decides.
  4. Switch one database at a time. Revoke IAMAllowedPrincipals from its tables, then run each team's usual queries. Hybrid access mode lets you opt in specific principals first if a whole database at once feels too big.
  5. Change the catalog defaults so new databases and tables are governed by Lake Formation from creation, instead of quietly starting in IAM-only mode.
  6. Tighten the S3 bucket policy last, removing direct read access for people and leaving it to Lake Formation's registered role.

Doing step 4 before step 3 is how most migrations go wrong: everyone is locked out of a table at once, and the morning's dashboards fail. Doing step 6 before step 4 breaks queries in a different way, because IAM-mode tables still read S3 with the caller's own permissions.

LF-Tags: permissions that scale

Granting table by table works for ten tables, not a thousand. LF-Tags (tag-based access control) let you label databases, tables and columns, for example domain=sales and sensitivity=pii, and grant on the labels: "the analyst role can SELECT anything tagged domain=sales and sensitivity=public". New tables tagged the same way are covered automatically, with no new grants.

  • Only data lake administrators create LF-Tags; they can let others assign them.
  • Tags on a database are inherited by its tables, and table tags by columns, unless overridden.
  • LF-Tag grants are also the simplest way to share with other accounts at scale.

From the CLI, the whole flow is three commands: create the tag, attach it, and grant on an expression of tags.

# 1. Create the tag and its allowed values (data lake administrators only)
aws lakeformation create-lf-tag --tag-key domain --tag-values sales stock hr

# 2. Attach it to a table
aws lakeformation add-lf-tags-to-resource \
  --resource '{"Table": {"DatabaseName": "sales", "Name": "orders"}}' \
  --lf-tags TagKey=domain,TagValues=sales

# 3. Grant on the tag, not the table: covers every table tagged domain=sales, now and later
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/analyst \
  --permissions SELECT DESCRIBE \
  --resource '{"LFTagPolicy": {"ResourceType": "TABLE",
               "Expression": [{"TagKey": "domain", "TagValues": ["sales"]}]}}'

Plan the tag keys before you create them. Two or three dimensions, such as domain, sensitivity and perhaps environment, cover most lakes; dozens of tags recreate the table-by-table mess you were escaping. Tag columns that hold personal data with sensitivity=pii at the column level, and the analyst's grant on sensitivity=public excludes them from every table automatically.

Row and column security: data filters

Column limits are part of a normal SELECT grant (include or exclude columns). For rows, you create a data filter on a table: a row filter expression such as region = 'EU', plus optional column limits, and grant SELECT on the filter instead of the table. A regional manager then sees only their region's rows in Athena or QuickSight, from the same table everyone else uses. Combining the two gives cell-level security: specific rows and specific columns.

For the shop, a filter store_id = 'north' gave the north store's manager their own figures without a separate copy of the data, and the analyst's grant excluded the two customer contact columns.

From the CLI, the filter is a small JSON file and two commands. Save this as north-filter.json:

{
  "TableCatalogId": "111122223333",
  "DatabaseName": "sales",
  "TableName": "orders",
  "Name": "north_store",
  "RowFilter": {"FilterExpression": "store_id = 'north'"},
  "ColumnWildcard": {"ExcludedColumnNames": ["customer_email", "customer_phone"]}
}
aws lakeformation create-data-cells-filter --table-data file://north-filter.json

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::111122223333:role/north-manager \
  --permissions SELECT \
  --resource '{"DataCellsFilter": {"TableCatalogId": "111122223333", "DatabaseName": "sales",
               "TableName": "orders", "Name": "north_store"}}'

Keep row filter expressions simple, such as comparisons, IN lists and AND/OR, and base them on columns that never change for a row, like a store or region code. The manager also needs DESCRIBE on the database, as always. One filter per audience, named after that audience, keeps the list readable when you later have a dozen of them.

Jake: "So it's one table, but everyone sees a different slice?"

Ethan: "One table, one copy of the data, and the slicing happens at query time."

Granting to people: IAM Identity Center users and groups

Grants to IAM roles work, but they describe machines, not people: when Priya from finance runs a query through a shared "analyst" role, the logs and permissions see the role. With IAM Identity Center connected to Lake Formation, you can grant directly to Identity Center users and groups, which usually come from your company's existing sign-in system, such as Microsoft Entra ID or Okta.

The pieces involved:

  • Identity Center syncs users and groups from your identity provider, so "Finance team" in your directory is a group Lake Formation can grant to.
  • Lake Formation grants permissions to those users and groups, exactly as it does to roles, including column limits, row filters and LF-Tags.
  • Athena workgroups with trusted identity propagation carry the signed-in person's identity into the query, so Lake Formation checks their own grants and CloudTrail records their name.
  • S3 Access Grants control access to the query results that Athena writes to S3 for those users.

Identity Center, the Athena workgroup, Lake Formation and S3 Access Grants must be in the same Region. The payoff is that a person who joins or leaves the finance group in your directory gains or loses the right data automatically, with no change in AWS. For a small business with three users, roles are simpler; for a company with a directory already in use, this is the cleaner long-term setup.

Glue jobs, EMR and Iceberg tables under Lake Formation

Athena is the easiest service to use with Lake Formation, because every query passes through it. The Spark-based services need a little more thought:

  • Glue ETL jobs and EMR respect table-level grants through the catalog, which is why their roles need DESCRIBE, SELECT, CREATE_TABLE and friends. Column and row limits inside Spark depend on the service version and its Lake Formation settings, so check that your Glue version or EMR release supports fine-grained Lake Formation access before relying on a column limit to hide data from a Spark job.
  • Jobs that write tables need data location access on the target path as well as CREATE_TABLE, the same pair a crawler needs.
  • Iceberg tables in the Glue Data Catalog are governed like any other table: grants, LF-Tags, column limits and row filters all apply when Athena reads them.
  • Amazon S3 Tables, the managed Iceberg table buckets, integrate with the Glue Data Catalog and use Lake Formation permissions when you query them through AWS analytics services.

A useful rule of thumb: if a query service reads through the Glue Data Catalog, ask whether it enforces Lake Formation's fine-grained rules or only table-level ones, and test with a restricted role before trusting it with sensitive columns.

Sharing data with other accounts

Lake Formation can share databases and tables with other AWS accounts or your whole organization, without copying data. Behind the scenes it uses AWS Resource Access Manager (RAM):

  1. The producer account's data lake administrator grants permissions to the other account (with grant option), using either named resources or LF-Tags.
  2. For accounts outside your organization using named resources, the receiving account's administrator accepts the RAM resource share invitation.
  3. The consumer account's administrator creates a resource link to the shared database or table in their own catalog.
  4. That administrator grants the shared resources (and the resource link) to their own users and roles.
  5. The consumer's users query through the resource link in Athena, like any local table.

Only data lake administrators in the receiving account see a newly shared resource at first, so "the other account cannot see it" often means a non-admin is looking.

Common sharing errors: "Association failed because the caller was not authorized" when accepting the invitation (the granting side's role needs glue:PutResourcePolicy), "Not authorized to grant permissions for the resource" (you are trying to re-share another account's resource to a third account), and "Organization ... not found" (sharing with Organizations is not turned on in RAM). Our AWS Organizations explainer covers the organization side, and our cross-account S3 guide covers the plain S3 alternative.

Lake Formation with Terraform

resource "aws_lakeformation_data_lake_settings" "this" {
  admins = [aws_iam_role.data_admin.arn]
}

resource "aws_lakeformation_resource" "sales_bucket" {
  arn = aws_s3_bucket.sales.arn
}

resource "aws_lakeformation_permissions" "analyst_db" {
  principal   = aws_iam_role.analyst.arn
  permissions = ["DESCRIBE"]
  database { name = "sales" }
}

resource "aws_lakeformation_permissions" "analyst_orders" {
  principal   = aws_iam_role.analyst.arn
  permissions = ["SELECT"]
  table_with_columns {
    database_name         = "sales"
    name                  = "orders"
    wildcard              = true
    excluded_column_names = ["customer_email", "customer_phone"]
  }
}

One caution: aws_lakeformation_data_lake_settings replaces the whole admin list and defaults, so include every administrator you want to keep, or you can lock yourself out of granting. The role that runs Terraform must itself be a data lake administrator to create grants.

Two more habits make Terraform and Lake Formation get along. Keep all grants for a table in one place in your code, since overlapping aws_lakeformation_permissions resources for the same principal and table can fight each other on every plan. And remember that grants made by hand in the console are invisible to Terraform: a quarterly list-permissions compared with your code catches grants someone added outside it. LF-Tags have their own resources (aws_lakeformation_lf_tag and aws_lakeformation_resource_lf_tags), and tag-based grants use an lf_tag_policy block in the permissions resource.

Lake Formation vs Glue vs S3

  • S3 stores the files. Its bucket policies control access to objects and prefixes, not tables or columns.
  • AWS Glue catalogs the data (the Data Catalog), discovers it (crawlers) and transforms it (ETL jobs). It is where the tables are defined.
  • Lake Formation governs access to what Glue catalogs: who sees which tables, columns and rows, across every query service, plus sharing across accounts.

They are layers, not alternatives: almost every Lake Formation setup uses all three. Lake Formation also includes blueprints and workflows to load data from databases into the lake, built on Glue jobs and crawlers.

The practical test for "do I need Lake Formation, or are S3 and Glue enough?" is the question you need to answer. "Can this role read this bucket?" is an S3 and IAM question. "Can this analyst see the email column in the orders table?" is a Lake Formation question, because S3 has no idea what a column is. The moment your access rules mention columns, rows or tables shared with another account, you have outgrown bucket policies.

The same test works in reverse when someone proposes Lake Formation for a problem it does not solve. Encrypting the data, controlling network access to S3, or limiting which buckets a role can write to are all jobs for S3, KMS and IAM. Lake Formation sits on top of them and assumes they are already right; it does not replace any of them.

Blueprints and workflows: loading data into the lake

Blueprints are templates that generate a Glue workflow (crawlers plus jobs) to bring data into the lake. They cover a few common sources: a one-time snapshot of a database table, incremental loads from a database using a column such as an ID or timestamp to find new rows, and log sources such as CloudTrail and load balancer logs. You choose the source, the target database and S3 path, and a schedule, and Lake Formation creates the workflow for you.

They are a reasonable starting point for a first lake, but most teams outgrow them: the generated jobs are generic, and changing them means editing Glue jobs you did not write. Many newer setups load data with their own Glue jobs, with database replication tools, or with zero-ETL integrations into the destination service, and use Lake Formation purely for permissions. That split is perfectly normal; Lake Formation's permissions work the same whichever tool wrote the tables.

AWS Lake Formation pricing

Setting up and enforcing permissions in Lake Formation has no extra charge. You pay for the services underneath and around it:

  • Amazon S3 storage and requests for the data.
  • AWS Glue: Data Catalog storage and requests beyond the free tier, crawler runs and ETL jobs.
  • Query services: Athena per data scanned, Redshift Spectrum, EMR clusters. Partitioning keeps Athena costs down; our Athena costs guide explains how.

Some newer or advanced capabilities may carry their own charges, so check the Lake Formation pricing page before relying on one at scale.

In practice, a governed lake costs the same as an ungoverned one of the same size. For a small business like Jake's, the monthly bill is S3 storage for a few gigabytes, a little Glue catalog use inside the free tier, and Athena scans measured in cents. Lake Formation itself adds nothing, and the hours it saves on access reviews are the real return.

Administrators, read-only administrators and auditing

Lake Formation has three kinds of special principal, and choosing them carefully is most of its security:

  • Data lake administrators can grant anything, see every resource and create LF-Tags. Keep this to a small team, plus the deployment role if you manage grants with infrastructure as code.
  • Read-only administrators can see every resource and every permission, but change nothing. This is the right role for auditors and security reviewers, who need the full picture without the power to alter it.
  • Database creators can create databases; whoever creates a database gets full permissions on it, so this is a quiet way that new owners appear.

For an audit, three sources answer most questions. aws lakeformation list-permissions, filtered by principal or by resource, shows who can do what right now. The console's Data lake permissions page shows the same with filters. And CloudTrail records both the grants (GrantPermissions, RevokePermissions) and every data access (GetDataAccess), so "who read the orders table last month" is a CloudTrail query, not a guess.

Jake: "So the accountant can check who sees what, but can't change it?"

Ethan: "Read-only administrator. Sees every grant, touches none."

Benefits, and when you do not need Lake Formation

The benefits are real but specific: one place for data permissions across every query service; column, row and cell-level security without copying data; sharing tables across accounts without moving files; LF-Tags that cover new tables automatically; and an audit trail of data access. If those match your problems, Lake Formation is the standard answer on AWS.

It is unnecessary overhead when one person or one small team uses all the data, when every user may see every column, or when the data lives in a single bucket with a single reader. In those cases, IAM and a bucket policy are simpler, and switching Lake Formation on early mostly adds a second place to debug permissions. A sensible rule: start without it, and turn it on the day you first need to hide a column, restrict rows, or share a table with another account.

Frequently asked questions

What is AWS Lake Formation?

A service that manages permissions for a data lake: who can see which databases, tables, columns and rows in the Glue Data Catalog, enforced in Athena, Redshift Spectrum, EMR, Glue and QuickSight.

What is AWS Lake Formation used for?

Fine-grained access to data in S3, column and row security, sharing data across accounts without copies, and auditing who can access what.

How much does AWS Lake Formation cost?

Managing permissions has no extra charge; you pay for S3, the Glue Data Catalog and the query services you use.

What does "Insufficient Lake Formation permission(s)" mean?

The principal lacks a Lake Formation grant (such as DESCRIBE, SELECT, CREATE_TABLE or DROP) on the named object, or data location access on the S3 path.

How do I fix "Required Describe on database"?

As a data lake administrator, grant DESCRIBE on that database to the principal in the error.

What is IAMAllowedPrincipals in Lake Formation?

A group whose grant on a resource lets IAM permissions alone control it. Remove it to put the resource under Lake Formation permissions.

Why can users still see columns I excluded?

The table still has an IAMAllowedPrincipals grant, so IAM permissions apply. Revoke it from the table.

What is a data lake administrator?

A principal that can grant any Lake Formation permission and see all catalog resources; read-only administrators can view but not change.

What permissions does a Glue crawler need with Lake Formation?

CREATE_TABLE on the target database and DATA_LOCATION_ACCESS on the S3 location, plus its normal IAM permissions.

What are LF-Tags?

Labels on databases, tables and columns that you grant permissions on, so new resources with the same tags are covered automatically.

How do I restrict rows in Lake Formation?

Create a data filter with a row filter expression and grant SELECT on the filter instead of the table.

How do I share Lake Formation tables with another account?

Grant the account permissions with grant option; it accepts the RAM invitation if needed, then grants to its own principals through a resource link.

What is the difference between Lake Formation and Glue?

Glue catalogs, crawls and transforms data; Lake Formation controls access to what Glue catalogs.

What is the difference between Lake Formation and S3?

S3 stores files and controls access to objects; Lake Formation controls access at the level of tables, columns and rows.

What does "Illegal combination" mean in Lake Formation?

You tried to share a resource that still has an IAMAllowedPrincipals grant; revoke it first.

Can I manage Lake Formation with Terraform?

Yes, with aws_lakeformation_data_lake_settings, aws_lakeformation_resource and aws_lakeformation_permissions; the Terraform role must be a data lake admin.

What does "Insufficient Lake Formation permission(s) on default" mean?

A Spark-based job (Glue or EMR) looked up the default database. Grant DESCRIBE on default to the job's role, creating the database if it does not exist.

What is a read-only administrator in Lake Formation?

A principal that can view every resource and permission but change nothing, which suits auditors.

Do users need S3 permissions when Lake Formation is used?

Not for registered locations. Lake Formation vends temporary credentials; the user's role needs lakeformation:GetDataAccess plus the query service's permissions.

What are Lake Formation blueprints?

Templates that create Glue workflows to load data from databases or logs into the lake, on a schedule.

How do I revoke IAMAllowedPrincipals with the CLI?

aws lakeformation revoke-permissions with the principal IAM_ALLOWED_PRINCIPALS, permissions ALL, and the table as the resource.

How do I grant permissions with LF-Tags?

Create the tag, attach it to resources, then grant on an LFTagPolicy resource with an expression such as domain=sales.

Does Lake Formation stop people downloading files from S3?

Only through the integrated services. Anyone with direct S3 permissions on the bucket can still read the files, so keep bucket access narrow.

Do I need Lake Formation for a small data lake?

Not if everyone may see all the data. Turn it on when you need column or row limits, or cross-account sharing.

How do I move an existing data lake to Lake Formation?

Inventory access, register locations, grant equivalent Lake Formation permissions, then revoke IAMAllowedPrincipals one database at a time, and tighten S3 last.

Can I grant Lake Formation permissions to IAM Identity Center groups?

Yes. With Identity Center connected, grant to its users and groups, and use Athena workgroups with trusted identity propagation.

Why can the other account see the resource link but not query it?

The user needs DESCRIBE on the resource link and SELECT on the shared target table. Granting only the link is not enough.

How do I create a row filter in Lake Formation with the CLI?

Write the filter as JSON (table, name, RowFilter expression, optional column limits), run create-data-cells-filter, then grant SELECT on the DataCellsFilter resource.

Who can see a newly shared Lake Formation table in the receiving account?

At first only that account's data lake administrators. They must create a resource link and grant access to other users.

Does Lake Formation log who accessed data?

Yes. CloudTrail records GetDataAccess events for data access and GrantPermissions or RevokePermissions for permission changes.

How do I see which S3 locations are registered with Lake Formation?

Run aws lakeformation list-resources, or open Data lake locations in the console.

Does Lake Formation work with Iceberg tables?

Yes. Iceberg tables in the Glue Data Catalog take grants, LF-Tags, column limits and row filters like other tables.

Does Lake Formation work with Athena?

Yes. Athena enforces Lake Formation grants, including column limits and row filters.

For readers preparing for an AWS certification

We have been covering AWS certifications alongside our fixes, so here is a short extra for readers studying for one. If you only came for the fix, you already have it; feel free to skip this part.

Exams that test this topic: Solutions Architect – Associate (SAA-C03) • Security – Specialty (SCS-C03) • Data Engineer – Associate.

Here are the exam-style questions this topic tends to raise, with the reasoning you need to answer them:

1. A company needs column-level and row-level access control for data in S3 queried with Athena. Which service?

AWS Lake Formation.

2. An Athena query fails with "Insufficient Lake Formation permission(s): Required Describe on DATABASE". What fixes it?

Grant the principal DESCRIBE on that database in Lake Formation.

3. A Glue crawler cannot create tables under Lake Formation. Which permissions does its role need?

CREATE_TABLE on the database and DATA_LOCATION_ACCESS on the S3 location.

4. Grants in Lake Formation seem to have no effect; users still see every column. Why?

The table still has a grant to IAMAllowedPrincipals, so IAM permissions control it.

5. How can permissions be applied automatically to new tables across a large data lake?

With LF-Tags (tag-based access control).

6. How does Lake Formation share tables with another AWS account without copying data?

Cross-account grants through AWS Resource Access Manager, used through resource links.

7. Which service holds the table definitions that Lake Formation governs?

The AWS Glue Data Catalog.

Exam codes and dates change often. Our AWS certification changes page keeps track of what is current.

πŸ“š ALSO READ

Where to go next, from Athena errors to cross-account access:

📌 Bookmark this; the permissions table and the error list are the parts you will come back for.

A month in, the shop's data lake has four roles, five LF-Tags and no bucket policies anyone is afraid to touch. The analyst sees every column except the two contact fields, each store manager sees their own store's rows, and a quarterly review is a single list-permissions command. Jake's favorite part: when someone asks "who can see customer emails?", the answer takes one screen instead of an afternoon.

📌 If you keep one line from this page

Lake Formation is the free permissions layer over S3 and the Glue Data Catalog; "Insufficient Lake Formation permission(s)" names the exact grant that is missing, and IAMAllowedPrincipals is why grants sometimes seem to do nothing.

DESCRIBE before SELECT · DATA_LOCATION_ACCESS for crawlers · LF-Tags to scale · data filters for rows.

Revision note. Written October 19, 2026. Lake Formation does nothing that is not granted; most errors are one missing grant, named in the message itself.

Related