AWS Glue Tutorial for Beginners: What It Is, Pricing (Glue 6.0 Is 30% Cheaper), Crawlers, Data Catalog, Jobs and Glue vs EMR

AWS Glue is the serverless data integration service in AWS. It finds your data with crawlers, describes it in a central catalog of tables (the Glue Data Catalog, which Athena, Redshift and EMR all read), and transforms it with Apache Spark or Python jobs that you write, generate or build visually, with no servers or clusters for you to manage. You pay only while your crawlers and jobs run, by the DPU-hour (one DPU is 4 vCPUs and 16 GB of memory), billed per second with a 1-minute minimum. Since Glue 6.0 arrived in August 2026, your Spark jobs on 6.0 cost $0.308 per DPU-hour in US East, 30% less than the $0.44 you still pay on Glue 5.1 and older; Flex jobs for work that can wait cost $0.203 on 6.0. Crawlers are $0.44 per DPU-hour, and your Data Catalog is free for the first million objects and million requests a month. A nightly 15-minute job on 10 DPUs costs you about $23 a month on Glue 6.0. This tutorial for beginners covers what Glue is and how it works, every price with worked examples, a step-by-step pipeline from CSV to Parquet, a job script you can copy, Glue vs EMR, Athena, Lambda, Databricks and Azure Data Factory, interview questions, and fixes for the errors you are most likely to hit.

Ethan's client, the online furniture store, dropped a CSV of the day's orders into an S3 bucket every night, and the analysts wanted dashboards. Their queries were slow and expensive because every query read every CSV, headers, duplicates and all. "We need it cleaned and turned into Parquet, partitioned by date," Ethan told the owner. Jake, who was at the store fixing the warehouse tablets, overheard and asked the obvious question: "What does glue have to do with any of that? Are you sticking spreadsheets together?"

New to data on AWS? The cards below cover Athena, S3, Lake Formation, IAM and Athena's costs, the pieces Glue connects for you.

What is AWS Glue?

The name is the job: AWS Glue is the glue between where your data is created and where you analyze it. It is an ETL service (extract, transform, load) and a metadata catalog in one, and it is serverless: you never pick servers, install Spark or keep a cluster running. You describe the work, Glue starts the capacity, runs it, and your bill stops when it finishes.

Glue does four jobs for you:

• Discover: crawlers scan your data in Amazon S3, databases and other stores, work out the format (CSV, JSON, Parquet, Avro, ORC and more), the columns and their types, and the partitions, so you do not type table definitions by hand.
• Catalog: the AWS Glue Data Catalog stores those findings as databases and tables. It is the shared catalog for Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, Amazon SageMaker and AWS Lake Formation, so a table you define once can be queried from all of them.
• Transform: your ETL jobs run Apache Spark (in Python or Scala), plain Python (Python shell jobs) or Ray, to clean, join, deduplicate, reshape and convert your data, then write it wherever you need it.
• Orchestrate and check: triggers and workflows run your crawlers and jobs on schedules or events, and Glue Data Quality checks your data against rules you define.

What you would use AWS Glue for: building a data lake on S3; turning raw CSV or JSON into compressed, partitioned Parquet or Apache Iceberg tables so your queries are fast and cheap; loading a data warehouse such as Amazon Redshift; copying data between databases; preparing training data for machine learning; streaming ETL from Kinesis or Kafka; and keeping one catalog of every dataset your company owns.

Is AWS Glue serverless? Yes. You choose a worker type and how many workers you want (or let Auto Scaling decide), and Glue provisions them for each run in seconds. There is nothing for you to patch, size in advance or shut down.

Jake: "So it's not glue, it's a kitchen. Raw stuff comes in, it gets cleaned and chopped, and plated food goes out."

Ethan: "And the Data Catalog is the menu that tells everyone what's in the kitchen. That part's free for most people."

How AWS Glue works (the architecture)

Here is what happens, step by step, when your nightly pipeline runs. Knowing the sequence makes the pricing and the errors much easier to follow:

1. A trigger fires. Your schedule (for example, 2 a.m. every day), an event, or the end of another job tells Glue to start.
2. A crawler updates the catalog (optional). It scans the new folders in your bucket and adds partitions or columns to the table in the Data Catalog.
3. Glue provisions workers. For your job's Glue version and worker type, Glue starts the Spark driver and executors in an AWS-managed environment, or inside your VPC if the job uses a connection. Billing starts here.
4. Your script reads. The job asks the Data Catalog where the table's files are, then reads them from S3 or your database. With a job bookmark, it reads only the files it has not processed before.
5. Your transforms run across the workers in parallel: casting types, dropping duplicates, joining, filtering, aggregating.
6. Your job writes the result, for example partitioned Parquet in a curated S3 folder or rows in Redshift, and can add the new partitions to a catalog table at the same time.
7. Logs and metrics go to Amazon CloudWatch: driver and executor logs, the Spark UI, and DPU hours used.
8. Glue releases the workers and your billing stops. The bookmark saves its position for tomorrow's run.

The components you will meet in the console:

• Data Catalog: a managed, Hive-compatible metastore. A database is a folder of tables; a table records a location (for example, s3://bucket/orders/), a format, columns and partitions. It is regional and per account, and you control access with IAM and, for fine-grained table and column permissions, AWS Lake Formation.
• Crawlers and classifiers: you point a crawler at a data store and an IAM role, and it creates or updates tables; classifiers are the format rules it uses, and you can add custom ones for grok, XML, JSON or CSV quirks. You can run crawlers on a schedule, on demand, or from S3 event notifications so they only re-scan the folders that changed.
• Connections: saved settings for reaching your JDBC databases (MySQL, PostgreSQL, SQL Server, Oracle), Amazon Redshift, MongoDB, Kafka, and SaaS sources such as Salesforce, SAP, ServiceNow and Snowflake. A connection also puts your job inside your VPC when the source is private.
• ETL jobs: Spark jobs (batch), Spark Streaming jobs, Python shell jobs for small tasks, and Ray jobs for distributed Python. Each job has a script in S3, an IAM role, a Glue version, a worker type and a number of workers.
• Glue Studio: where you build jobs visually (drag sources, transforms and targets), edit the generated scripts, monitor runs, or work in notebooks with interactive sessions. Generative AI in the console can write a job from your plain-English description and upgrade your old Spark jobs to new Glue versions.
• Job bookmarks: remember which files your job already processed, so tomorrow's run only reads new data.
• Triggers and workflows: schedules (cron), "run after job X succeeds," or on demand; workflows chain your crawlers and jobs into one pipeline with a graph. For larger pipelines you may prefer AWS Step Functions or Amazon MWAA (managed Airflow).
• Glue Data Quality: rules you write in DQDL, such as "order_id is unique" or "amount between 0 and 100000," evaluated in your jobs or on catalog tables, with anomaly detection.
• Glue DataBrew: a visual, no-code data preparation tool with more than 250 ready-made transformations, aimed at analysts rather than engineers.
• Schema Registry: stores and versions your Avro, JSON and Protobuf schemas for streaming data, at no charge.
• Zero-ETL integrations: managed replication from applications such as Salesforce, SAP or ServiceNow into Redshift or an S3 lakehouse, with no job code for you to write.

AWS Glue pricing: every rate

Glue charges you for compute in DPU-hours. One DPU (data processing unit) is 4 vCPUs and 16 GB of memory. A job with 10 G.1X workers uses 10 DPUs, so one hour of it is 10 DPU-hours. Your billing is per second with a 1-minute minimum on Glue 2.0 and later. The formula for any run is simple: DPUs x hours x price per DPU-hour. US East (N. Virginia) prices in October 2026:

What runs	Glue 6.0 and later	Glue 2.0 to 5.1
Spark and Spark Streaming ETL jobs (G workers)	$0.308 per DPU-hour	$0.44 per DPU-hour
Flex ETL jobs (can start later, for non-urgent work)	$0.203 per DPU-hour	$0.29 per DPU-hour
Memory-optimized jobs (R workers)	$0.364 per M-DPU-hour	$0.52 per M-DPU-hour
Interactive sessions and notebooks	$0.308 per DPU-hour	$0.44 per DPU-hour
Python shell jobs (0.0625 or 1 DPU)	$0.44 per DPU-hour, 1-minute minimum
Ray jobs and Ray sessions	$0.44 per M-DPU-hour
Crawlers	$0.44 per DPU-hour, billed per second
Catalog statistics, Iceberg compaction, materialized view refresh	$0.44 per DPU-hour, 1-minute minimum
Data Catalog storage	First 1 million objects free each month, then $1 per 100,000 objects
Data Catalog requests	First 1 million requests free each month, then $1 per million
DataBrew	$1 per 30-minute interactive session; jobs $0.48 per node-hour
Zero-ETL from applications	$1.50 per GB ingested, plus $0.44 per DPU-hour to write to an S3 lakehouse
Schema Registry	Free

The 6.0 discount applies only to jobs and sessions that run on Glue 6.0 or later. If you leave a job on Glue 5.1, you keep paying $0.44, so upgrading is the single biggest cost lever you have in Glue right now.

Worker types decide how many DPUs each of your workers uses:

Worker	DPUs	vCPU / memory	Use it when
G.1X	1	4 / 16 GB	You run most transforms, joins and conversions
G.2X	2	8 / 32 GB	Your joins are heavier or your rows wider
G.4X / G.8X	4 / 8	16 / 64 GB and 32 / 128 GB	You have demanding aggregations and large shuffles
G.12X / G.16X	12 / 16	48 / 192 GB and 64 / 256 GB	You run the largest compute-heavy jobs
R.1X to R.8X	1 to 8 M-DPU	Double memory: 32 to 256 GB	Your job keeps running out of memory

Is AWS Glue expensive? For scheduled batch work, usually not: you pay only for the minutes your jobs run, and small pipelines cost a few dollars to a few tens of dollars a month. It gets expensive when you size jobs far bigger than the data needs, leave notebooks running, crawl a whole bucket every hour, or run a busy cluster's worth of work all day, where a tuned EMR cluster can cost you less. The AWS Pricing Calculator at calculator.aws has a Glue estimator: enter your DPUs, run time and runs per month for each job and crawler.

Worked examples: what Glue really costs

Example 1: the furniture store's nightly job. 10 G.1X workers (10 DPUs) for 15 minutes, every night. That is 10 x 0.25 = 2.5 DPU-hours a night, or 75 DPU-hours a month.

• Glue 5.1: 75 x $0.44 = $33.00 a month
• Glue 6.0: 75 x $0.308 = $23.10 a month
• Glue 6.0 Flex (fine for a job that only has to finish by morning): 75 x $0.203 = $15.23 a month

Example 2: a crawler after each load. A crawler that uses 2 DPUs for 6 minutes each night: 2 x 0.1 = 0.2 DPU-hours x $0.44 = $0.088 a run, about $2.64 a month. If you crawl only the changed folders instead of your whole bucket, this stays small as your data grows.

Example 3: a tiny Python shell job every hour. 0.0625 DPU for 2 minutes per run: 0.0625 x (2/60) = 0.00208 DPU-hours x $0.44 = $0.0009 a run. Your 720 runs a month cost about $0.66. Python shell is the cheapest way for you to run small API pulls and file moves on Glue.

Example 4: a big daily job. 50 G.2X workers (100 DPUs) for 2 hours a day: 200 DPU-hours a day, 6,000 a month.

• Glue 5.1: 6,000 x $0.44 = $2,640 a month
• Glue 6.0: 6,000 x $0.308 = $1,848 a month
• Upgrading saves you $792 a month, $9,504 a year, for the cost of testing your job on Spark 4.1.

Example 5: a development notebook left open. An interactive session with 5 DPUs. If you use it for 2 hours, it costs 5 x 2 x $0.308 = $3.08 on Glue 6.0. But the default idle timeout for Spark sessions is 2,880 minutes (48 hours), and your session bills while it waits: forget it on a Friday evening and it runs until Sunday, 5 x 48 x $0.308 = $73.92. Put %idle_timeout 30 in the first cell of every notebook you open.

Example 6: a large Data Catalog. 3 million objects (tables, partitions and table versions all count) and 5 million requests a month: storage (3,000,000 - 1,000,000) / 100,000 x $1 = $20, and requests (5 - 1) million x $1 = $4, so $24 a month. If your data lake is small or medium, you will most likely stay entirely inside the free tier.

Ethan: "Twenty-three dollars a month to turn their mess into clean tables, and their Athena bill drops because every query reads a tenth of the data."

Jake: "So the kitchen pays for itself in dishes you don't have to wash."

To keep your Glue costs visible, tag jobs by team or pipeline, watch the DPU hours and execution time on each run page, and turn on Auto Scaling (Glue 3.0 and later) so a job uses fewer workers when it can. Your Athena scan costs drop too, thanks to the Parquet and partitions Glue creates, as our Athena costs guide shows.

Is AWS Glue free? The free tier

Part of it is. Your Data Catalog is free for the first 1 million objects stored and the first 1 million requests every month, with no time limit, which covers most small data lakes. Schema Registry is free. If you are new to DataBrew, you get free interactive sessions to try it. Jobs, crawlers and interactive sessions are not free: they bill you from the first second (with the 1-minute minimum), so a test job on 2 workers for 1 minute costs about 1 cent on Glue 6.0 (2 x 1/60 x $0.308 = $0.01). If your account is on the newer credit-based Free Tier, you can spend credits on Glue usage, as our explainer on the AWS Free Tier types covers.

AWS Glue tutorial: from CSV in S3 to Parquet you can query

This builds the furniture store's pipeline end to end, and you can follow it in about 30 minutes. You need an S3 bucket with some CSV files in a folder such as s3://my-shop-lake/raw/orders/.

1. Create an IAM role for Glue. In IAM, create a role for the Glue service and attach the AWSGlueServiceRole managed policy, plus a policy allowing s3:GetObject, s3:PutObject and s3:ListBucket on your bucket. Your own user also needs iam:PassRole on this role; our iam:PassRole guide explains that error if you meet it.
2. Create a database in your Data Catalog: Glue console, Data Catalog, Databases, Add database, and name it shop.
3. Create a crawler: Crawlers, Create crawler, data source s3://my-shop-lake/raw/orders/, the role from step 1, target database shop, schedule on demand. Run it. In a minute or two you will see a new table such as orders.
4. Check your table in Athena: SELECT * FROM shop.orders LIMIT 10;. If the header row shows up as data, set the table property skip.header.line.count to 1, or add a CSV classifier that marks the first row as a header. Our Athena step-by-step guide covers querying in detail.
5. Build your job in Glue Studio, Visual ETL: source Data Catalog table shop.orders; transforms Change Schema (fix types, drop junk columns), Drop Duplicates on order_id and Filter; target Amazon S3, format Parquet with Snappy compression, path s3://my-shop-lake/curated/orders/, partition key order_date, and "Create a table in the Data Catalog and on subsequent runs, update the schema and add new partitions" so your curated table stays current.
6. Job details: pick your IAM role, Glue version 6.0 for the lower price, worker type G.1X, 2 to 10 workers with Auto Scaling, job bookmark enabled, and a timeout (for example 60 minutes) so a stuck run cannot bill you for days.
7. Run it and watch the run page. When it succeeds, query shop.curated_orders in Athena and compare the "Data scanned" figure with your raw table.
8. Schedule it: add a trigger (for example cron(0 2 * * ? *) for 2 a.m. UTC), or a workflow that runs your crawler first and your job after it succeeds.
9. Add a quality check: an Evaluate Data Quality node with rules such as IsUnique "order_id" and ColumnValues "amount" >= 0, set to fail your job or route bad rows aside.

You can do the same from the AWS CLI:

# Run the crawler, then the job
aws glue start-crawler --name raw-orders-crawler
aws glue start-job-run --job-name orders-to-parquet \
  --worker-type G.1X --number-of-workers 10

# Check the run (use the JobRunId the previous command printed)
aws glue get-job-run --job-name orders-to-parquet --run-id jr_0123456789abcdef \
  --query "JobRun.[JobRunState,ExecutionTime,DPUSeconds,ErrorMessage]"

Jake: "That's it? A role, a crawler and some boxes on a screen?"

Ethan: "For this job, yes. The hard part was getting the analysts to agree on what 'cleaned' means."

An AWS Glue job script you can copy (PySpark)

Every visual job you build is a script underneath, and you can write or edit one directly when the visual nodes are not enough. A Glue script is ordinary PySpark plus a few Glue helpers: getResolvedOptions reads your job parameters, GlueContext wraps Spark and adds catalog-aware readers, and Job handles bookmarks. Glue's own data type, the DynamicFrame, tolerates messy, changing schemas; you can convert it to a normal Spark DataFrame with toDF() whenever you want the full Spark API. This version reads your catalog table, cleans it with DataFrame code and writes partitioned Parquet:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext
from pyspark.sql import functions as F

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue = GlueContext(SparkContext.getOrCreate())
job = Job(glue)
job.init(args["JOB_NAME"], args)

# transformation_ctx lets the job bookmark remember which files were read
raw = glue.create_dynamic_frame.from_catalog(
    database="shop", table_name="orders", transformation_ctx="raw_orders")

df = (raw.toDF()
      .withColumn("amount", F.col("amount").try_cast("decimal(10,2)"))
      .withColumn("order_date", F.to_date("order_ts"))
      .filter(F.col("amount").isNotNull())
      .dropDuplicates(["order_id"]))

(df.write.mode("append")
   .partitionBy("order_date")
   .parquet("s3://my-shop-lake/curated/orders/"))

job.commit()   # saves the bookmark; without it every run re-reads everything

Two details catch people. First, job bookmarks need both the transformation_ctx on your source and job.commit() at the end. Second, Glue 6.0 runs Spark 4.1, where ANSI SQL mode is on by default: a bad value in a plain cast now fails your job instead of quietly becoming null, which is why the script uses try_cast. Test your old scripts on 6.0 before you switch production jobs.

To pass your own settings, such as a date to reprocess, add them as job parameters (for example --run_date 2026-10-20) and read them with getResolvedOptions(sys.argv, ["JOB_NAME", "run_date"]). To write an Apache Iceberg table instead of plain Parquet, set the job's --datalake-formats parameter to iceberg and use df.writeTo("glue_catalog.shop.orders_iceberg").append(), which also gives you row-level updates, time travel and automatic compaction later.

AWS Glue examples: five real pipelines

If you are wondering where Glue fits in your own work, these are the patterns you will see most often, each with the pieces it uses:

• CSV or JSON to a queryable data lake (this page's tutorial): a crawler on the raw folder, a Spark job that cleans and writes partitioned Parquet or Iceberg, Athena or QuickSight on top. Typical cost: a few dollars to a few tens of dollars a month.
• Database to data lake, every night: a JDBC connection to your MySQL or PostgreSQL database, a job with bookmarks on an updated_at column so each run copies only changed rows, and Parquet output for analytics. Your production database is read once a night instead of by every analyst's query.
• DynamoDB to analytics: a DynamoDB export to S3 (no read capacity used), then a Glue job that flattens the nested JSON into tables. Our guide to DynamoDB export to S3 covers the export side.
• Loading a warehouse: a Glue job that joins and aggregates raw data, then writes to Amazon Redshift through its connector, with Data Quality rules that stop bad batches before they reach your reports.
• Streaming ETL: a Spark Streaming job reading from Kinesis Data Streams or Kafka, parsing events and writing micro-batches to S3 every minute; Glue 6.0's real-time mode cuts that to under a second for stateless work.

For SaaS sources such as Salesforce or ServiceNow, a zero-ETL integration may replace a job entirely: you pick the source objects and the target, and Glue keeps them replicated at $1.50 per GB ingested.

Glue versions: what Glue 6.0 changes

• Glue 6.0 (generally available August 21, 2026): Apache Spark 4.1.1, Python 3.13, Scala 2.13, Java 17, and 30% lower prices for you. It adds full Apache Iceberg v3 support (the VARIANT type, deletion vectors, row lineage, geometry and geography types), Spark Declarative Pipelines, real-time mode for sub-second streaming, Spark Connect for interactive sessions and Arrow-native Python UDFs. It uses the AWS SDK for Java v2 only and the S3A connector (EMRFS is removed).
• Glue 5.1: Spark 3.5.6 and Python 3.11, and the default for new jobs if you do not set a version.
• Glue 5.0: Spark 3.5.4, SageMaker Lakehouse and S3 Tables support, and requirements.txt for your extra Python libraries.
• Glue 4.0 and 3.0: Spark 3.3 and 3.1; still running, but older.
• Glue 0.9, 1.0 and 2.0 reached end of life on April 1, 2026; if you still have jobs on them, move them to a supported version.

Because the 6.0 price cut is tied to the version, set Glue version 6.0 explicitly on your new jobs instead of relying on the default, and use the console's generative AI Spark upgrade feature to analyze and upgrade your older jobs.

Glue vs EMR, Athena, Lambda, Databricks, Azure Data Factory and Airflow

• Glue vs Amazon EMR: both run Spark. Glue is serverless and opinionated: no cluster settings, per-second billing per job, built-in catalog, bookmarks and Studio. EMR (on EC2, EKS or EMR Serverless) gives you full control of Spark versions, instance types, Spot pricing and other engines (Hive, Presto, Flink, HBase), and is usually cheaper for large, steady, all-day processing run by a team that tunes clusters. Pick Glue for your scheduled pipelines with little ops work; EMR for heavy, continuous, finely tuned workloads.
• Glue vs Amazon Athena: Athena runs SQL queries on your data in S3 at $5 per TB scanned, using the Glue Data Catalog for table definitions. Glue transforms your data; Athena queries it. They work together: Glue makes the Parquet, Athena reads it cheaply. Athena's CTAS and INSERT INTO can do light transforms in SQL too.
• Glue vs AWS Lambda: Lambda is great for small, event-driven work under 15 minutes and 10 GB of memory, such as reacting to one uploaded file. Glue is for datasets too big for one function and for jobs that run longer.
• Glue vs Databricks: Databricks is a full data and AI platform with collaborative notebooks, Delta Lake, Unity Catalog and its own pricing (DBUs plus the cloud VMs). Glue is AWS-native, serverless and pay-per-job, with no platform fee. If your team standardizes on Databricks, you can still use the Glue Data Catalog as a metastore.
• Glue vs Azure Data Factory: Data Factory is Microsoft Azure's equivalent: a visual pipeline and data movement service, with Spark data flows, for data in Azure. Glue is the AWS counterpart for data in AWS. If your data lives in Azure, Data Factory (or Microsoft Fabric) is the natural choice; if it lives in S3 and AWS databases, Glue is.
• Glue vs Apache Airflow: Airflow is an orchestrator, not a data processing engine. It schedules and monitors tasks, which can include starting your Glue jobs. Many teams run Airflow (self-managed or Amazon MWAA) to coordinate pipelines and Glue to do the processing.
• Glue vs dbt: dbt transforms data with SQL models inside a warehouse or query engine (Redshift, Athena, Snowflake). Glue transforms with Spark or Python before or outside the warehouse. You can use both: Glue to land and clean raw data, dbt to model it for analysts.
• Glue vs Step Functions: not competitors. Step Functions orchestrates steps (start a Glue job, wait, branch, notify you); Glue does the data processing inside those steps.

AWS Glue alternatives, then, depend on what you want to replace: EMR or EMR Serverless for more Spark control, Athena CTAS or Redshift SQL for SQL-only transforms, Lambda for small event-driven work, Databricks or Snowflake for a full platform, and Fivetran or Airbyte if you only need to copy data from SaaS apps.

Ethan: "If they had a team of data engineers running Spark all day, I'd look at EMR. For one nightly job, Glue means nobody has to babysit a cluster."

AWS Glue interview questions (with short answers)

If you are preparing for a data engineering interview or an AWS data exam, these are the questions you are most likely to get, with answers you can expand on:

• What is a DPU, and how is a Glue job billed? 4 vCPUs and 16 GB of memory; you pay DPUs x hours x the rate, per second with a 1-minute minimum.
• DynamicFrame vs DataFrame? A DynamicFrame is Glue's schema-flexible structure that handles inconsistent types (choice types); a DataFrame is Spark's structure with a fixed schema and the full Spark API. You convert with toDF() and DynamicFrame.fromDF().
• How do job bookmarks work? They store state about processed data per source (transformation_ctx) and commit it with job.commit(), so the next run reads only new data.
• When would you use a Python shell job instead of Spark? For small tasks that fit on one machine, such as API calls or a few SQL statements, because they cost a fraction of a Spark job.
• How do you make a job read only new partitions? Bookmarks, a pushdown predicate on the partition columns, or S3 event-based processing.
• How do you handle a job that runs out of memory? Avoid collect() on the driver, fix data skew, group small files, use larger G workers or R workers, and check the Spark UI.
• What does a crawler do, and when would you skip it? It infers schemas and partitions into the catalog; you skip it when your job updates the catalog itself or when you use Athena partition projection.
• How does Glue access a private database? Through a connection in your VPC subnet, with a self-referencing security group rule and routes to S3.
• What is Glue Flex? A cheaper execution class for non-urgent jobs that may start later.
• Glue vs EMR? Serverless and managed vs full cluster control; Glue for scheduled pipelines, EMR for heavy, tuned, continuous processing.

Common AWS Glue errors and fixes

• ConcurrentRunsExceededException: your job's maximum concurrency defaults to 1, so starting it while a run is still going fails. Wait for the run to finish, raise the job's max concurrency, or check that a trigger is not firing twice.
• OutOfMemoryError or "Container killed by YARN for exceeding memory limits": the driver or an executor ran out of memory, often from collect(), data skew or too many small files. Our Glue OutOfMemoryError guide walks you through every cause, and R workers double the memory per DPU.
• AccessDenied on S3 or KMS: your job's IAM role, not your user, needs the S3 and KMS permissions. Check the role on the job's details tab.
• "iam:PassRole" not authorized when you create or start a job: your user needs iam:PassRole on the job's role.
• Connection timeouts to a database or to S3 from a VPC job: a job with a connection runs in your subnet. Its security group needs a self-referencing inbound rule (all TCP from the same security group), and your subnet needs a route to S3 (an S3 gateway endpoint) and to any other AWS services your job calls, through VPC endpoints or a NAT gateway.
• The crawler creates many tables instead of one: the files in your folder have different schemas or formats. Make them consistent, or in the crawler's output settings choose to create a single schema for each S3 path and set the table level.
• Athena returns HIVE_BAD_DATA after a crawl: a file's contents do not match the table's types. Our HIVE_BAD_DATA guide shows you how to find the bad file.
• Job bookmark not working, every run re-reads everything: bookmarks are disabled on your job, your source lacks a transformation_ctx, or your script never calls job.commit().
• "Command failed with exit code 1": the real error is in your job's CloudWatch logs. Open the run, choose Error logs (log group /aws-glue/jobs/error) and read the Python or Spark traceback at the bottom.
• "Internal service error" or InternalServiceException: usually temporary, or a capacity shortage for the larger G.12X, G.16X and R workers, which can take longer to start. Retry after a few minutes, try fewer workers or a different worker type, and if your crawler or job fails the same way every time, check its role's KMS and S3 permissions before you open a support case.
• Your job runs slower or fails after moving to Glue 6.0: Spark 4.1's ANSI mode and removed APIs (EMRFS, AWS SDK for Java v1) break some old scripts. Fix casts with try_cast, replace EMRFS paths and settings with S3A, and run the generative AI upgrade analysis.
• Is AWS Glue down? Check the AWS Health Dashboard (health.aws.amazon.com) for Glue in your Region. Regional outages are rare; failures are far more often a permission, network or script problem you can see in the logs.

AWS Glue best practices and cost optimization

• Use Glue 6.0 for your new jobs and upgrade the old ones: same work, 30% lower price.
• Use Flex for anything that only needs to finish by a deadline, such as nightly batches.
• Turn on Auto Scaling and set a sensible maximum number of workers.
• Set a job timeout on every job and a %idle_timeout on every notebook.
• Write Parquet or Iceberg, partitioned by the columns your users filter on, and compact small files; this makes every downstream query cheaper.
• Enable job bookmarks for incremental loads.
• Crawl only what changed (S3 event-based crawls), let your jobs update the catalog themselves, or use Athena partition projection.
• Add Data Quality rules at the boundary between your raw and curated data.
• Keep your scripts in Git and deploy jobs with CloudFormation, CDK or Terraform rather than editing in the console.
• Right-size from the metrics: if your executors sit mostly idle in the job's metrics, halve the workers and compare the run time.

Frequently asked questions

A month later the store's analysts had stopped complaining. The nightly job ran on Glue 6.0 Flex for about $15 a month, the curated table was a fifth of the size of the raw CSVs, and the Athena bill had dropped by more than the job cost. The one surprise was a $40 notebook session someone had left open over a weekend; Ethan added %idle_timeout 30 to the team's notebook template and sent Jake a photo of a jar of paste labeled "AWS Glue." Jake replied with a single word: "Kitchen."

Revision note. Written October 21, 2026. Prices are US East (N. Virginia) list prices on that date. We will update this page when Glue prices, versions or features change.