AWS Glue OutOfMemoryError and "Container Killed by YARN for Exceeding Memory Limits", Fixed: Driver vs Executor, Data Skew, collect(), Worker Sizes From G.1X to R.8X, and DPU Cost

AWS Glue OutOfMemoryError or Container killed by YARN for exceeding memory limits means some part of a Spark job exceeded its available memory. The fix depends on which process ran out: the driver, an executor, or an executor container including off-heap memory. Start with the failing stage in Spark UI and the driver/executor logs. Remove driver-side collect(), isolate skewed join keys, keep tiny files from overwhelming the driver, and adjust worker size or count only after identifying the pressure. Here is the expensive misconception: doubling worker count can leave a single oversized partition failing in exactly the same way, because that partition still runs inside one executor.

Jake runs a mobile phone repair business. A daily AWS Glue Spark job combines repair transactions with inventory and supplier records to produce stock forecasts. The pipeline had completed all week, but today it fails halfway through its join.

The job shows a YARN message saying an executor container exceeded memory limits. Jake increases the worker count, reruns the job, and sees the same exception. He assumes AWS Glue ignored the new setting.

The new workers were available, but one hot inventory key sent a huge fraction of the join to a single task. That executor still required more memory than its container could provide. The solution starts with the Spark stage and data distribution, not simply the total cluster size.

Decode the three memory messages before resizing

Your Spark job uses a driver to coordinate work and executors to process partitions. AWS Glue manages the infrastructure, but the same division of responsibilities still matters. The component identified by the stack trace tells you which class of fix is likely to work.

A driver-side java.lang.OutOfMemoryError: Java heap space may arise when application code gathers too many rows or Spark has to track an enormous file listing or execution plan. The failure can occur before meaningful executor work begins, or after a large action returns data to the driver.

An executor-side OutOfMemoryError typically points to transformations such as sorting, joining, caching, aggregating, or processing a very large partition. Each executor has only its assigned memory, no matter how much memory your workers have combined.

The YARN message indicates the container exceeded its physical-memory allocation. The container includes more than Spark heap: native allocations, Python processes, buffers, and overhead can contribute. So the suggested spark.yarn.executor.memoryOverhead is a clue for you, not an instruction to override Glue defaults blindly.

Jake: "Why did adding machines not rescue the job?"

Ethan: "Because the failed join partition still landed on one executor. Total memory and per-task memory are different limits."

Read the failure from the job run and CloudWatch logs

Open AWS Glue, choose your ETL job, and select the exact failed run. Capture its run ID, worker type, number of workers, Glue version, job arguments, and elapsed time. Compare these details with the most recent successful run to rule out a changed deployment or configuration.

Use your run’s Logs view to open CloudWatch logs. In Glue 5.0, error and system information normally appears in /aws-glue/jobs/error; application standard output appears in /aws-glue/jobs/output, unless custom prefixes are configured. Earlier versions can have different log paths.

Search for OutOfMemoryError, Container killed by YARN, ExecutorLostFailure, GC overhead, TaskSetManager, and stage. Read the earliest relevant failure you can find, not merely the final generic job-aborted message.

aws glue get-job-run \
  --job-name repair-inventory-join \
  --run-id jr_EXAMPLE \
  --query "JobRun.{State:JobRunState,Error:ErrorMessage,WorkerType:WorkerType,Workers:NumberOfWorkers,ExecutionTime:ExecutionTime}"

If your job repeatedly loses the same executor at the same stage, investigate data skew or a particular input partition. If the driver disappears while enumerating objects, investigate file counts and driver-side collections. A container message with a peak physical-memory number points toward total container utilization.

Find the exact Spark stage and task that failed

Enable Spark UI event logging and open the Spark UI for your failed run. For supported recent Glue versions, AWS provides a console Spark UI that shows stages, jobs, SQL plans, and executors. Event logs are written to a configured S3 location when the feature is enabled.

Select the last failed job, open its stage, and compare task durations, shuffle read sizes, records, memory spills, disk spills, and failed-task attempts. If you see a single task running much longer than the others, that is a classic skew clue. Spilling is normal at some scale, but enormous spill alongside executor loss suggests pressure.

Use the SQL tab to identify the join or aggregate and see whether the plan performs a sort-merge join, shuffle hash join, or broadcast join. Locate the associated transformation in your script. Confirm the physical plan rather than assuming Spark used your intended strategy.

AWS Glue Observability metrics, supported on Glue 4.0 and newer, include reliability and performance signals. Its skew indicators can help support a diagnosis, but the task-level stage details remain valuable for identifying the precise expensive operation.

# Job arguments to enable Spark UI event logging
--enable-spark-ui true
--spark-event-logs-path s3://example-bucket/glue-spark-events/

Your S3 log bucket must exist, and your job role must be allowed to write events. Turn logging on before your next test run; old executions cannot produce detailed event logs after the fact.

Understand worker sizes and usable memory

You can pick from general-purpose G workers and memory-optimized R workers. Each G-type DPU is defined as 4 vCPUs and 16 GB memory. AWS R-series memory-optimized workers provide twice the memory at corresponding worker-size labels, with memory-optimized pricing that should be checked for the selected Region.

The amounts listed here are worker memory; your Spark heap gets only part of it. Spark, the OS, Python workers, overhead, and service processes consume part of the capacity. Executor heap and usable task memory are consequently smaller.

Larger workers give you more memory per executor. Additional workers usually provide more executors and parallelism. An R worker makes the most sense when your profiler shows memory pressure far above CPU load.

Worker-type availability depends on Glue version, job type, Region, and capacity. G.025X is a 0.25-DPU, 4-GB option intended for low-volume streaming; it is not a general substitute for a large batch join. Some large workers and R workers may have longer startup times.

Jake: "Does R.2X simply run twice as many tasks as G.2X?"

Ethan: "Not necessarily. Its main advantage is more memory for an equivalent size. Use it when a task needs a larger memory envelope."

Understand DPU cost before raising capacity

Glue Spark ETL is billed by the DPU-hour. A standard Spark ETL job costs $0.44 per DPU-hour in US East (N. Virginia), and your rate can differ by Region, Glue features and pricing tier, so check it before you budget.

At that illustrative standard rate, four G.2X workers use 8 DPUs, equivalent to $3.52 for one hour of active billed capacity. Eight G.2X workers use 16 DPUs, equivalent to $7.04 per hour. Four G.4X workers also use 16 DPUs; they provide fewer, larger executor nodes.

These equal-DPU comparisons matter: eight smaller nodes can create more distributed slots, while four larger nodes can accommodate larger tasks. Neither choice universally runs faster. Data skew, shuffle volume, and CPU saturation decide the practical result.

For R workers, check the published regional memory-optimized rate instead of assuming their doubled memory is included in a standard G price. The worked optimization below uses G-worker estimates and separates cost of failed runs from a successful tuned run.

Decide whether more workers or bigger workers will help

More workers increase your aggregate capacity and may improve throughput when your job has enough independent partitions. For evenly distributed scanning, filtering, and joins, increasing worker count can reduce runtime. This is less effective when the input has a single hot key or huge unsplittable file.

Bigger workers give you more executor memory and can solve some memory problems without increasing the number of parallel executors. They are especially relevant when each task is genuinely large or overhead is high. But using larger workers to mask badly skewed data can remain wasteful.

Start with evidence: if every executor is busy with comparable tasks and memory runs high, add parallelism or workers; if one executor disproportionately crashes, fix partition balance and consider a larger node; if the driver is at fault, target driver-side logic and file discovery.

Run an A/B test on the same input snapshot. Record duration, DPU-hours, failure rate, spill amount, and peak memory, then select the lowest-cost configuration that meets reliability targets. Automatic scaling can help variable workloads, but it cannot repair a pathological single partition.

Fix driver OOM from collect and toPandas

Your Spark driver manages distributed work, but APIs such as collect() return all result rows to the driver process. If the result is many gigabytes, the driver can run out of memory even when executors have ample resources.

The same problem applies to toPandas(), which materializes a DataFrame into driver-side memory. A large result that fits distributed across executors can still exceed one driver’s heap and Python memory limits.

# Risky: copies all rows to the driver
rows = joined.collect()
pdf = joined.toPandas()

# Better for inspection
joined.select("order_id", "status").limit(20).show(truncate=False)

# Better for production: keep the result distributed
joined.write.mode("overwrite").parquet(output_path)

If your downstream systems need an aggregated summary, aggregate in Spark before you collect a genuinely small result. A limit() is useful for inspection but is not a universal safeguard against expensive transformations upstream.

Keep full DataFrames and large serialized objects out of your logs. Driver-side maps and dictionaries built from millions of rows can create the same failure even without an explicit collect call.

Fix executor memory failure during joins and aggregations

Joins, sorts, aggregations, window functions, and wide transformations move or retain substantial data. When Spark cannot keep the working set in memory, it may spill to disk. When a partition or operator exceeds available heap and overhead, the executor can fail.

Review your join cardinality first. A one-to-many join can multiply rows dramatically; joining two datasets with duplicate keys on both sides can create a many-to-many explosion. If the result is inherently enormous, no reasonable worker setting can change that mathematical growth.

Filter unused rows and select only the columns you need before the join. Prefer columnar Parquet with predicate pushdown when the input supports it. Deduplicate a supposed dimension table when the business definition requires one row per key.

from pyspark.sql import functions as F

orders_small = orders.select("order_id", "sku", "qty").filter(F.col("sku").isNotNull())
stock_small = stock.select("sku", "stock_level").dropDuplicates(["sku"])

joined = orders_small.join(stock_small, "sku", "left")
joined.write.mode("overwrite").parquet(output_path)

Only deduplicate if duplicates are invalid for the business model. Blindly dropping rows might make the Spark job succeed while silently losing valid information. Verify expected join cardinality using grouped key counts before changing the data.

Detect and correct data skew in a join

Data skew occurs when some partition keys contain vastly more records than others. A hash partitioner sends equal join keys to the same partition, so one frequently occurring key can overload one task while the rest of the cluster becomes idle.

Look for one task that handles far more shuffle bytes or takes much longer than its peers; that is usually your hot key. In a repair inventory dataset, an empty SKU, default UNKNOWN value, or one extremely popular item might account for a disproportionate percentage of rows.

from pyspark.sql import functions as F

orders.groupBy("sku").count()     .orderBy(F.desc("count"))     .limit(15).show(truncate=False)

Turn on or inspect Adaptive Query Execution (AQE) for Spark SQL joins when your Glue version supports it. Adaptive execution can optimize partitions and split skewed join work in appropriate scenarios. Validate the actual plan rather than assuming AQE eliminates all hot keys.

# Example Spark SQL configurations: validate for your Glue version
--conf spark.sql.adaptive.enabled=true
--conf spark.sql.adaptive.skewJoin.enabled=true
--conf spark.sql.shuffle.partitions=200

Where AQE cannot adequately balance the data, split hot keys into a separate processing path or use carefully designed salting. Salting distributes a hot key over several partitions, but requires an appropriate strategy on both join sides and can multiply data. Test counts to prove correctness.

Use repartition and coalesce deliberately

The number and size of your partitions directly shape task memory. Too few partitions can force large chunks of input into a small number of executors. Too many partitions increase scheduling overhead and can burden the driver.

repartition(n, columns) performs a shuffle to distribute records, potentially reducing oversized partitions when keys are reasonably balanced. coalesce(n) usually reduces the number of partitions without a full shuffle, which is useful for certain output workloads but can worsen imbalance before a memory-intensive operation.

# Rebalance based on join keys when key distribution is healthy
orders_balanced = orders.repartition(200, "sku")
stock_balanced = stock.repartition(200, "sku")

result = orders_balanced.join(stock_balanced, "sku")

# Avoid coalescing to one partition before a huge write
result.write.mode("overwrite").parquet(output_path)

The number 200 is a testing starting point, not a recommended constant for every 40-GB job. Use the Spark UI and your partition sizes to adjust it. If a key is exceptionally hot, repartitioning by the same key will not split that key’s data.

Repartitioning adds shuffle cost. Add repartitions only where they help, not after every transformation. Prefer targeted changes before expensive wide operations and verify that the planner did not introduce needless extra exchanges.

Prevent driver pressure from millions of small files

A dataset with hundreds of thousands or millions of tiny S3 objects can exhaust driver resources while Spark enumerates files and manages metadata. This is different from a single oversized join task: the driver may fail before most executors receive useful work.

Compaction is your durable solution. Rewrite small input records into appropriately sized Parquet files in a controlled upstream process. Use partition pruning to avoid discovering irrelevant paths; where supported, push predicates down before loading all files.

Glue also provides workload partitioning and grouping mechanisms for some input patterns. They help break up very large backlogs, but should be evaluated against your exact source format and Glue version.

A single unsplittable compressed object can create the opposite problem: one large input task. Use splittable formats and balanced files when feasible. Data-file layout is part of Spark memory engineering, not merely S3 housekeeping.

Tune Spark settings that actually matter in Glue

Useful SQL-level settings include spark.sql.shuffle.partitions, spark.sql.adaptive.enabled, and spark.sql.adaptive.skewJoin.enabled. Their values affect task parallelism and adaptive processing for supported queries.

spark.sql.autoBroadcastJoinThreshold influences whether Spark may broadcast a small relation. Broadcasting can reduce large shuffles, but an incorrectly forced broadcast of a large table risks driver and executor memory pressure. Use explicit broadcast hints only when measured table size supports the choice.

Spark executor memory overhead is relevant to YARN container deaths because non-heap allocations matter. However, AWS Glue manages many Spark resources using its own worker model. Overriding low-level heap or overhead settings can conflict with container sizing and is not a reliable substitute for selecting appropriate workers and fixing data distribution.

Inspect the effective Spark configuration in Spark UI before tuning. Apply one change at a time, record the result, and verify support before you copy EMR or self-managed Spark settings into Glue.

Worked example: rescue a failed 40 GB join

In this illustrative case, Jake joins 40 GB of repair transactions with 6 GB of inventory history. The input consists of Parquet files in S3. The original Glue Spark run uses four G.2X workers: 8 DPUs of provisioned capacity and 32 GB of memory per worker.

The job reads successfully but fails during a sort-merge join. Spark UI shows most tasks finishing quickly while one task processes far more shuffle data, spills heavily, and loses its executor. The error ends with Container killed by YARN for exceeding memory limits.

A key-frequency check shows that a placeholder SKU UNKNOWN accounts for 35% of transactions. Joining on this value against multiple inventory records produces a huge intermediate result. First, Jake verifies that unknown SKUs should not join against the inventory reference table and routes those records to a separate data-quality output.

from pyspark.sql import functions as F

orders = spark.read.parquet("s3://example-bucket/repairs/")
stock = spark.read.parquet("s3://example-bucket/inventory/")

unknown = orders.filter(F.col("sku") == "UNKNOWN")
valid = orders.filter((F.col("sku") != "UNKNOWN") & F.col("sku").isNotNull())

# Preserve anomalous records for review; do not silently discard
unknown.write.mode("overwrite").parquet("s3://example-bucket/quarantine/unknown-sku/")

stock_one_row = stock.orderBy(F.col("updated_at").desc())     .dropDuplicates(["sku"])  # Illustrative only: nondeterministic after shuffle

joined = valid.select("order_id", "sku", "qty")     .repartition(200, "sku")     .join(stock_one_row.select("sku", "available_qty"), "sku", "left")

joined.write.mode("overwrite").parquet("s3://example-bucket/results/")

One caution about that code: dropDuplicates() after orderBy() keeps an arbitrary row per SKU, because the order is not guaranteed after a shuffle. For a predictable record, build stock_latest with a window function instead, keeping the latest timestamp per SKU and breaking ties on inventory_record_id (the example assumes that field exists and is unique), then join against stock_latest in place of stock_one_row:

from pyspark.sql import Window

w = Window.partitionBy("sku").orderBy(
    F.col("updated_at").desc(),
    F.col("inventory_record_id").desc()
)
stock_latest = stock.withColumn("rn", F.row_number().over(w))     .filter(F.col("rn") == 1).drop("rn")

After fixing the data logic, Jake tests six G.2X workers (12 DPUs) with the same snapshot and enables AQE. The example assumes the run completes in 18 minutes rather than the original failed run stopping at 25 minutes. These timings are illustrative, not measured AWS benchmark results.

At the example standard $0.44 per DPU-hour rate, a successful 12-DPU run lasting 18 minutes costs about $1.58 in compute (12 × 0.3 × $0.44). The original failed 8-DPU run lasting 25 minutes would cost about $1.47, producing no output. The small increase in one-run compute spend can be worthwhile if it replaces repeated failures.

The primary rescue came from correcting hot-key semantics and the join plan, not only resizing. If the hot key were legitimate and could not be separated, Jake would test AQE skew handling, salting, and larger G.4X or R workers based on the largest observed task.

Jake: "So the fix was not to jump straight to the biggest worker?"

Ethan: "Correct. We first removed an invalid join explosion, then used a measured amount of extra parallel capacity. Bigger workers are a fallback for tasks that truly need more memory."

Make broadcast joins safer

A broadcast join copies a sufficiently small side of a join to executor nodes and can avoid a large shuffle. This is often efficient for a large fact table joined to a tiny validated reference table.

However, if the supposed small side expands unexpectedly or a manual broadcast hint forces a multi-gigabyte dimension, the driver and executors can run out of memory. Examine actual statistics and the physical plan before forcing broadcast.

from pyspark.sql.functions import broadcast

# Only when stock_latest is genuinely small enough for broadcasting
joined = valid.join(broadcast(stock_latest), "sku", "left")
joined.explain(True)

If the dimension does not fit comfortably in memory, use a shuffle join with appropriate partitioning or reduce the dimension to necessary columns first. Prefer measured improvements to blanket broadcast hints.

Handle Python UDF and native-memory overhead

Your PySpark tasks may use Python worker processes in addition to the executor JVM. Large Python objects, poorly bounded UDFs, and Arrow or Pandas conversions can increase container resident memory even when JVM heap metrics appear moderate.

The YARN message reports total container memory pressure, so a native-memory or Python-process spike can kill a container without a classic Java heap exception. Inspect the code paths that materialize lists or load large models per row or partition.

Replace your row-wise Python UDFs with built-in Spark SQL functions when you can. For unavoidable Python processing, initialize reusable objects carefully, process bounded batches, and measure the Python memory footprint.

Raising spark.executor.memory will not solve a native allocation problem. A larger worker or reduced Python batch sizes may be more appropriate, and the cluster configuration must have room for overhead.

Use caching and checkpointing with restraint

cache() and persist() can improve repeated computations, but they consume executor storage memory. Caching a very large intermediate relation can compete with join and shuffle memory, increasing spills or OOM risk.

Cache only if you reuse the DataFrame enough to justify its memory cost. Select needed columns before caching, and call unpersist() when a cached intermediate is no longer needed.

trimmed = orders.select("order_id", "sku", "qty")
trimmed.persist()
# Reuse in more than one downstream operation when justified
trimmed.unpersist()

Checkpointing can shorten expensive lineage in some workflows, but it introduces storage and recomputation tradeoffs. It is not a general cure for data skew or incorrect join cardinality. Confirm the suspected bottleneck before adding new materialization steps.

Compare Auto Scaling and bounded workload strategies

Glue Auto Scaling can add or remove workers within the bounds you configure for supported jobs, helping workloads with changing parallel demand. It can reduce wasted provisioned capacity and improve throughput when work is distributed across many independent tasks.

But Auto Scaling does not guarantee that a single hot partition will fit into memory. A task still runs within one executor and inherits its node-specific limits. If the problem is skew, repartitioning or data-model changes remain necessary.

For extremely large backlogs, consider splitting processing into bounded batches based on time, partitions, or other business-safe boundaries. Smaller runs reduce the maximum working set and make retries more targeted. AWS Glue includes workload partitioning features for supported use cases.

Batch boundaries must preserve correctness. If a join relies on data across partitions, splitting without carrying the needed dimension keys can produce incomplete results. Validate output counts and reconciliation totals between the original and bounded designs.

Watch for fixes that simply move the failure

If you add workers without changing partition balance, the same executor can keep crashing. Reducing spark.sql.shuffle.partitions to make a job faster can create larger partitions and increase memory usage. Setting coalesce(1) for convenient output can force excessive work into one task.

Blindly increasing the broadcast threshold can turn a shuffle problem into a broadcast memory problem. Similarly, collecting debug results to the driver can introduce a new driver OOM even after an executor issue has been fixed.

Changing every Spark memory property at once makes your diagnosis much harder. Glue worker configurations already reserve resources for various processes; a copied setting from a self-managed Spark cluster can be counterproductive.

Treat every change as a hypothesis. A useful fix improves completion rate, stage balance, peak memory, and total cost on repeatable data, not simply the appearance of the job configuration.

Python shell jobs have different memory constraints

AWS Glue Python shell jobs are not Spark ETL jobs. They run Python scripts without the Spark driver/executor architecture described here. Accordingly, spark.sql.shuffle.partitions, Spark UI shuffle stages, and YARN executor tuning do not apply to an ordinary Python shell job.

For Python shell, AWS Glue supports 0.0625 DPU and 1 DPU allocations, with 0.0625 DPU as the default. One standard DPU corresponds to 4 vCPUs and 16 GB of memory, but actual usable Python-process memory is subject to the runtime environment and overhead. Python shell jobs get 20 GB of local disk.

Your Python shell script using pandas can still run out of memory if it loads a huge file or constructs a massive in-memory DataFrame. In that situation, use chunked reads, streaming approaches, or move the workload to distributed Spark ETL when appropriate.

These job types also differ in supported Python versions and runtime behavior. Python shell 3.6 has reached end of life, and Python 3.9 is the version to specify for Python shell jobs. Spark worker sizing tables do not apply to Python shell capacity.

Build a repeatable memory-debugging checklist

For every OOM incident, record your first failing log entry, failing stage ID, last successful stage, executor ID, input bytes, task skew, worker size, worker count, and job arguments. Save the exact input snapshot or partition range used in the test.

Next, classify the issue as driver metadata/collection, an executor task, whole-container overhead, or excessive data growth from a join. Each classification has a different highest-value fix. Collecting evidence before a costly rerun reduces guesswork.

Run a baseline and one controlled change. Reconcile row counts and business totals; a job that completes by losing unmatched records is not a successful repair. Then compare per-stage time, shuffle, spills, worker utilization, and actual billed DPU-hours.

Finally, add monitoring. Glue job-run failures can trigger EventBridge notifications, while CloudWatch metrics and Glue Observability help identify recurring memory pressure. Your alerts should point to the job name and run ID, so whoever responds can open the right logs right away.

Frequently asked questions about Glue OOM and YARN memory limits

These answers focus on the distinctions most likely to change your next troubleshooting action. The full diagnosis still depends on the job-run logs, Spark physical plan, and input distribution.

What causes OutOfMemoryError in AWS Glue?

Driver collections, large file listings, joins, wide shuffles, skewed partitions, caches, and Python memory growth can exceed the allocated driver or executor memory. Identify which process failed before resizing.

What does Container killed by YARN for exceeding memory limits mean?

A Spark executor container exceeded its allowed physical-memory allocation. Heap, off-heap and native allocations, and Python processes may contribute. Inspect the executor logs and failing task.

How do I distinguish driver OOM from executor OOM?

Driver OOM appears in the driver process and often follows collect, toPandas, or metadata pressure. Executor OOM points to task-level joins, aggregations, partitions, or native overhead. Use the specific log stream and Spark UI.

How much memory does a G.1X Glue worker have?

G.1X has 16 GB of worker memory and 4 vCPUs. The amount available to the Spark executor heap is lower after system and overhead reservations.

How much memory does a G.2X Glue worker have?

G.2X has 32 GB of memory, 8 vCPUs, and 2 standard DPUs. Each G.2X worker runs one Spark executor.

How much memory do G.4X and G.8X have?

G.4X has 64 GB of worker memory and G.8X has 128 GB. They provide larger per-worker memory envelopes but cost more DPUs per hour.

What is the advantage of Glue R workers?

R.1X, R.2X, R.4X, and R.8X have 32, 64, 128, and 256 GB respectively. They are intended for memory-intensive Spark workloads; check regional availability and pricing.

Should I add more Glue workers or use bigger workers?

Add workers when balanced parallel tasks need more aggregate capacity. Use larger or memory-optimized workers when an individual task needs more executor memory. Fix skew before paying to scale it.

How do I find the Spark stage that failed?

Open the failed Glue job run and its Spark UI. Find the failed stage and inspect task attempts, shuffle read, spill, runtime, and executor losses. Correlate with driver and executor CloudWatch logs.

How do I handle skewed joins in AWS Glue?

Identify hot keys with grouped counts and inspect skewed stage tasks. Use supported AQE skew-join optimization, correct invalid keys, or split and salt legitimate hot-key processing with output validation.

Does increasing spark.sql.shuffle.partitions fix memory errors?

It may reduce per-task data size when partitions are reasonably balanced, but increases overhead and does not split a single hot join key assigned to one partition. Validate task sizes after changing it.

Why does collect cause Glue driver out-of-memory errors?

collect brings distributed rows into the driver process, which has finite memory. Use distributed writes and aggregations, or collect only a genuinely small final result.

Do many small S3 files cause Glue OOM?

Yes. Discovering and managing enormous numbers of files can exhaust driver memory. Use compaction, partition pruning, and supported workload grouping or bounded processing.

How much does increasing Glue workers cost?

Cost generally rises with DPU-hours. Standard Spark ETL costs $0.44 per DPU-hour in US East (N. Virginia), but actual rates and features depend on Region and worker type.

Are Glue Python shell memory limits the same as Spark?

No. Python shell jobs use 0.0625 or 1 DPU allocations and do not have Spark driver/executor stages. Their pandas or Python processes can still exhaust available memory.

Can Glue Auto Scaling solve an OOM error?

Auto Scaling can help jobs with balanced parallel work by adjusting total worker capacity. It cannot necessarily fix one oversized task, invalid join cardinality, or a driver collecting too much data.

Final checks before rerunning the Spark job

When the job is ready for another run, verify that the resource settings were actually saved to the intended Glue job. Check its Region, worker type, number of workers, Glue version, script path, and runtime arguments. Make sure the run uses the same data snapshot as the failed baseline if you want a fair comparison.

Examine the Spark plan to confirm that Spark is no longer processing the problematic join as a single huge task. Ensure that the output is complete by reconciling row counts, unique order IDs, and inventory totals. Preserve quarantined anomalous records rather than silently deleting them.

For Jake’s repair-shop pipeline, this means correcting the hot SKU behavior, selecting a measured worker configuration, and recording memory and shuffle metrics after the fix. If the job later grows, the team can recognize the same pattern quickly instead of repeatedly guessing at worker counts.

The lasting lesson is that Glue memory failures are usually easier to solve when you identify the exact failing process and stage. Use bigger workers for larger tasks, more workers for distributed parallel work, and better data design for skew and explosive joins. I hope your next ETL run finishes reliably and with fewer wasted DPU-hours.

Revision note. Written October 19, 2026. "Many hands make light work," the proverb says; with Spark, many hands help only when the work is shared out evenly.