What Is Amazon EMR? Full Form, Pricing, EMR Serverless vs EC2 vs EKS, Spark Tutorial, EMR vs Glue and Error Fixes

Amazon EMR (formerly Elastic MapReduce) is AWS's managed big data platform: it runs open-source engines such as Apache Spark, Hive, Trino, Presto, HBase and Flink on data in S3, without you installing or tuning a Hadoop cluster. You can run it three ways: EMR on EC2 (clusters of EC2 instances you size and control), EMR Serverless (submit a Spark or Hive job and pay only while it runs) and EMR on EKS (Spark jobs on your Kubernetes cluster). On EC2 you pay the EC2 price plus an EMR fee per instance-hour, about 25 percent on top: an m5.xlarge costs $0.192 an hour for EC2 plus $0.048 for EMR, billed per second. EMR Serverless costs $0.052624 per vCPU-hour and $0.0057785 per GB-hour of memory (about 20 percent less on Graviton), and EMR on EKS adds a small uplift to your EKS costs. There is no free tier. The two habits that decide your bill are terminating clusters when jobs finish and running worker nodes on Spot Instances. Below: what EMR is and how it works, EC2 vs Serverless vs EKS, every price with worked examples, a step-by-step Spark tutorial, Spot and managed scaling, EMR vs Glue, Databricks, Athena and Redshift, limits, and fixes for the errors that stop Spark jobs.

The furniture store's Kinesis pipeline had been archiving every website click to S3 for two years: about 2.4 TB of compressed JSON. The owner wanted "customers who looked at this sofa also bought..." suggestions on product pages, recalculated every night from all of it. Athena could query the data, but the recommendation logic needed real code: joins across months of sessions, sessionizing clicks, and counting co-purchases. Jake asked whether a bigger laptop would do. Ethan said 2.4 TB does not fit on a laptop, and opened the EMR console.

Building a data platform on AWS? The cards below cover Glue, Athena, Spot and Savings Plans, Lake Formation and billing alerts, the pieces you will most often combine with EMR.

What is Amazon EMR?

Big data means datasets too large or too fast for one computer: billions of clicks, years of transactions, terabytes of logs. The open-source answer is distributed processing: split the data into pieces, process them on many machines at once, and combine the results. Apache Spark is the most widely used engine for that today, alongside Hive (SQL over large files), Trino and Presto (fast interactive SQL), HBase (a wide-column database) and Flink (stream processing). Running these yourself means installing them on a cluster, configuring YARN, tuning memory, patching and replacing failed nodes.

Amazon EMR is AWS's managed platform for these engines. You choose the applications and the capacity; EMR installs, configures and connects everything, reads and writes data in Amazon S3 (so the data outlives any cluster), integrates with the Glue Data Catalog for table definitions, and scales and shuts down on your terms. Each EMR release (for example the 7.x line) is a tested bundle of versions: Spark, Hadoop, Hive, Trino, Iceberg, Hudi, Delta Lake and the rest, built and optimized by AWS. The EMR runtime for Spark is tuned to run many Spark jobs faster than open-source Spark on the same hardware.

Four concrete examples: a retailer joins a year of point-of-sale records with weather data to forecast demand per store; a media company processes 50 TB of video player logs every night into viewing statistics; a bank recomputes risk models over millions of transactions with Spark MLlib; a genomics lab aligns sequencing reads across thousands of samples with open-source bioinformatics tools on Spark. Each is too big for one machine, runs on a schedule, and reads and writes S3.

What people use it for: large ETL jobs that clean and join raw data into tables, building data lakes and lakehouses with Iceberg, Hudi or Delta tables, machine learning feature engineering and training data preparation, clickstream and log analysis, genomics and scientific computing, interactive SQL with Trino or Presto, and real-time processing with Flink or Spark Structured Streaming.

Amazon EMR full form: EMR stands for Elastic MapReduce. When AWS launched it in 2009, it ran Apache Hadoop's MapReduce, the original programming model for big data: a map step processes pieces of the data in parallel on many machines (for example, counting words in each file), and a reduce step combines the partial results (adding up the counts). "Elastic" meant you could grow and shrink the cluster at will, the same idea as EC2's "elastic". Today the service is simply called Amazon EMR, and MapReduce itself is rarely used: Spark replaced it for most work because it keeps data in memory between steps and is many times faster, but the name stuck.

Jake: "So it rents you a hundred computers that already know how to share the work."

Ethan: "For an hour, if that's all the job needs. Then it gives them back, and the data stays in S3."

How EMR works: clusters, nodes and storage

1. You create a cluster (or a serverless application) and choose an EMR release and applications, such as Spark, Hive and Livy.
2. EMR launches EC2 instances in three roles: one primary node (formerly called master) that coordinates the cluster and runs YARN's resource manager, core nodes that run tasks and store HDFS data, and optional task nodes that only run tasks, which makes them ideal for Spot Instances.
3. You submit work as steps (for example a spark-submit of a Python script in S3), from a notebook in EMR Studio, or over SSH or Livy.
4. Spark splits the job into tasks, YARN schedules them across the nodes, and executors read the input from S3, process it in parallel and shuffle data between nodes as needed.
5. Results are written back to S3 as Parquet files or Iceberg tables, where Athena, Redshift and dashboards can use them.
6. The cluster scales or terminates: managed scaling adds and removes nodes with the load, and auto-termination shuts the cluster down after it has been idle for the time you set.

Transient vs long-running clusters: a transient cluster starts, runs its steps and terminates, the cheapest pattern for scheduled batch jobs. A long-running cluster stays up for interactive users, HBase or streaming; it needs managed scaling and a budget alert. Because the data lives in S3 rather than on the cluster's disks (HDFS is only for temporary data), losing or deleting a cluster loses nothing but its compute.

EMR architecture in more detail: YARN, HDFS, EMRFS and Spark

Four layers make up an EMR on EC2 cluster, and knowing them makes tuning and debugging much easier:

• Storage: EMRFS lets every engine read and write S3 as if it were a file system (s3://bucket/path), and is where your real data should live. HDFS, spread across the core nodes' disks, holds temporary data such as shuffle files and intermediate results, and disappears with the cluster. Local disks on every node hold spill and cache.
• Resource management: YARN (Yet Another Resource Negotiator) runs on the primary node and hands out containers, slices of memory and CPU, on the core and task nodes to applications that ask for them. Most memory errors are really YARN container limits.
• Processing engines: Spark, Hive, Trino and the others request containers from YARN. In Spark, the driver plans the job and the executors run its tasks in parallel, one task per data partition per core.
• Applications and interfaces: the step API, EMR Studio notebooks, Livy for REST job submission, the Spark history server and the YARN resource manager UI, all reachable from the console.

For a sense of scale: a cluster of ten m6g.2xlarge core nodes (8 vCPU and 32 GiB each) gives Spark roughly 70 cores and over 200 GB of executor memory after YARN and the operating system take their share, enough to process a few terabytes of compressed data in well under an hour for most transformations. EMR Serverless hides all four layers: you set the worker size and Spark settings, and it handles the rest.

EMR on EC2 vs EMR Serverless vs EMR on EKS

	EMR on EC2	EMR Serverless	EMR on EKS
What you manage	Cluster size, instance types, scaling settings	Only the job and optional capacity limits	Your EKS cluster and its nodes
Engines	Spark, Hive, Trino, Presto, HBase, Flink, and more	Spark and Hive	Spark (and Flink)
Pricing	EC2 + EMR fee per instance-second	vCPU, memory and extra storage per second while workers run	EKS and EC2 or Fargate + EMR uplift per vCPU and GB
Startup	Several minutes for a new cluster	Seconds to minutes (instant with pre-initialized capacity)	As fast as pods schedule
Spot savings	Yes, especially for task nodes	No	Through your EKS node groups
Best for	Heavy, steady or custom workloads; HBase; many engines	Occasional or unpredictable Spark and Hive jobs	Teams already standardized on Kubernetes

A simple rule: start with EMR Serverless for scheduled Spark jobs that run for minutes to a few hours a day; there is nothing to size or forget. Move to EMR on EC2 when jobs run most of the day, when you want Spot savings, need engines Serverless does not run, or want full control of the configuration. Choose EMR on EKS if your platform team already runs everything on Kubernetes and wants Spark there too.

Amazon EMR pricing

US East (N. Virginia) prices in October 2026. EMR on EC2 adds a per-instance EMR fee to the normal EC2 price (and EBS volumes), all billed per second with a one-minute minimum:

Instance (4 vCPU)	Memory	EC2 On-Demand per hour	EMR fee per hour	Total per hour
m5.xlarge	16 GiB	$0.192	$0.048	$0.240
m6g.xlarge (Graviton)	16 GiB	$0.154	$0.039	$0.193
m7g.xlarge (Graviton3)	16 GiB	$0.1632	$0.0408	$0.204
r5.xlarge	32 GiB	$0.252	$0.063	$0.315
r6g.xlarge (Graviton)	32 GiB	$0.2016	$0.0504	$0.252
c7g.xlarge (Graviton3, compute)	8 GiB	$0.145	$0.03625	$0.181

• The EMR fee is fixed whether the EC2 instance is On-Demand, Reserved, under a Savings Plan or Spot, so Spot discounts apply to the EC2 part only.
• EMR Serverless: $0.052624 per vCPU-hour and $0.0057785 per GB-hour on x86; on ARM (Graviton) $0.042094 and $0.004628, 20 percent less. Each worker gets 20 GB of disk free; extra standard storage up to 200 GB per worker costs $0.000111 per GB-hour. Billing runs per second, with a one-minute minimum, from when workers are ready until they stop.
• EMR on EKS: an uplift of $0.007084 per vCPU-hour and $0.000777875 per GB-hour on the vCPU and memory your pods request, plus your EKS cluster ($0.10 an hour) and its EC2 or Fargate capacity.
• EMR Studio (the browser IDE with Jupyter notebooks) has no charge of its own; you pay for the compute it uses.
• No free tier: EMR has no free allowance, though accounts created from July 15, 2025 can spend Free Tier credits on it.
• Also on the bill: S3 storage and requests for your data, EBS volumes on cluster nodes, CloudWatch logs, data transfer across Availability Zones or Regions, and the Glue Data Catalog if you go past its free tier.

Worked examples: what EMR really costs

Months are 720 hours; EBS and S3 costs are small and left out unless noted.

Example 1: the store's nightly recommendation job, as a transient cluster. One primary and four core m6g.xlarge nodes for 1.5 hours a night: 5 x 1.5 x 30 = 225 instance-hours x $0.193 = $43.43 a month.

Example 2: the same cluster left running. Five m6g.xlarge nodes around the clock: 5 x 720 x $0.193 = $694.80 a month, sixteen times more for the same nightly result.

Ethan: "Forty-three dollars if it shuts down after the job. Six hundred and ninety-five if someone forgets."

Jake: "So the expensive part is forgetting."

Ethan: "Which is why the cluster terminates itself after 15 idle minutes, whether anyone remembers or not."

Example 3: the job on EMR Serverless. 16 workers of 4 vCPU and 16 GB for 75 minutes a night: 16 x 4 x 1.25 = 80 vCPU-hours x $0.052624 = $4.21, plus 16 x 16 x 1.25 = 320 GB-hours x $0.0057785 = $1.85: $6.06 a night, $181.80 a month on x86, or about $145 on Graviton. Higher than the transient cluster, but with no cluster to size, start or forget.

Example 4: a big batch with Spot task nodes. A 3-hour daily job on 1 primary and 2 core r6g.xlarge On-Demand nodes plus 10 r6g.xlarge Spot task nodes. The On-Demand nodes cost 3 x 3 x 30 x $0.252 = $68.04. If Spot runs at 70 percent below On-Demand, each task node costs about $0.0605 for EC2 plus the full $0.0504 EMR fee, so 10 x 3 x 30 x $0.1109 = $99.81. About $168 a month, against $294.84 if the task nodes were On-Demand.

Example 5: a long-running interactive cluster. A primary m5.xlarge and 3 core r5.xlarge for analysts during working hours (10 hours a day, 22 days a month), with managed scaling adding up to 6 task nodes at peak (say 2 extra nodes on average): (1 x $0.24 + 5 x $0.315) x 220 = $399.30 a month.

Example 6: Graviton vs x86. The same five-node cluster for a month on m5.xlarge costs 5 x 720 x $0.240 = $864; on m6g.xlarge, $694.80. Most Spark jobs run unchanged on Graviton, so it is a 20 percent saving for changing an instance type.

Serverless or a cluster: where is the break-even? For the same 4 vCPU and 16 GB of capacity, EMR Serverless costs about $0.303 an hour on x86 (4 x $0.052624 + 16 x $0.0057785) and $0.242 on Graviton, while an m6g.xlarge on EMR on EC2 costs $0.193 an hour, before Spot. So a cluster is roughly 20 to 40 percent cheaper per hour of work when it is busy, but you also pay for its startup time, its primary node, and any idle minutes before it terminates. For jobs that run under an hour or two a day, the difference is a few dollars and Serverless's simplicity usually wins; for jobs that keep capacity busy most of the day, or that can use Spot, clusters win clearly.

How to cut your EMR bill

• Auto-termination: set an idle timeout (for example 15 minutes) on every cluster that is not deliberately long-running, and run scheduled jobs on transient clusters that terminate after their last step.
• Spot Instances for task nodes: task nodes hold no HDFS data, so losing one only re-runs some tasks. Use instance fleets with several instance types (for example r6g, r7g, r5 and m6g in the xlarge and 2xlarge sizes) so EMR can find Spot capacity; keep the primary and core nodes On-Demand for stability. Our guide to Spot interruptions in a pipeline covers the two-minute warning.
• Managed scaling: set a minimum and maximum, and EMR adds or removes nodes from YARN metrics, so you pay for peaks only during peaks.
• Graviton instances: about 20 percent cheaper for EC2 and the EMR fee, with similar or better Spark performance.
• Savings Plans and Reserved Instances cover the EC2 part of long-running clusters; the EMR fee stays. See our decision tree.
• Right-size memory: memory-optimized r-family nodes for joins and caching, compute-optimized c-family for CPU-bound transforms; check Spark UI for idle executors.
• Columnar formats: converting raw JSON or CSV to Parquet or Iceberg once makes every later job read a fraction of the data, the biggest saving in most data lakes.
• Budget alerts: a billing alert and a tag policy (owner, purpose) catch forgotten clusters in days, not months.

EMR tutorial: run a PySpark job on a cluster

1. Put your data and script in S3: the input (for example s3://store-datalake/clicks/), the PySpark script, and an output prefix. Create an S3 location for cluster logs too.
2. Open EMR in the console, choose EMR on EC2, then Create cluster.
3. Release and applications: pick the latest EMR 7.x release and the Spark application bundle (add Livy and JupyterEnterpriseGateway if you will use notebooks).
4. Cluster configuration: choose Instance groups or Instance fleets; one primary node, two or more core nodes (m6g.xlarge or r6g.xlarge are good starting points), and optional task nodes on Spot.
5. Scaling and termination: turn on managed scaling with a sensible maximum, and set automatic termination after, say, 15 minutes idle.
6. Networking and security: choose a VPC subnet (a private subnet with an S3 gateway endpoint is ideal), an EC2 key pair if you want SSH, and the default IAM roles (EMR_DefaultRole for the service and EMR_EC2_DefaultRole for the instances, which needs read access to your input and write access to your output).
7. Add a step: type Spark application, the S3 path of your script, and its arguments. Set the action on failure to Terminate cluster for a transient job.
8. Create the cluster. It starts in a few minutes, runs the step, writes the output to S3, and terminates. Logs land in your S3 log location and in the console.

A small PySpark script, saved as s3://store-scripts/also_bought.py:

import sys
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("also-bought").getOrCreate()
src, dst = sys.argv[1], sys.argv[2]

orders = (spark.read.json(src)
          .where(F.col("type") == "order")
          .select("session_id", "sku"))

pairs = (orders.alias("a").join(orders.alias("b"), "session_id")
         .where(F.col("a.sku") < F.col("b.sku"))
         .groupBy(F.col("a.sku").alias("sku"), F.col("b.sku").alias("also"))
         .count()
         .where(F.col("count") >= 5))

pairs.write.mode("overwrite").parquet(dst)
spark.stop()

The same cluster and step from the AWS CLI:

aws emr create-cluster --name also-bought-nightly --release-label emr-7.13.0 \
  --applications Name=Spark --use-default-roles --log-uri s3://store-emr-logs/ \
  --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m6g.xlarge \
                    InstanceGroupType=CORE,InstanceCount=4,InstanceType=m6g.xlarge \
  --auto-terminate \
  --steps Type=Spark,Name=also-bought,ActionOnFailure=TERMINATE_CLUSTER,Args=[s3://store-scripts/also_bought.py,s3://store-datalake/clicks/,s3://store-datalake/also-bought/]

--auto-terminate shuts the cluster down after the last step; for clusters that should stay up between jobs, use --auto-termination-policy IdleTimeout=900 instead. Release labels change several times a year, so check the console for the current 7.x label before you copy the command.

EMR Serverless in practice

With EMR Serverless you create an application (Spark or Hive, with a release label and architecture), then submit job runs to it. EMR finds workers, runs the job, and releases them; the application stops on its own after an idle period (15 minutes by default) and starts again with the next job.

aws emr-serverless create-application --name also-bought --type SPARK \
  --release-label emr-7.13.0 --architecture ARM64 \
  --maximum-capacity '{"cpu": "200 vCPU", "memory": "800 GB"}'

aws emr-serverless start-job-run --application-id 00fabcdexample \
  --execution-role-arn arn:aws:iam::111122223333:role/EMRServerlessJobRole \
  --job-driver '{"sparkSubmit": {"entryPoint": "s3://store-scripts/also_bought.py",
      "entryPointArguments": ["s3://store-datalake/clicks/", "s3://store-datalake/also-bought/"],
      "sparkSubmitParameters": "--conf spark.executor.cores=4 --conf spark.executor.memory=14g"}}'

• Worker sizes: from 1 vCPU with 2 to 8 GB up to 16 vCPU with up to 120 GB; Spark's executor settings map onto them.
• Maximum capacity on the application is your safety cap: jobs cannot use more vCPU or memory than that in total.
• Pre-initialized capacity keeps a number of workers warm so jobs start in seconds; you pay for them while they are warm, so use it for interactive or latency-sensitive jobs only.
• Logs and Spark UI: job logs go to S3 or CloudWatch, and the console opens the Spark UI for running and finished jobs.
• Scheduling: trigger job runs from Step Functions, MWAA (managed Airflow) or EventBridge Scheduler.

Configurations, bootstrap actions and monitoring

Configurations set application properties when the cluster starts, as JSON classifications. For example, to turn on Spark's dynamic allocation and adaptive execution and raise the executor memory overhead:

[
  {"Classification": "spark-defaults",
   "Properties": {
     "spark.dynamicAllocation.enabled": "true",
     "spark.sql.adaptive.enabled": "true",
     "spark.executor.memoryOverhead": "2g"
   }}
]

Bootstrap actions are scripts in S3 that run on every node before applications start, for installing Python libraries, agents or certificates. Keep them short and idempotent: a failing bootstrap action fails the whole cluster. For Python dependencies on EMR Serverless, package them into a virtual environment archive or a custom image instead.

Monitoring: CloudWatch receives EMR metrics such as IsIdle (useful for alarms on forgotten clusters), YARNMemoryAvailablePercentage, ContainerPending and AppsPending (signs you need more capacity), and HDFS utilization. The console links to the Spark history server, YARN timeline server and Tez UI even after a cluster terminates, through persistent application UIs, which is where you find skewed stages, spills and slow tasks. Step and application logs are archived to your S3 log location.

Data lake tables, notebooks and streaming

• Open table formats: EMR releases include Apache Iceberg, Apache Hudi and Delta Lake, so Spark jobs can write tables with ACID transactions, updates and deletes, time travel and schema evolution on top of S3. Iceberg tables in S3 Tables add automatic compaction.
• Hudi, Iceberg or Delta? All three add transactions and updates to files in S3. Apache Hudi grew up around fast upserts and incremental pulls from change data, and is popular for CDC pipelines; Apache Iceberg has the broadest engine support across AWS (Athena, Redshift, Glue, EMR, S3 Tables) and is the common default for new lakehouses; Delta Lake suits teams that also use Databricks. On EMR you enable each by choosing the matching application or Spark configuration, so the decision is about your other tools, not EMR.
• Glue Data Catalog: use it as the metastore so tables created on EMR appear in Athena, Redshift Spectrum and Glue jobs, and govern access with Lake Formation.
• EMR Studio: a browser IDE with Jupyter notebooks that attach to EMR on EC2 clusters, EMR Serverless or EMR on EKS, with Git integration and SQL explorers, at no extra charge.
• Streaming: Spark Structured Streaming or Flink on EMR can read from Kinesis or Kafka and write continuously to Iceberg tables, though Managed Service for Apache Flink is often simpler for pure streaming.
• Interactive SQL: Trino or Presto on a long-running EMR cluster give analysts fast SQL over the lake when Athena's pay-per-query model is not the right fit.

Security and permissions

• IAM roles: a service role lets EMR manage EC2 on your behalf; the instance profile role is what jobs use to read and write S3, so scope it to your data prefixes. Runtime roles let different steps or users on a shared cluster run with different permissions.
• Fine-grained access: with Lake Formation, EMR enforces table, column and row permissions defined in the catalog.
• Encryption: security configurations turn on encryption at rest (S3 with KMS, local disks, EBS) and in transit (TLS between nodes).
• Network: run clusters in private subnets with an S3 gateway endpoint; block public access to the primary node, and use Session Manager instead of open SSH ports.
• Authentication: Kerberos for multi-user Hadoop clusters, and IAM Identity Center for EMR Studio users.

Choosing instance types, and EMR vs plain EC2

• General purpose (m6g, m7g, m8g): a balanced default for primary nodes and mixed ETL.
• Memory optimized (r6g, r7g, r8g): for joins, aggregations, caching and anything that spills to disk on m-family nodes; the usual choice for Spark core and task nodes.
• Compute optimized (c6g, c7g): for CPU-bound work such as parsing, compression and simple transforms with small shuffles.
• Storage optimized (i4i, i4g, d3) and d variants (r6gd, m6gd): local NVMe disks for heavy shuffles, HDFS-heavy work and HBase.
• GPU (g5, g6): for Spark RAPIDS acceleration and deep learning preprocessing.

Start with Graviton xlarge or 2xlarge nodes, watch the Spark UI and CloudWatch for spills and idle CPU, and adjust. Fewer, larger nodes reduce per-node overhead; more, smaller nodes spread Spot risk.

EMR vs running Spark yourself on EC2: you can install Spark on plain EC2 instances and skip the EMR fee, but then you build and maintain everything EMR provides: cluster provisioning, YARN and HDFS configuration, the optimized Spark runtime, S3 integration tuned for performance and consistency, managed scaling, Spot handling, security integration, logging and upgrades. For most teams the roughly 25 percent fee buys back far more engineering time than it costs; self-managed Spark on Kubernetes (or EMR on EKS) is the usual alternative when a platform team wants full control.

EMR versions and release labels

Every cluster and serverless application runs one release label, such as emr-7.13.0 (released in April 2026), which fixes the versions of every bundled application. The 7.x series runs on Amazon Linux 2023 with Spark 3.5 and Java 17 by default, and is the line to use for most new work; in April 2026 AWS also started a separate AWS runtime for Apache Spark 8.0 release line on EMR on EKS, for teams that want the newest Spark runtime; 6.x releases run on Amazon Linux 2 with Spark 3.x and suit only existing jobs you have not moved to 7.x yet; 5.x releases (Spark 2) are legacy. AWS publishes several 7.x releases a year with new application versions, performance improvements and security fixes. Because a running cluster keeps its release forever, transient clusters pick up new releases naturally, while long-running clusters need to be replaced to upgrade. Test a new release on a copy of your job first: minor Spark changes can alter behavior around dates, nulls and casts.

EMR vs Glue, Databricks, Athena, Redshift and Dataproc

• EMR vs AWS Glue: Glue runs serverless Spark ETL jobs at $0.44 per DPU-hour (1 DPU is 4 vCPU and 16 GB), with built-in crawlers, the Data Catalog, visual job authoring and job bookmarks; there is nothing to size. EMR gives you more engines, full Spark configuration, Spot and Graviton savings, and long-running clusters. The same 4 vCPU and 16 GB on EMR Serverless costs about $0.30 an hour, on an EMR on EC2 m6g.xlarge $0.193, and less on Spot. Choose Glue for straightforward ETL with little operations work, EMR for heavy, custom or cost-sensitive processing.
• EMR vs Databricks: Databricks is a commercial platform built on Spark with its own runtime, collaborative notebooks, Unity Catalog governance, Delta Lake and ML tooling, running in your AWS account and billed in Databricks units on top of EC2. EMR is AWS-native, cheaper per compute hour, and integrates with IAM, Lake Formation and the Glue catalog; Databricks offers a more polished workspace and cross-cloud consistency. Teams pick on features and skills, not raw price alone.
• EMR vs Athena: Athena runs SQL on S3 with no infrastructure, billed per TB scanned; it is perfect for ad-hoc queries and dashboards. EMR is for complex processing in code, machine learning pipelines, and steady heavy workloads where per-query billing would cost more.
• EMR vs Redshift: Redshift is a data warehouse for fast SQL analytics on structured data, with concurrency for many users. EMR processes raw and semi-structured data at scale and often feeds Redshift: EMR cleans and aggregates, Redshift serves the reports.
• EMR vs Kinesis and Managed Flink: for continuous streams, Managed Service for Apache Flink runs Flink without clusters; EMR suits batch work and teams that want streaming and batch on one platform.
• EMR vs Google Dataproc and Azure HDInsight: the equivalent managed Hadoop and Spark services on the other clouds, with similar per-node pricing; choose the one where your data lives.

Limits and quotas

• Cluster size: limited mainly by your EC2 vCPU quotas per instance family and Region; large clusters may need a quota increase, and instance fleets help find capacity.
• Instance fleets: up to 30 instance types per fleet (5 with the console's simple view), with Spot and On-Demand targets.
• Steps: up to 256 active steps per cluster; finished steps stay visible in the history.
• EMR Serverless: per-account vCPU quotas by Region (adjustable), workers from 1 to 16 vCPU and up to 120 GB memory, up to 200 GB of standard disk per worker.
• Logs: cluster logs are archived to your S3 log location every few minutes; keep the bucket in the same Region.

Common EMR errors and fixes

• "Container killed by YARN for exceeding memory limits": an executor used more memory than its container allows, usually from large shuffles or skewed keys. Increase spark.executor.memoryOverhead, use more executors with less memory each, repartition skewed data, or move to memory-optimized instances.
• java.lang.OutOfMemoryError on the driver: the job collected too much data to the driver (collect(), a huge broadcast join). Write results to S3 instead, raise spark.driver.memory, or lower spark.sql.autoBroadcastJoinThreshold.
• "No space left on device": shuffle and spill filled the local disks. Add bigger or more EBS volumes, use instances with NVMe storage (the d variants), or reduce shuffle with better partitioning.
• Cluster TERMINATED_WITH_ERRORS during bootstrap: a bootstrap action script failed, the instance role cannot read its S3 path, or the subnet has no route to S3. Check the bootstrap logs in your S3 log location.
• VALIDATION_ERROR or "The requested instance type is not supported": the release does not support that instance type, or the subnet's Availability Zone does not offer it; pick another type or use instance fleets across several subnets.
• Insufficient capacity: not enough On-Demand or Spot capacity for an instance type in that zone; instance fleets with several types and subnets fix most cases (see our InsufficientInstanceCapacity guide).
• Access Denied on S3: the instance profile role (or the Serverless job role) lacks permission for that bucket or KMS key, or Lake Formation has not granted access to the table.
• S3 503 Slow Down: too many requests to the same prefix at once; EMRFS retries automatically, but spreading output across more prefixes and writing fewer, larger files helps.
• Step failed with no obvious error: open the step's stderr in the console or S3 logs, and the Spark UI (history server) for the failed stage; the real error is usually a few lines above the final stack trace.
• Bill higher than expected: clusters without auto-termination, pre-initialized Serverless capacity left on, or oversized core nodes. Check Cost Explorer by cluster tags.

Best practices

• Keep data in S3, in columnar formats, partitioned by the columns you filter on.
• Prefer transient clusters or Serverless for scheduled jobs; reserve long-running clusters for genuinely interactive or always-on workloads.
• Use Graviton and Spot wherever jobs allow, with instance fleets for capacity.
• Turn on auto-termination and managed scaling on every cluster.
• Use the Glue Data Catalog and Lake Formation so every engine sees the same tables and permissions.
• Pin release labels in your automation, and upgrade deliberately after testing.
• Watch the Spark UI for skew, spills and idle executors before adding hardware.
• Tag clusters and applications with owner and purpose, and set a budget alert.

The habits behind all of these are the same: treat compute as disposable and data as permanent. A cluster you can recreate from a script and a release label in ten minutes is one you never hesitate to terminate, upgrade or resize, and that freedom is where most of EMR's cost and reliability advantages come from.

Key EMR concepts for interviews and AWS exams

• Node roles: primary (coordinates, one per cluster, or three for high availability), core (tasks plus HDFS), task (tasks only, Spot-friendly).
• Storage: persistent data in S3 through EMRFS; HDFS is temporary and tied to the cluster.
• Cluster types: transient (terminate after steps) vs long-running (interactive, streaming, HBase).
• Deployment options: EMR on EC2, EMR Serverless, EMR on EKS, EMR on Outposts.
• Cost levers: Spot for task nodes, instance fleets, managed scaling, Graviton, auto-termination, Savings Plans for the EC2 part.
• Glue vs EMR: Glue for serverless ETL with a catalog and crawlers; EMR for control, more engines and lower cost at scale.
• Security: service role and instance profile, runtime roles, security configurations for encryption, Lake Formation for fine-grained access, Kerberos for multi-user clusters.
• Typical exam answer: "large-scale Spark or Hadoop processing of data in S3 at the lowest cost" usually means EMR with Spot task nodes and a transient cluster.

Frequently asked questions

The recommendation job now runs at 2 a.m. on a five-node Graviton cluster that exists for about 80 minutes, reads two years of clicks from S3, and writes a small Parquet table of "also bought" pairs that the website reads in the morning. The first night it failed with a YARN memory error on one skewed sofa that half the visitors had looked at; a repartition fixed it. The monthly EMR bill is about $43. Jake asked whether the cluster minded working nights. "It doesn't exist during the day," Ethan said. "That's the whole trick."

Revision note. Written October 21, 2026. Prices are US East (N. Virginia) list prices on that date. We will update this page when EMR prices or features change.