Athena HIVE_BAD_DATA and HIVE_CURSOR_ERROR, Fixed: Find the Bad S3 File, CSV Header Rows and Blank Numbers, Broken gzip, Parquet Type Changes, and Querying Around One Bad File

Amazon Athena HIVE_BAD_DATA means a source value cannot be interpreted as the column type Athena expects, while HIVE_CURSOR_ERROR indicates Athena could not correctly read or decode an underlying data file. The fastest repair is not to change every column to STRING or rerun a Glue crawler: find the actual S3 object, compare its physical format with the table schema and SerDe, and decide whether to repair the source or change the table. A single newly uploaded CSV with a header row, a blank numeric field, a corrupt gzip stream, or a Parquet column written under a different type can break queries that worked yesterday. Below is a file-first troubleshooting method, including a recoverable CSV example and safe ways to query healthy data while isolating a bad file.

SELECT DISTINCT "$path" AS source_file
FROM repairdb.bookings_raw
ORDER BY source_file;

Jake manages a phone repair business whose daily repair bookings arrive in Amazon S3 as CSV files. Athena gives him a simple dashboard of devices received, estimated repair costs, and completed orders. The query has worked reliably for several weeks, and the analyst has not edited the SQL.

On Monday morning, the dashboard fails with HIVE_BAD_DATA. Jake suspects AWS changed the query engine because the SQL is unchanged. Ethan asks what changed in the S3 folder, and they discover a new export with a header row and a blank value in the numeric repair cost field.

The lesson for you: Athena is a schema-on-read engine. Your table definition does not repair bytes in S3. When Athena encounters a newly added incompatible object, a formerly successful query can fail without any change to the SQL, the table name, or the dashboard.

Understand the difference between HIVE_BAD_DATA and HIVE_CURSOR_ERROR

HIVE_BAD_DATA commonly describes a mismatch between a value in a file and the type declared in the table. A column declared INT may contain the text repair_id, an empty string, a decimal value, or another token that its SerDe cannot convert. The table can exist perfectly well in the catalog while its records are inconsistent.

HIVE_CURSOR_ERROR is a more general read or decode failure. Athena may be unable to decompress an object, read a Parquet page, parse a JSON document, or use the configured serializer/deserializer (SerDe) on the actual bytes. The error does not always imply a physically damaged file: an unexpected format or invalid record can cause it.

Because the failures originate during scanning and deserialization, a SQL LIMIT 10 is not a trustworthy workaround. A query engine can read file splits in a different order, and a corrupt split may still be visited even when only a few result rows are requested.

The error name narrows your investigation, but the text after the colon usually tells you more. Capture your full query ID, workgroup, error message and timestamp. Where available, preserve the referenced object path and specific parse or compression exception.

Jake: "Both errors mention Hive. Is Athena itself broken?"

Ethan: "Usually the table is trying to read bytes that do not match its assumptions. Start with the object and the complete error text."

Read the complete error and identify the offending S3 object

Open the Athena console, select your workgroup, and choose the failed query in recent query history. Expand its error details. Some errors include the fully qualified S3 URI, such as s3://example-bucket/bookings/date=2026-10-18/part-002.parquet. Copy it exactly before you change anything.

If your message includes only a column name or a parse failure, inspect the table location using SHOW CREATE TABLE and list the S3 objects under that prefix. Sort the files by modification time; the most recent arrival is a useful clue when the same SQL worked yesterday.

The hidden "$path" column returns the S3 source path for rows that Athena can successfully read. It gives you a powerful way to list source files or check which object produced a known row, but it cannot guarantee your query will finish when one file is unreadable. Scanning the corrupt file may cause the path query itself to fail.

For a failing query without an exposed filename, isolate the data by partition, then by progressively smaller S3 prefixes using temporary external tables. Another option is inspecting file manifests from the ingestion job. Hold off on deleting the newest object until you have confirmed it is the bad one.

SHOW CREATE TABLE repairdb.bookings;

SELECT DISTINCT "$path" AS source_file
FROM repairdb.bookings
WHERE booking_date = '2026-10-17'
ORDER BY source_file;

aws s3 ls s3://example-bucket/bookings/ --recursive
aws s3api head-object \
  --bucket example-bucket \
  --key bookings/date=2026-10-18/part-002.csv

Understand why the table schema can disagree with a real file

Athena does not ingest and permanently convert every row when you create a normal external table. It stores metadata in the AWS Glue Data Catalog and interprets external S3 objects only when a query reads them. This is why your table can be created without errors even though a later query fails.

A schema declaration such as repair_cost INT expresses your expectation. It does not mean every future exporter will produce a valid integer. Empty fields, currency symbols, thousands separators, quoted CSV anomalies, and header text can all break that expectation.

The choice of SerDe matters as well. OpenCSVSerde and LazySimpleSerDe do not have identical behavior for quoted fields and numeric parsing. In particular, OpenCSVSerde does not safely translate every empty numeric field to SQL NULL. Test with your real data, not only an ideal sample.

When your source format is unreliable, a raw-zone table with columns defined as strings is often safer. A cleaned view or CTAS step can validate and convert fields explicitly. Keep the original raw data available for auditing while making the curated table strict and predictable.

Fix an INT column containing a header row, empty field, or text

If an exporter writes the header booking_id,repair_cost,status into every file, a numeric table can attempt to parse the text repair_cost as an integer. For ordinary CSV with one header line per file, the table property skip.header.line.count tells Athena to ignore that header.

Use this property with care. If some files include a header and others do not, skipping one line from every file discards valid data. Standardize your exporter or separate the file layouts before you decide that a single table property is safe.

An empty numeric cell gives you a different problem. Skipping the header will not transform a blank string into an integer. A robust rescue pattern is to expose the field as a raw string and apply TRY_CAST(NULLIF(TRIM(repair_cost_raw), '') AS INTEGER) in a view or cleanup query.

TRY_CAST returns NULL when conversion fails instead of failing the query. That is useful for diagnosis, but it must be accompanied by validation or quarantine rules: otherwise bad financial values can quietly disappear from summaries.

ALTER TABLE repairdb.bookings
SET TBLPROPERTIES ('skip.header.line.count'='1');

SELECT booking_id,
       TRY_CAST(NULLIF(TRIM(repair_cost_raw), '') AS INTEGER) AS repair_cost
FROM repairdb.bookings_raw;

Find compressed files whose contents or extensions do not match

When your HIVE_CURSOR_ERROR message includes incorrect header check, invalid block type, or Unexpected end of input stream, verify the compressed object itself. A file named .gz must contain a valid gzip stream, not merely uncompressed text with an altered suffix.

Compression mismatches can also happen after transfers that truncate an upload or when a pipeline writes a partially completed object to the final destination. Inspect the object size, its checksum when you have one, and how your writer completes files.

For a gzip object, download it to a controlled machine and run gzip -t. If that test fails, fix your source export and replace the object with a valid copy. Renaming a broken archive does not make its payload correct.

Even if gzip integrity passes, a file may still contain invalid CSV or JSON. Decompression integrity and record schema validation are independent checks. A successful compression test only clears one layer of potential failures.

aws s3 cp s3://example-bucket/bookings/bad.csv.gz ./bad.csv.gz
gzip -t bad.csv.gz
file bad.csv.gz

gzip -dc bad.csv.gz | head -n 5

Stop Athena from scanning mixed CSV, JSON, and Parquet in one prefix

Athena generally reads every object under your table's S3 location or selected partition. If the prefix mixes CSV files with JSON or Parquet files, the chosen SerDe can attempt to interpret incompatible bytes and fail.

Configuring AWS Glue crawler exclude patterns does not guarantee Athena excludes those objects. The crawler affects catalog discovery, while an Athena table location controls which S3 objects Athena attempts to read. This difference can make a crawler appear successful while a query still produces HIVE_CURSOR_ERROR.

Your durable fix is to separate file types and schema generations into distinct S3 prefixes, such as raw/csv/, raw/json/, and curated/parquet/. Point each external table at the appropriate prefix.

Keep your logs, success markers, manifests and temporary outputs out of a queryable data prefix. Athena hides certain filenames beginning with a dot or underscore, but relying on filename tricks is not a substitute for a clean data layout.

-- Different formats must point at different data locations.
CREATE EXTERNAL TABLE repairdb.bookings_csv (
  booking_id string, repair_cost string, status string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES ('separatorChar'=',','quoteChar'='"')
LOCATION 's3://example-bucket/raw/bookings/csv/';

CREATE EXTERNAL TABLE repairdb.bookings_parquet (
  booking_id string, repair_cost double, status string
)
STORED AS PARQUET
LOCATION 's3://example-bucket/curated/bookings/parquet/';

Investigate Parquet schema evolution rather than forcing a table change

Parquet stores its own physical schema and column metadata. Athena reads Parquet columns by name by default. If the table declares amount DECIMAL(10,2) but a newly written Parquet object stores that field as an incompatible string representation, the reader may fail even though earlier objects worked.

Parquet can accommodate some schema evolution, including adding or reordering columns when read by name. That does not make arbitrary physical type changes safe. A writer changing INT32 to UTF-8 string or introducing incompatible decimal precision can trigger read or schema verification errors.

Inspect the physical Parquet schema of both a known-good object and your failing object using a tool such as parquet-tools schema or PyArrow. Compare column names, types, decimal precision and scale, nested structures, nullability, and writer output.

Athena supports a parquet.column.index.access SerDe property that changes name-based access to index-based access. Changing that flag can enable certain rename scenarios, but it is not a general cure for incompatible physical types. Repair your writer or rewrite inconsistent objects to a single canonical schema.

parquet-tools schema good-file.parquet
parquet-tools schema suspect-file.parquet

Handle known Parquet cursor errors and statistics problems carefully

Some Parquet cursor errors arise while reading metadata or page statistics rather than while converting a row value. One workaround for certain Parquet read failures is the parquet.ignore.statistics SerDe property.

Treat this setting as a targeted experiment after you examine the exact error. It is not a universal repair for corrupt files, incompatible schemas, or unreadable compression blocks. If the source object is truly truncated or its pages are damaged, rewrite or replace it.

Keep in mind that Athena engine versions can differ in type coercion behavior. A query that happened to tolerate a physical string where a decimal was expected may fail after an engine upgrade because newer validation is stricter. Compare your workgroup's engine version when you diagnose a regression.

For production data quality, generate consistent Parquet from your own controlled transformation job, validate schema changes before publication, and write new output atomically to a versioned prefix. Then switch the table location only after validation.

-- Investigative workaround for specific Parquet statistics errors.
ALTER TABLE repairdb.bookings_parquet
SET TBLPROPERTIES ('parquet.ignore.statistics'='true');

Model nested JSON correctly and respect the chosen JSON SerDe

A JSON object can include nested data such as {"customer":{"id":"C104"}}. Declaring customer as a flat string field may not represent that structure correctly. Use a suitable STRUCT or inspect the source as raw JSON and extract the nested value.

For JSON-lines processing, the Hive JSON SerDe expects each JSON document to be a single line. Pretty-printed multi-line JSON can cause HIVE_CURSOR_ERROR with messages saying a row is not a valid JSON object or the end of input was unexpected.

OpenX JSON SerDe offers useful mapping behavior but also has its own parsing rules and options. Your choice of SerDe and record format matters at least as much as the field names in the catalog.

If several JSON generations coexist in the same prefix, transform them to one normalized shape. Before you mask malformed records, find out how often they occur and which producer wrote them.

CREATE EXTERNAL TABLE repairdb.bookings_json (
  booking_id string,
  customer struct<id:string,name:string>,
  repair struct<amount:double,status:string>
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://example-bucket/raw/bookings/jsonl/';

SELECT booking_id, customer.id, repair.amount
FROM repairdb.bookings_json;

Worked example: rescue a CSV dataset broken by one new file

Assume the S3 prefix contains day-01.csv, day-02.csv, and day-03.csv. The first two files contain valid rows. The third file arrived with a repeated header and a blank repair_cost value. Jake’s table declares repair_cost INT, so the new object causes HIVE_BAD_DATA.

Before making changes, download the suspect object and inspect its contents. The first line contains booking_id,repair_cost,status. One later row contains B-302,,OPEN. Skipping the header addresses only the first anomaly; the numeric blank remains.

The safer recovery is to create a raw table that treats repair cost as text, uses OpenCSVSerde to handle properly quoted CSV fields, and skips one known header line per file. Run diagnostics with TRY_CAST to identify rows where the numeric field is invalid. Then correct the exporter and replace the bad file, or publish validated output to a curated prefix.

This example intentionally preserves records with missing repair costs in the raw table. Whether they are retained as NULL, rejected, or returned to the producer for correction is a business rule, not something Athena should silently decide.

-- Illustrative contents of day-03.csv:
booking_id,repair_cost,status
B-301,850,OPEN
B-302,,OPEN
B-303,1200,CLOSED

CREATE EXTERNAL TABLE repairdb.bookings_raw (
  booking_id string,
  repair_cost_raw string,
  status string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
  'separatorChar'=',',
  'quoteChar'='"'
)
LOCATION 's3://example-bucket/repair-bookings/csv/'
TBLPROPERTIES ('skip.header.line.count'='1');

SELECT "$path" AS file_path, booking_id, repair_cost_raw
FROM repairdb.bookings_raw
WHERE NULLIF(TRIM(repair_cost_raw),'') IS NULL
   OR TRY_CAST(TRIM(repair_cost_raw) AS INTEGER) IS NULL;

The diagnostic query highlights the blank row. Next create a cleaned view that exposes typed values but keeps the original source path for traceability. Use an explicit rule for blanks and invalid values, and keep a missing price from being read as zero.

CREATE OR REPLACE VIEW repairdb.bookings_clean AS
SELECT booking_id,
       TRY_CAST(NULLIF(TRIM(repair_cost_raw),'') AS INTEGER)
           AS repair_cost,
       status,
       "$path" AS source_file
FROM repairdb.bookings_raw;

SELECT * FROM repairdb.bookings_clean
WHERE repair_cost IS NULL;

After fixing the exporter and replacing the object with valid numeric values, refresh the query. A stricter curated Parquet table is a good next step for analytics workloads. The essential outcome is that the problem is located, explained, and corrected—not merely hidden with a permissive query.

Jake: "The header property fixed part of it, but the query still found a blank price. Is that another error?"

Ethan: "Yes. A header row and an empty numeric value are different defects. The raw string table lets us see both without making up a repair price."

Query healthy data while one file is being repaired

You have several ways to keep unaffected data accessible. The safest method is to select known-good partitions or point a temporary table at a clean S3 prefix. Partition pruning can prevent Athena from scanning a bad partition when the filter is applied to an actual partition key.

The hidden "$path" column can help identify and selectively query files that deserialize successfully. However, a WHERE "$path" != ... predicate is not a guaranteed way to avoid opening an unreadable object in every plan. A failing SerDe may encounter the object before the row-level filter can exclude it.

When your reporting must keep running, quarantine the invalid object outside the table prefix, or publish a new known-good prefix and repoint a temporary table. Copy before removing any production object, and coordinate with the ingestion pipeline so the invalid file is not immediately written again.

For a partitioned table, a query restricted to yesterday’s healthy partition can often run while today’s bad partition is investigated. This is more predictable than relying on LIMIT, row ordering, or a post-read predicate.

-- Only works as intended if dt is a real partition key:
SELECT count(*)
FROM repairdb.bookings_partitioned
WHERE dt = '2026-10-17';

-- Useful for healthy files, not guaranteed to bypass a corrupt file:
SELECT *
FROM repairdb.bookings_raw
WHERE "$path" = 's3://example-bucket/repair-bookings/csv/day-01.csv';

Understand partition projection and why it does not repair corrupted data

Partition projection tells Athena how to derive partition values and S3 paths from table properties instead of retrieving every partition from the Glue catalog. It helps with predictable partition layouts and can reduce partition metadata overhead.

It is not a data-cleaning tool. If a projected location contains corrupt CSV or Parquet, Athena can still fail when that partition is scanned. Similarly, an incorrectly configured projection range or location template can cause queries to read the wrong prefix or return no results.

Use projection.enabled with each partition column’s supported projection type and range or values. A custom storage.location.template must contain placeholders for every projected partition column and point to the intended S3 layout.

When partition projection is enabled, Athena ignores registered partition metadata for that table. Test your projection changes in a staging table and verify the chosen locations with representative filtered queries before enabling them on an established production table.

CREATE EXTERNAL TABLE repairdb.bookings_projected (
  booking_id string, repair_cost_raw string, status string
)
PARTITIONED BY (dt string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES ('separatorChar'=',','quoteChar'='"')
LOCATION 's3://example-bucket/bookings/'
TBLPROPERTIES (
  'skip.header.line.count'='1',
  'projection.enabled'='true',
  'projection.dt.type'='date',
  'projection.dt.range'='2026/01/01,NOW',
  'projection.dt.format'='yyyy-MM-dd',
  'storage.location.template'='s3://example-bucket/bookings/dt=${dt}/'
);

Why a Glue crawler does not automatically fix HIVE_BAD_DATA

A Glue crawler discovers data structure and updates the Data Catalog. It does not automatically rewrite malformed CSV rows, replace a corrupt compressed file, or normalize every Parquet writer’s physical types. A crawler can infer a different schema, but that metadata change alone may be insufficient.

A crawler may inspect a sample that does not contain the bad row and infer INT. The query can later fail on an invalid record outside the sample. Conversely, a crawler may widen a column to STRING after encountering mixed values, which avoids one conversion problem but changes downstream query semantics.

Partition-specific schema differences are another concern. Glue may register a partition with a schema different from its table. Depending on format and read mode, Athena can reject that mismatch before querying data. This is distinct from a broken file.

Treat crawlers as discovery automation, not quality assurance. A durable pipeline validates records, enforces a canonical schema, isolates invalid files, and publishes only approved objects to curated prefixes.

Explain why the query worked yesterday and fails today

In most cases, your answer is new data. Someone uploaded an incompatible object, a scheduled export changed field formatting, or a partition with a different schema became visible. The SQL query was stable; the set of source bytes was not.

Other plausible changes include a Glue crawler modifying catalog schema, a job switching Parquet writer settings, a workgroup changing Athena engine version, or an object being overwritten with an incomplete compressed file. Review your CloudTrail and pipeline logs for the relevant changes.

A query can also appear intermittent when some predicates prune problematic partitions and others force Athena to scan them. A daily report constrained to an old date might succeed, while the unrestricted weekly report reaches the new object and fails.

Compare the successful query ID and failed query ID, their workgroups and engine versions, the table definition at those times, and S3 modification dates. You should be able to connect the first failure to a changed input, format, or configuration.

Jake: "The report ran fine for weeks. How can nothing change and still break it?"

Ethan: "Something did change: the files. The SQL stayed the same, but Athena read a new object it could not parse."

Distinguish HIVE_BAD_DATA from a SQL CAST error

In HIVE_BAD_DATA, Athena often fails while deserializing source data to satisfy the external table schema. The query may fail even if you never explicitly wrote CAST, because Athena tries to interpret the file as the types declared in the catalog.

A SQL cast error happens later in query evaluation. If the table already exposes a value as VARCHAR and your query executes CAST(value AS INTEGER), an invalid string can trigger INVALID_CAST_ARGUMENT or a related conversion error.

The repair boundary is different. To address a file-read failure, change the physical input, SerDe, or raw table schema. To address an explicit SQL cast, inspect the expression and choose validation, filtering, or TRY_CAST for expected bad values.

TRY_CAST cannot rescue a column whose underlying SerDe cannot produce rows. The engine must successfully deserialize the source field before a query-level expression can operate on it.

-- Explicit cast might raise INVALID_CAST_ARGUMENT:
SELECT CAST(repair_cost_raw AS INTEGER)
FROM repairdb.bookings_raw;

-- Diagnostic conversion, yielding NULL on invalid input:
SELECT booking_id,
       TRY_CAST(NULLIF(TRIM(repair_cost_raw),'') AS INTEGER)
FROM repairdb.bookings_raw;

Prevent recurrence with ingestion contracts and quarantine

Treat your schema as a contract between producer and consumer. Define the expected file format, encoding, compression type, delimiter, header policy, field names, numeric conventions, null representation, and partition path. Test every producer change against your representative files.

Upload into a staging prefix, run format and schema checks, and publish to the Athena-visible prefix only after validation. Failed files should enter a quarantine location with a diagnostic record containing the object key and reason for rejection.

For Parquet, enforce a consistent writer schema including decimals and nested fields. For CSV, be explicit about header rows and empty numeric cells. For JSON, choose newline-delimited objects or another format supported by the selected SerDe.

Include the source file path and ingestion timestamp in your curated records when you can. That makes future investigations easier, especially when a single error affects hundreds of partitions or multiple producer teams.

A repeatable file-first troubleshooting checklist

Begin by capturing your exact error text, query ID, execution engine and table name. Find the table location and determine whether the exception identifies a specific S3 object. Save a copy of suspect files before replacing or moving them.

If the error indicates a field parsing problem, inspect the column data type, headers, nulls, delimiter, and SerDe. If the error indicates reading or decompression, test the actual file contents, not merely its extension. For Parquet, compare the physical schemas of healthy and failing files.

Use partition isolation or temporary tables aimed at small prefixes to find the failure boundary. A LIMIT or a "$path" filter will not always avoid a malformed object, and your production data should stay put until you have an approved recovery plan.

Finally, choose a lasting fix. Change the table only when its declared schema is wrong; correct or rewrite files when the producer violated the contract. Validate the repaired query and add an automated ingestion check so the same defect cannot silently return.

Use file-by-file and partition-by-partition isolation without losing evidence

When your error lacks a file path, make the investigation systematic rather than guessing. Start with the table location and known partitions, then narrow the scanned set. If a monthly query fails, test separate day partitions. Once you locate a day with the failure, inspect object listings within that prefix and compare each object to the export contract.

Create temporary external tables with the same SerDe and columns but increasingly narrow LOCATION values. For example, a temporary table pointed only at the dt=2026-10-18/ prefix helps determine whether the error is unique to that partition. The next temporary table can point to a staging prefix containing copies of one or two suspect files.

S3 prefixes are logical key-name ranges, not real filesystem directories. Copy only the objects you intend to test into an isolated prefix. Keep the diagnostic copy away from the original table location, or Athena may read both versions. Preserve the original object key, ETag or checksum when available, size, and modified timestamp in your incident notes.

For compressed or columnar files, keep the bytes intact instead of splitting them with text utilities. A gzip stream or Parquet row group has structure that must remain intact. Use format-aware inspection tools and the same parsing library family as the producer where possible.

If several query failures occur at once, compare their offending prefixes and upload timestamps. A common upstream batch or writer deployment is a stronger clue than independent Athena bugs. This analysis also tells you whether a temporary query restriction is safe or would omit critical business data.

The query that isolates bad files might still fail until all files in the selected prefix are readable. Treat each failed test as evidence about the group being scanned rather than assuming its SELECT predicate was ineffective or Athena ignored your WHERE clause.

Diagnose CSV and JSON parsing details that mimic schema problems

CSV is deceptively complicated once values can contain quotes, commas, or line breaks. A record such as "Screen, cracked",850,OPEN requires quote-aware parsing. If a table uses a simple delimiter SerDe instead of the expected CSV SerDe, a comma inside a quoted value can shift the apparent column positions.

OpenCSVSerde supports configurable separators and quote characters, but it does not support every possible multiline CSV structure. If source fields contain embedded line breaks, normalize them with a suitable ETL parser before querying rather than assuming that all quoted multiline exports are safe in Athena.

For JSON, separate structural validity from the table mapping. Each JSON-lines record needs to be a valid standalone JSON object in the form expected by the selected SerDe. A file containing one pretty-printed object spanning many physical lines can fail even when a general JSON parser accepts the whole file.

A nested JSON field may require a STRUCT declaration, and an array requires an array-compatible schema. If the same producer sometimes sends a scalar and sometimes an object for the same field, neither a flat STRING nor a fixed STRUCT is a perfect long-term contract without normalization.

Use a small representative sample containing both typical and edge-case values. Validate empty strings, literal null values, non-ASCII text, commas, escape sequences, and unexpected fields. Test the same file with the same SerDe settings that Athena uses, not just with a generic editor.

Ignoring malformed JSON records can be acceptable for an intentionally best-effort exploratory view, but it is risky for financial or compliance reporting. Maintain counters for rejected records, retain raw files, and treat unexplained differences between source and query row counts as data-quality incidents.

Know when a table fix is appropriate and when files must be rewritten

Change table metadata when the catalog is wrong about valid, consistent source bytes. For example, if every CSV writer legitimately uses decimal repair costs, declaring the field as INTEGER was the mistake. Updating the table to a suitable type or using a raw STRING representation may solve the mismatch without changing source data.

Rewrite files when the producer violated an otherwise correct contract. If the dataset contract requires integers but one file contains a header in every tenth row or arbitrary words such as unknown, changing the entire analytics schema to accommodate invalid input may weaken reports unnecessarily.

For Parquet, physical type mismatches generally call for a controlled rewrite. Changing the Glue type from string to decimal cannot magically reinterpret an incompatible Parquet encoding. Normalize decimal precision, scale, and compatible nested structures in a staging transformation before publishing.

For compression failures, a file extension proves nothing about what is inside. A failed gzip integrity check or a truncated Parquet footer signals a file-level issue. Replace the object with a known-good export and document the production failure that permitted partial publication.

Keep your schema changes versioned. A separate versioned external table or a new curated prefix makes it possible to test a new producer contract before replacing the production table. This is particularly valuable when multiple dashboards rely on the same column names and types.

When making a temporary permissive raw table, explicitly label it as diagnostic. Consumers should not assume every record is valid simply because a SELECT succeeds. Enforce typed, tested views or curated output as the default interface for downstream analytics.

Build a regression test before returning the table to production

A repaired query is only the beginning. Verify that your dataset still contains the expected records. Compare source object counts with ingestion manifests, and compare row totals before and after the repair for every affected partition. A solution that quietly drops invalid rows can be worse than a visible error.

Test critical numeric columns for unexpected NULL values introduced by TRY_CAST. Group them by "$path" where possible to identify any source file still violating the contract. If a report depends on complete repair prices, quarantine missing-price records rather than including them in an average as though they were legitimate zeroes.

For Parquet, read a sample across both old and new partitions, not just the latest object. A schema change that fixes a new file can break an older one. Inspect representative nested fields and decimal values, and verify partition columns match the catalog layout.

For JSON, test both valid and deliberately malformed records in a staging table to make sure your chosen SerDe behavior is understood. For compression, verify checksums or integrity before the object enters the queryable prefix. Your pipeline, not Athena, should be the first system to discover a truncated export.

Keep a stable recovery path: store the original object, corrected object, validation results, and changed table DDL under a change record. If the new schema creates a downstream regression, that evidence lets you restore the earlier location or table definition without guessing.

Finally, add pipeline alerts for validation failures and quarantine growth. A data quality system that silently moves thousands of records to quarantine is not operationally healthy. A useful alert includes source prefix, producer, partition, failed validation rule, and affected object count.

Frequently asked questions

What does HIVE_BAD_DATA mean in Amazon Athena?

It means Athena could not interpret a source value as the external table’s declared type. A header string or empty numeric field in an INT column is a common example.

What does HIVE_CURSOR_ERROR mean in Athena?

It indicates a failure reading or decoding an input file or record. Common causes include malformed JSON, damaged compression, incompatible SerDes, and certain Parquet read problems.

How do I find the file causing an Athena query error?

Read the full query error for an S3 path. If none appears, inspect the table location and isolate partitions or prefixes. For rows that can be read, the hidden "$path" column identifies their S3 object.

Can I query around a bad file in Athena?

Query known-good partitions or use a temporary table aimed at a clean S3 prefix. Filtering by "$path" is useful but not guaranteed to prevent a corrupt object from being opened.

Does LIMIT 10 prevent HIVE_CURSOR_ERROR?

No. LIMIT does not guarantee that the query engine avoids an unreadable split or object. Repair or isolate the underlying file.

How do I skip CSV headers in Athena?

Use the table property skip.header.line.count set to the known header line count. Apply it only when files consistently use that header format.

How do I handle an empty INT field in CSV?

Use a raw STRING column and convert with TRY_CAST and NULLIF, or fix the exporter to produce the expected representation. Keep missing financial values from turning into zeros.

Why did my Athena query work yesterday?

A new S3 object, modified partition, changed Glue schema, different Parquet writer, or engine upgrade can make previously successful SQL encounter incompatible data.

Will an AWS Glue crawler fix HIVE_BAD_DATA?

Not necessarily. A crawler changes catalog metadata and does not repair malformed S3 objects. It may even infer an unsuitable schema from a sample.

Why does Athena fail when CSV and JSON share a prefix?

A table’s SerDe can attempt to read objects in its S3 location regardless of their extensions. Separate formats into distinct prefixes and tables.

How does Parquet schema evolution work in Athena?

Parquet reads columns by name by default, supporting certain column additions and reordering. Incompatible physical type changes can still fail and often require rewriting files.

Can I ignore Parquet statistics to fix HIVE_CURSOR_ERROR?

Athena has a parquet.ignore.statistics property that helps certain known Parquet reader failures. It is not a universal repair for damaged files or incompatible schemas.

Does partition projection repair malformed files?

No. Partition projection computes partition values and paths. Files selected through those paths must still be readable and schema-compatible.

Why does a nested JSON field fail against a flat Athena column?

The declared schema or SerDe may not match the JSON structure. Use an appropriate STRUCT mapping or normalize the JSON before querying.

What is the difference between HIVE_BAD_DATA and INVALID_CAST_ARGUMENT?

HIVE_BAD_DATA often happens while Athena deserializes a file against the table schema. INVALID_CAST_ARGUMENT commonly comes from an explicit query CAST on an incompatible value.

Should I change every Athena column to STRING?

A raw STRING landing table is useful for investigating inconsistent text data, but curated tables should enforce meaningful types. Validate bad rows and fix producer contracts instead of hiding them.

For Jake, the decisive discovery was the new CSV file, not an AWS outage. A header row and a blank numeric field violated the schema that Athena was asked to enforce. A raw string table exposed those defects, and a corrected export restored the trusted report.

When your Athena query fails only on some records or files, begin with the full error, the source path, and the actual object bytes. Once the table and file format agree, a repeatable validation process can keep tomorrow’s upload from breaking today’s working report.

Revision note. Written October 19, 2026. "Garbage in, garbage out," the old computing saying goes; with Athena, garbage in usually means no output at all.