AWS Glue ETL, crawlers and job bookmarks
AWS Glue is the main serverless ETL service on the DEA-C01 exam. Most questions ask which Glue feature solves a specific pipeline problem with the least effort.
The Data Catalog and crawlers
The AWS Glue Data Catalog is a central metadata store: databases, tables, schemas and partitions. It is Hive-metastore compatible. Athena, Redshift Spectrum, EMR and Glue jobs all use it.
A crawler scans a data store (S3, JDBC databases, DynamoDB), uses classifiers to detect the format, infers the schema and writes tables to the catalog. You can run it on a schedule or on demand.
A crawler creates one table for a prefix when the files share a compatible format, compression type and schema. Mixed schemas or inconsistent folder structures cause many tables. Use an exclude pattern or separate prefixes to control this.
If you already have a Hive metastore, import its metadata into the Data Catalog with a Glue job or a migration utility instead of rebuilding it by hand.
Glue jobs
| Job type | Use it for |
|---|---|
| Spark ETL | Large batch transforms, serverless Apache Spark |
| Spark Streaming | Continuous ETL from Kinesis or Kafka |
| Python shell | Small tasks, no Spark needed |
| Ray | Python workloads that scale across nodes |
Capacity is measured in DPUs or in worker types (G.1X, G.2X). Turn on auto scaling so Glue adds and removes workers during the run. Glue reads from and writes to S3, Redshift, RDS and many other sources through connections. A DynamicFrame is Glue’s schema-flexible version of a Spark DataFrame, and it handles inconsistent record types well.
Job bookmarks
A job bookmark stores state between runs so a job only processes new data. For S3 sources it tracks which objects were already read. For JDBC sources it uses a key column that only increases, such as an ID or timestamp.
Bookmarks must be set to Enable on the job. Disabled (the default), a job reprocesses everything. Use Pause to run again without advancing the bookmark, and reset the bookmark to reprocess from scratch.
Triggers and workflows
A Glue workflow chains crawlers and jobs. Triggers start them on a schedule, on demand, or when a previous job or crawler finishes. For event-driven starts, an Amazon EventBridge rule can start a workflow when a file lands in S3.
Exam traps
- Workflows and triggers control order and timing. They do not give incremental loading. That is job bookmarks.
- A classifier detects the format of data. It does not decide how many tables are created.
- Job bookmarks do not remove duplicates from a target. Duplicate rows after a failed rerun need an upsert pattern, such as a staging table plus MERGE in Redshift.
- Python shell jobs are not distributed. Large datasets need a Spark job.
- Crawlers update metadata only. They never change the data files.
Practice questions
Choose an answer, then review the explanation to check your understanding.
1. A nightly AWS Glue job reads new files from an S3 prefix and loads them into Amazon Redshift. Files from earlier runs keep being loaded again. What is the most likely cause?
Show answer
Answer: B.Without job bookmarks enabled, Glue keeps no record of what it has already processed, so every run reads all the files again.
2. A crawler creates many tables for one S3 prefix, but the team expects one table. What helps the crawler create a single table?
Show answer
Answer: B.Crawlers group files into one table when the format, compression and schema are compatible and the prefix layout is consistent.
3. A team needs to convert 2 MB CSV files arriving in S3 to Parquet with the least operational effort and no servers to manage. Which choice fits best?
Show answer
Answer: B.AWS Glue is serverless Spark ETL and can write Parquet directly, so there is no cluster to manage.
4. Where does a Glue crawler store the table definitions it discovers?
Show answer
Answer: B.Crawlers write databases and tables to the AWS Glue Data Catalog, which Athena, Redshift Spectrum and EMR can read.