search

AWS Glue suite

hashtag
The AWS Glue Ecosystem

Think of "Glue" not just as one tool, but as a suite of tools for "Serverless Data Integration."

AWS Glue (The ETL Engine)

  • What it is: A serverless environment to run Python or Scala code (usually Apache Spark) to transform data.

  • Use Case: Reading messy JSON files from S3, cleaning them up, and writing them as Parquet tables.

  • Data Engineer Note: Under the hood, it just spins up a temporary Spark cluster, runs your script, and shuts down. It eliminates the need to manage EC2 servers for ETL.

  • Equivalents:

    • GCP: Cloud Dataflow (closest for serverless processing) or Dataproc Serverless.

    • Azure: Azure Data Factory (Data Flows) or Azure Databricks.

AWS Glue Data Catalog

  • What it is: The central metadata repository. It doesn't store your actual data (that's in S3). It stores the definitions of your data: "Where is the file? What are the columns? Is it a CSV or Parquet?"

  • Why it matters: It is the "brain" that allows Athena and Redshift Spectrum to query S3. Without the Catalog, S3 is just a dumb hard drive.

  • Equivalents:

    • GCP: Dataplex / BigQuery Metastore.

    • Azure: Microsoft Purview (formerly Azure Data Catalog).

AWS Glue Crawlers

  • What it is: A bot that scans your S3 buckets, guesses the schema (columns/types), and automatically updates the Glue Data Catalog.

  • Use Case: "I just dumped 1,000 new CSV files. I don't want to type CREATE TABLE manually. Crawler, go figure it out."

AWS Lake Formation

  • What it is: A security layer on top of Glue.

  • The Problem: S3 permissions are hard (you allow access to whole files/folders).

  • The Solution: Lake Formation lets you say, "User Alice can query the Sales table, but she cannot see the 'Credit Card' column." It handles fine-grained access control (Row/Column level security).

  • Equivalents:

    • GCP: Dataplex (Policy management).

A diagram that explains the place AWS Glue and other tools take.

spinner

1. The Hot Path (Real-Time Ingestion)

  • Sources (IoT & Web Apps): Data is generated here. A sensor beeps or a user clicks a button. This data is pushed instantly to Kinesis Data Streams (KDS).

  • Kinesis Data Streams: This is the high-speed "highway." It catches millions of events per second.

  • Managed Flink: (Optional) It sits on the highway. It grabs data, does quick math (e.g., "Alert if temperature > 100"), and puts it back.

  • Kinesis Firehose: This is the "Off-Ramp." It takes the streaming data off the highway, buffers it (waits for 5MB or 5 minutes), compresses it, and saves it as a file in S3.

2. The Storage Layer (S3)

  • S3 Raw Zone: The "Landing Zone." This is where Firehose dumps the raw files. They might be messy JSON or CSVs. We never delete these; they are the "Source of Truth."

  • S3 Processed Zone: The "Clean Zone." This is where the optimized, clean data (Parquet files) will live after we fix it.

3. The Batch & Metadata Layer (Glue)

  • Glue Crawler: It’s a robot that wakes up periodically. It looks at the S3 Raw Zone, notices new files, and figures out the columns/types. It writes this "Schema" into the Glue Catalog.

  • Glue Catalog: The "Phonebook." It doesn't store data; it stores the definitions (e.g., "Table X is in Folder Y").

  • Glue ETL Job: The "Workhorse." It reads the messy data from S3 Raw, cleans it (removing duplicates, fixing dates), and writes the clean version to S3 Processed.

4. The Consumption Layer (Security & Analytics)

  • Lake Formation: The "Bouncer." It sits in front of your data. It decides who gets to see what.

  • Athena: The "Analyst." You type a SQL query (SELECT * FROM sales). Athena asks Lake Formation for permission. If allowed, Athena looks up the file location in the Glue Catalog, scans the files in S3 Processed, and gives you the answer.

PreviousAWS FargateNextKinesis and streaming stack

Last updated