Orchestrators
AWS Step Functions
What it is: A serverless state machine. You draw a flowchart: "Start -> Run Lambda -> If Success, Run Glue -> If Fail, Send Email."
Pros: Built into AWS, super cheap, event-driven.
Cons: Proprietary to AWS (you can't take the code to GCP).
Equivalents:
GCP: Cloud Workflows.
Azure: Logic Apps.
Amazon MWAA (Managed Workflows for Apache Airflow)
What it is: Managed Airflow.
Context: Airflow is the #1 tool for data orchestration globally. MWAA is just AWS hosting it for you so you don't have to install it on an EC2 server yourself.
Equivalents:
GCP: Cloud Composer (This is the best managed Airflow service in the market).
Azure: Azure Data Factory Managed Airflow.
Last updated