SQS, SNS

---

Amazon SQS (Simple Queue Service)

• What it is: A message queue. It holds messages in a line until a server is ready to process them. It prevents your system from crashing if too many requests come in at once.

• Equivalents:

  • GCP: Pub/Sub (GCP combines "Queues" and "Streaming" into one product)

  • Azure: Azure Queue Storage

The SQS "Decoupling" Architecture

The Scenario: Imagine it's Black Friday. Your e-commerce website is getting thousands of orders per second.

• The Problem: The "Order Processing" backend (checking inventory, charging credit cards, generating shipping labels) is slow. It takes 2 seconds per order.

• Without SQS: If the website tries to connect directly to the slow backend, the backend will crash under the load, and the website will stop accepting orders.

• With SQS: The website drops the order in the queue and immediately tells the user "Success!" The backend works through the pile at its own pace.

[ USER ]
    |
    | (1) Clicks "Buy"
    v
[ WEB APP ] <-----------------------+
    |                               |
    | (2) Send Message (Instant)    | (User gets "Success" response)
    v                               |
[ SQS QUEUE ]                       |
    |  [Msg 1] [Msg 2] [Msg 3]      |
    |                               |
    | (3) Worker asks: "Any jobs?"  | (5) Worker says: "Job Done, Delete!"
    v                               ^
[ WORKER SERVER ] ------------------+
    |
    | (4) Processes Order (Slow)
    v
[ DATABASE ]

SNS vs. SQS (The Confusion)

Beginners often mix these up.

• SQS (Queue): One-to-One. You put a letter in the mailbox. One person picks it up and processes it.

• SNS (Topic): One-to-Many. You shout into a microphone. Everyone in the room hears it.

Data Engineer Pro Tip: You almost never use SQS alone for data pipelines. You almost always use SNS + SQS.

• Why? Because if your processing server crashes while SNS is "shouting" at it, that message is lost forever.

• The Fix: You make SNS shout into an SQS Queue. The Queue holds the message safely until the server comes back online.

---

While SNS is the "Shouting Megaphone" (Push), SQS is the "Mailbox" or "Buffer" (Pull).

Amazon SNS (Simple Notification Service)

• What it is: The "Megaphone" or "Broadcaster".

• The Concept: It implements the Pub/Sub (Publish/Subscribe) pattern.

  • Publisher: Sends a message once to a "Topic."

  • SNS Topic: Instantly pushes copies of that message to everyone who is subscribed.

• Key Characteristic: It is Push-based. Unlike SQS (where the worker has to ask "Do you have work?"), SNS forces the message onto the subscriber immediately.

• Use Cases:

  • System Alerts: "CloudWatch says CPU is 99% -> SNS Topic -> Email to Admin."

  • Mobile Push: Sending notifications to iOS/Android apps.

  • Fan-Out Architecture: (See below - this is the #1 Data Engineering pattern).

The "Fan-Out" Pattern (Crucial for Data Engineers)

This is the specific architecture where SNS shines.

The Scenario: Imagine a user uploads a photo to your app. You need to do three things:

1. Generate a thumbnail (Resize).

2. Run image recognition (ML).

3. Update the user's profile database.

The "Bad" Way: Your app tries to do all three one by one. If step 2 fails, the whole thing crashes.

The "Fan-Out" Way:

1. Your app sends one message to an SNS Topic: "New Photo Uploaded: ID 123".

2. SNS "Fans Out" this message to three separate SQS queues.

3. Three separate workers process the queues independently.

Here is the diagram of this pattern:

---