SQS, SNS


Amazon SQS (Simple Queue Service)

  • What it is: A message queue. It holds messages in a line until a server is ready to process them. It prevents your system from crashing if too many requests come in at once.

  • Equivalents:

    • GCP: Pub/Sub (GCP combines "Queues" and "Streaming" into one product)

    • Azure: Azure Queue Storage

The SQS "Decoupling" Architecture

The Scenario: Imagine it's Black Friday. Your e-commerce website is getting thousands of orders per second.

  • The Problem: The "Order Processing" backend (checking inventory, charging credit cards, generating shipping labels) is slow. It takes 2 seconds per order.

  • Without SQS: If the website tries to connect directly to the slow backend, the backend will crash under the load, and the website will stop accepting orders.

  • With SQS: The website drops the order in the queue and immediately tells the user "Success!" The backend works through the pile at its own pace.

[ USER ]
    |
    | (1) Clicks "Buy"
    v
[ WEB APP ] <-----------------------+
    |                               |
    | (2) Send Message (Instant)    | (User gets "Success" response)
    v                               |
[ SQS QUEUE ]                       |
    |  [Msg 1] [Msg 2] [Msg 3]      |
    |                               |
    | (3) Worker asks: "Any jobs?"  | (5) Worker says: "Job Done, Delete!"
    v                               ^
[ WORKER SERVER ] ------------------+
    |
    | (4) Processes Order (Slow)
    v
[ DATABASE ]

SNS vs. SQS (The Confusion)

Beginners often mix these up.

  • SQS (Queue): One-to-One. You put a letter in the mailbox. One person picks it up and processes it.

  • SNS (Topic): One-to-Many. You shout into a microphone. Everyone in the room hears it.

Data Engineer Pro Tip: You almost never use SQS alone for data pipelines. You almost always use SNS + SQS.

  • Why? Because if your processing server crashes while SNS is "shouting" at it, that message is lost forever.

  • The Fix: You make SNS shout into an SQS Queue. The Queue holds the message safely until the server comes back online.


While SNS is the "Shouting Megaphone" (Push), SQS is the "Mailbox" or "Buffer" (Pull).

Amazon SNS (Simple Notification Service)

  • What it is: The "Megaphone" or "Broadcaster".

  • The Concept: It implements the Pub/Sub (Publish/Subscribe) pattern.

    • Publisher: Sends a message once to a "Topic."

    • SNS Topic: Instantly pushes copies of that message to everyone who is subscribed.

  • Key Characteristic: It is Push-based. Unlike SQS (where the worker has to ask "Do you have work?"), SNS forces the message onto the subscriber immediately.

  • Use Cases:

    • System Alerts: "CloudWatch says CPU is 99% -> SNS Topic -> Email to Admin."

    • Mobile Push: Sending notifications to iOS/Android apps.

    • Fan-Out Architecture: (See below - this is the #1 Data Engineering pattern).

The "Fan-Out" Pattern (Crucial for Data Engineers)

This is the specific architecture where SNS shines.

The Scenario: Imagine a user uploads a photo to your app. You need to do three things:

  1. Generate a thumbnail (Resize).

  2. Run image recognition (ML).

  3. Update the user's profile database.

The "Bad" Way: Your app tries to do all three one by one. If step 2 fails, the whole thing crashes.

The "Fan-Out" Way:

  1. Your app sends one message to an SNS Topic: "New Photo Uploaded: ID 123".

  2. SNS "Fans Out" this message to three separate SQS queues.

  3. Three separate workers process the queues independently.

Here is the diagram of this pattern:

spinner

Last updated