RabbitMQ
Most of the information you see here is from this training.
What Is RabbitMQ?
About RabbitMQ, Tutorials, Networking on RabbitMQ, AMQP 0-9-1 Model Explained
RabbitMQ is a popular open-source message broker that implements the AMQP protocol. It's great because it's open-source, flexible, powerful, and reasonably easy to use.
Book about RabbitMQ and CloudAMQP
Free training on CloudAMPQ
Optimization and tuning, section 7
RabbitMQ Architecture and Components
Let's explore the key components of RabbitMQ's architecture and understand how they work together.
What is a Message Broker?
A message broker acts as a middleman between services that need to communicate asynchronously. Instead of services talking directly to each other, they send and receive messages through the broker. This decouples your applications and makes your system more flexible and scalable.
Core Components
Producer and Consumer
At the heart of message-based communication are two key players:
Producer: The application that sends messages to the broker. Think of it as the sender in a postal system.
Consumer: The application that receives and processes messages from the broker. This is your message recipient.
Messages can contain various types of information depending on your application's needs. Once sent, messages are collected in sequential order and stored in queues until a consumer is ready to process them.
The AMQP Protocol
RabbitMQ uses AMQP (Advanced Message Queuing Protocol), which defines how messages are transmitted between applications and the broker. Understanding two key concepts is essential:
Connection: A TCP connection between your application and the RabbitMQ broker. This is the actual network link.
Channel: A virtual connection inside a physical connection. When you publish or consume messages, you do it over a channel. A single connection can have multiple channels, making it efficient to handle many operations without creating numerous TCP connections.
Routing Messages: Exchanges and Bindings
Here's where RabbitMQ gets interesting. Messages aren't sent directly to queues. Instead, they go through an exchange first.
Exchange
An exchange receives messages from producers and routes them to the appropriate queues based on predefined rules. Think of an exchange as a mail sorting facility that ensures each message reaches its correct destination. Every queue must be bound to at least one exchange to receive messages.
The routing decision depends on the exchange type and the routing key.
Routing Key
The routing key is like an address label on your message. It's a string that the exchange examines to determine which queue(s) should receive the message. For example, you might use routing keys like "order.created" or "payment.processed" to categorize different message types.
Binding
A binding is the link that connects a queue to an exchange. It defines the relationship and rules for how messages should flow from the exchange to the queue. You can think of bindings as the routing rules in your messaging system.
Message Structure and Acknowledgments
What Messages Look Like
Every message in RabbitMQ consists of two parts:
Payload: The actual content or data you're sending
Headers: Metadata properties that define message characteristics such as durability (whether the message survives a broker restart) and priority
Acknowledgments (ACK)
AMQP uses acknowledgments to ensure reliable message delivery. When a consumer successfully receives and processes a message, it sends an ACK command back to the broker. This confirms that responsibility for the message has been transferred and the broker can safely remove it from the queue.
Virtual Hosts (VHOST)
A virtual host provides logical separation within a single RabbitMQ instance. This allows you to:
Run multiple isolated environments on the same broker
Set different user permissions for different virtual hosts
Create queues and exchanges that exist only within a specific virtual host
Think of VHOSTs as separate apartments in the same building—each tenant has their own space, but they share the same infrastructure.
The Complete Message Flow
Putting it all together, here's how a message travels through RabbitMQ:
A Producer creates a message and sends it through a Connection and Channel
The message arrives at an Exchange with a Routing Key
The Exchange examines the routing key and Bindings to determine the destination
The message is routed to one or more Queues
Messages wait in the queue in sequential order
A Consumer retrieves messages from the queue through its own channel
The consumer processes each message in order
Upon successful processing, the consumer sends an ACK to confirm receipt
This architecture provides a robust, flexible messaging system that can handle complex routing scenarios while maintaining reliability and scalability.
AMQP Concepts
AMQP protocol
Advanced Message Queuing Protocol is the core protocol used by RabbitMQ for messaging.
RabbitMQ implements an extension of the open-standard AMQP 0.9.1 specification and is used as the basis for a set of standards controlling the entire message passing process.
AMQP is a binary protocol suitable for large amounts of data. It is optimized for machine efficiency and speed rather than human readability.
Other valuable characteristics about AMQP protocol is that it's a multiple implementation standard, meaning that you can run AMQP with various brokers and other applications.
AMQP defines both the network layer protocol and a high-level architecture for message brokers.
It defines a set of message capabilities which must be available for an AMQP compliant server implementation. This includes the rules of how messages must be routed and stored within the broker to follow the AMQ model.
AMQP is an application layer protocol that focuses on process-to-process communication across IP networks. An encoding schema and a set of procedures allow for two different servers to communicate regardless of the technology used. Overall, the goal of AMQP is to enable message passing through broker services over TCP/IP connections.
AMQP is considered a compact protocol since it's binary, meaning that everything sent over AMQP is binary data.
Connections and Channels
In a conversation, parties greet each other, exchange verbal banter, and eventually continue on their way. A similar form of communication occurs over connections exposing lightweight channels in RabbitMQ.
A connection is a link between the Client and the Broker, that performs underlying networking tasks. These tasks include initial authentication, IP resolution, and networking.
RabbitMQ supports both IPv4 and IPv6 connections. It is also possible to encrypt connections using TLS. A connection can multiplex into several "light-weight connections".
This "lightweight connection" is called a Channel. Each Connection can maintain a set of underlying Channels. Services usually want to have multiple connections to the broker, but instead of having many heavy-weight connections, a service can reuse the resources by creating and deleting Channels. A Channel reuses a connection, forgoing the need to reauthorize and open a new TCP stream.
Channels simply allow you to use resources more efficiently. Every AMQP protocol-related operation occurs over a Channel. A Connection is created by opening a physical TCP connection to the target server.
The client resolves the hostname to one or more IP addresses before sending a handshake. The receiving server then authenticates the client.
To send a message or manage queues, a Channel needs to be established. The channel packages the messages and handles protocol operations.
Clients send messages through the channel's "basic_publish" method. Queue creation and maintenance are also sent through a channel. AMQP commands such as "queue.create" and "exchange.create" are all sent over a channel.
Closing a connection closes all associated channels. A Channel can be opened right after successfully opening a connection.
Queues
The queue is the place where messages are stored until they are consumed by the consumer, or in other ways removed from the queue. Queues have properties that define how they behave, and these properties are passed to the broker when the queue is declared.
A queue has some required properties and some optional. A queue always has a name, so that services can reference them. A queue declared with no name, is given a random name by most client libraries.
A queue can be marked as durable, which specifies if the queue should survive a broker restart.
A queue can be exclusive, which specifies if the queue can be used by only one connection. An exclusive queue is deleted when that connection closes.
A queue can also be declared with the auto-delete property, meaning that a queue that has had at least one consumer is deleted when the last consumer unsubscribes.
There are also some optional properties used by plugins and broker-specific features, like TTL, which is telling an unused queue when to expire after a period of time.
Before a queue can be used it has to be declared. Declaring a queue will cause it to be created if it does not already exist.
Let’s follow the life-cycle for a temporary message queue.
The client creates the message queue (Declare). The server confirms (Declare-Ok).
The client starts a consumer on the message queue.
The client cancels the consumer, either explicitly or by closing the channel and/or connection.
When the last consumer disappears from the message queue, the server deletes the message queue.
Exchanges
More explanations about the types of exchanges here.
Theoretically, the exchange is the first entry point for a message entering the message broker. Messages are not published directly to a queue. Instead, the producer sends messages to an exchange, which you can think of as a message routing agent. An exchange is responsible for routing messages to different queues with the help of header attributes, bindings, and routing keys.
Exchanges can, as with queues, be configured with parameters such as durable, temporary, and auto-delete upon creation. Durable exchanges survive server restarts and exist until they are explicitly deleted. Temporary exchanges exist until RabbitMQ is shut down. Auto-deleted exchanges are removed once the last bound object is unbound from the exchange.
Let’s follow the life-cycle for an exchange.
The client asks the server to make sure the exchange exists (Declare). The exchange is created if it does not already exist.
The client publishes messages to the exchange.
The client may choose to delete the exchange (Delete).
In RabbitMQ, there are four different types of exchanges that route the message differently using different parameters and bindings setups. Clients can create their own exchanges or use the predefined default exchanges which are created when the server starts for the first time.
How a message is routed depends on several things, including Exchange type which specifies the number of routing rules, routing key, and header attributes.
The 4 mentioned attributes act as addresses for messages. From a Queue's perspective, you can check which exchanges and routing rules are linked to your specific Queue. These links are called Bindings. A binding links a Queue to an Exchange while the routing key is like the address for the message. This is mainly what the Exchange looks for when deciding how to route the message to Queues.
In RabbitMQ, there are 4 main types of Exchanges:
direct
topic
fanout
headers
Existing Exchanges can be seen in the Management UI or through rabbitmqadmin admin.
Examples of using various exchange types
Let's look at an example from a taxi company to explain the different exchange types.
Every request for a taxi is made through an app that communicates with an application services that uses RabbitMQ.
Direct Exchange
Direct Exchange directs the message to a specific Queue by looking at the routing key. The routing key in the message is compared for equality with the routing keys on Bindings.
In our example, the direct is used when a user requests a specific taxi, for example choosing their favorite driver.
Topic exchange
Topic exchange routes messages to one or many queues by looking at the routing key. The routing key in the message is compared for matches with routing key-patterns on the bindings.
In our example a customer with a group of friends ask for a large environmentally-friendly taxi. This order is routed through an exchange bound to taxis of this type.
The routing key must be a list of words delimited by a period (e.g. routing_key: taxi.eco.large, taxi.eco.small). The topic exchange supports "strict" routing key-matching, like a direct exchange, but will also perform "wildcard" matching using star (*) and hash (#) as placeholders.
In another example a customer orders a large taxi but does not care about the taxi type (e.g. routing_key: taxi.any.large, in which case all taxi.*.large will match). The topic exchange routes this message to all taxis bound as a large taxi.
Fanout exchange
Fanout exchange copies and routes a received message to all queues that are bound to it regardless of routing keys. A provided routing key is simply ignored.
In our example the Fanout Exchange is used when the taxi coordinators inform all taxi drivers about a blocked road.
Headers Exchange
Headers exchanges are very similar to topic exchanges but route messages based on the header values instead of routing keys.
Headers exchanges are not very common, but in our example it is used by a reporting service at the taxi company. Taxi car status data is sent to the exchange every now and then and this data is used to build reports by other parts of the system.
A special argument named "x-match" added in the binding between exchange and queue, specifies if headers must match "all" or "any".
In this example one service writes a report for all taxis around Manhattan, New York, keeping track of fuel consumption and miles traveled. This data is included in the message. When "x-match" is set to "any", together with the following arguments, the New York report receives all messages from taxis with trips from, to, or within New York. A trip that starts in New York and ends in Jersey will therefore be included.
Another report may be interested in trips within New York alone. This can be achieved by simply setting the "x-match" to "all". The new report will then only get messages where "from" and "to" are both set to New York - trips that never leaves the city.
Bindings
A binding is an association, or relation between a queue and an exchange. It describes which queue is interested in messages from a given exchange. Bindings can take an extra parameter called routing_key. If you remember, a routing key can also be sent with a message. The routing key on the binding sometimes called a binding key, and the routing key in the message are the things the exchange is looking at while delivering messages.
Consumer Acknowledgements and Publisher Confirm
Messages in transit might get lost in the event of a connection failure and need to be retransmitted. Acknowledgments let the server and clients know when to retransmit messages.
Consumer Acknowledgements
The client can either ack the message when it receives it, or when the client has completely processed the message.
Delivery Confirmation Timing
A message can be considered successfully delivered at one of three points:
Immediately once it is sent out
Once it's written to a TCP socket
When an explicit acknowledgment is received from the client
The manually sent acknowledgment can be positive or negative.
Publisher Confirm
Publish confirm is the same concept but for publishing. The server confirms when it has received a message from a publisher.
Message Handling After Acknowledgement
Successful Processing
Once an acknowledgment is received, the message can be discarded from the queue.
Failed Processing
If the consumer cannot process a message, the desired outcome could be:
Requeue it and let another consumer receive and handle it
Retry processing at a later time
Such deliveries can be discarded by the broker or requeued.
Requeue Behavior
This behavior is controlled by the requeue field:
When the field is set to true, the broker will requeue the delivery
A requeued message is placed back into its original position in its queue, if possible
The message can therefore immediately be ready for redelivery
Requeue Loop Prevention
If all consumers requeue because they cannot process a delivery, they create a requeue/redelivery loop.
Solution: It is possible to track the number of redeliveries and:
Reject messages for good (discard them), or
Schedule a requeue after a delay
Virtual Hosts
Virtual hosts (or vhosts) in RabbitMQ provide a way to segregate applications using the same RabbitMQ instance. RabbitMQ vhosts create a logical group of connections, exchanges, queues, bindings, user permissions, etc. within an instance.
Conceptual Model
Think of vhosts as individual, uniquely named containers. Inside each vhost container is a logical group of:
Exchanges
Connections
Queues
Bindings
User permissions
Other system resources
How Virtual Hosts Work
Resource Segregation
Different users can have different permissions to different vhosts
Queues and exchanges can be created so they only exist in one vhost
Resources such as exchanges and queues are named resources inside the vhost container, making each vhost essentially a RabbitMQ mini-server
Connection Specification
When a client establishes a connection to the RabbitMQ server, it specifies the vhost within which it will operate.
Default Configuration
When configuring RabbitMQ, at least one vhost is needed, which in default is just a slash "/".
Creating Virtual Hosts
Virtual hosts can be created through:
The management portal
The HTTP API
Via rabbitmqctl
To create via management portal:
View vhosts by entering the admin tab and select the Virtual Hosts
Select the Add New Virtual Host option to create a new vhost
Permissions and Users
The permissions within the vhost and the users assigned to it depend on your system requirements
It's up to you to assign users to the vhost
A newly created vhost always has a default set of exchanges, but no other entities and no user permissions
Isolation Characteristics
What is NOT Shared
Virtual hosts do not share exchanges or queues between them, and users, policies, etc. are unique to each vhost.
Virtual vs. Physical Separation
Essentially, RabbitMQ vhosts are like a virtual machine for a physical server, allowing for multiple secure application operations through virtual rather than physical separation.
Important: As the separation is virtual, it is important to remember that the vhosts are not physically separated from each other and therefore they might affect each other's performance.
Use Cases
Multi-Application Environment
The same broker can be used on parts of different applications. You can separate environments (e.g., production to one vhost and staging to another vhost) within the same broker instead of setting up multiple brokers.
Service Isolation
If one service is experiencing a traffic spike or a code bug, this may cause problems for other services and affect their performance. A vhost can be created for each service to hold all of the logical infrastructure, which also allows for better individual topology management.
Interacting with RabbitMQ
https://www.rabbitmq.com/docs/use-rabbitmq
There are three main components you'll use to interact with RabbitMQ:
The RabbitMQ server: The message broker itself. It's a server you can run locally or on a production server to send and receive messages. Installation guides
Client libraries and developer tools
Python AMQP library - Most commonly used:
pika (official, most popular):
pip install pikaaio-pika (async version):
pip install aio-pikakombu (used by Celery):
pip install kombu
For Go programming, you'll use the AMQP library to interact with the RabbitMQ server.
Management UI: RabbitMQ comes with a web-based management UI that you can access in your browser to monitor and manage your RabbitMQ server.
Management interface
Important information of the RabbitMQ state you can monitor in the Management UI.
The Overview tab provides a quick snapshot of RabbitMQ's current state through charts and metrics.
Main Charts
1. Queued Messages Chart
Shows total queued messages across all queues
Ready: Messages available for delivery
Unacked: Messages awaiting server acknowledgment
2. Message Rates Chart
Shows message processing speed
Publish: Rate of messages entering the server
Confirm: Rate of server confirmations
Chart Controls: Time intervals can be adjusted by clicking "chart: last minute" above the charts. Press the question mark (?) for message status information.
Global Counts
Displays total numbers across ALL accessible virtual hosts:
Connections
Channels
Exchanges
Queues
Consumers
Nodes Tab
Provides information about RabbitMQ cluster nodes (servers), including:
Server memory usage
Number of Erlang processes per node
Node-specific information
Enabled plugins (via Info section)
Churn Rate Monitoring
Connection/channel opening/closure rates are important metrics to monitor. High connection and channel churn might lead to node resource exhaustion.
Additional Features
Ports and Contexts
Displays listening ports for different protocols.
Export and Import Definitions
Download a JSON file representing your broker configuration
Contains RabbitMQ settings for exchanges, queues, virtual hosts, policies, and users
Can be used for backup and restoration
Allows saving old settings before making configuration changes
Connection States
RabbitMQ connections and channels can exist in nine different states:
Starting
Tuning
Opening
Running
Flow
Blocking
Blocked
Closing
Closed
Flow State (Important)
The "flow" state indicates that the publishing rate has been restricted to prevent RabbitMQ from running out of memory.
Triggered automatically when a connection publishes too quickly for a queue to keep up
A flow-controlled connection will block and unblock several times per second to maintain a manageable rate
Connections Tab
Overview Information
Displays all established connections to the RabbitMQ server with the following details:
vhost: Which virtual host the connection operates in
Username: User associated with the connection
Channels: Number of channels using the connection
SSL/TLS: Whether the connection is secured with SSL
Detailed Connection View
Clicking on a specific connection provides:
Overview of that connection
Channels within the connection
Data rates
Client properties
Option to close the connection
Additional Reference: More information about connection attributes can be found in the rabbitmqctl manual page (the command-line tool for managing RabbitMQ).
Channels Tab
Overview Information
Shows information about all current channels:
vhost: Which virtual host the channel operates in
Username: User associated with the channel
Mode: Channel guarantee mode
Detailed Channel View
Clicking on a specific channel provides:
Detailed overview of that channel
Message rate information
Number of logical consumers retrieving messages via the channel
Exchanges Tab
The Exchanges tab lists all exchanges with the following details:
Virtual host: The vhost where the exchange resides
Type: The exchange type (direct, topic, headers, fanout)
Features: Exchange parameters displayed as abbreviations
D: Durable
AD: Auto-delete
Note: Features and types are specified when the exchange is created.
Default Exchanges
The list includes exchanges created by default:
amq.* exchanges
The default (unnamed) exchange
Detailed Exchange View
Clicking on an exchange name shows a detailed page where you can:
View existing bindings to the exchange
Add new bindings to the exchange
Publish a message to the exchange
Delete the exchange
Queues Tab
The Queues tab displays queues for all virtual hosts or one selected vhost.
Queue Parameters and Features
Queues have different parameters and arguments depending on how they were created. The features column shows parameters such as:
Durable queue (D): Ensures RabbitMQ will never lose the queue
Message TTL: How long a message published to a queue can live before it is discarded
Auto expire: How long a queue can be unused before it is automatically deleted
Max length: How many (ready) messages a queue can contain before it starts to drop them
Max length bytes: The total body size for ready messages a queue can contain before it starts to drop them
Queue Creation
You can create a new queue directly from this view.
Detailed Queue View
Clicking on a specific queue displays comprehensive information:
Charts
The first two charts show:
Number of queued messages (specific to that queue)
Message rates (specific to that queue)
These mirror the overview charts but focus on the individual queue.
Consumers
Shows the consumers/channels that are connected to the queue.
Running RabbitMQ
RabbitMQ Streams
RabbitMQ streams is a new feature available from RabbitMQ version 3.9, and is a completely new part of RabbitMQ, making RabbitMQ an option for a lot of new use cases.
It's a new queue type optimized for high-throughput, append-only logs
Similar to Kafka topics - stores messages persistently in an ordered log
Allows multiple consumers to read from any point in the stream
It's just storage/transport - RabbitMQ Streams itself doesn't "process" data
Benefits of Using RabbitMQ Streams
Large number of consumers can easily consume the same message.
Messages can be consumed multiple times.
Messages will stay in queues until they are expired with retention policies.
High throughput when using the stream protocol.
Streams can easily store millions of messages without issues (which is not always the case with traditional queue-type messages in RabbitMQ). Alternative to Apache Kafka.
Monitoring RabbitMQ
https://www.rabbitmq.com/docs/monitoring
Managing RabbitMQ
https://www.rabbitmq.com/docs/manage-rabbitmq
Last updated