Lately, Code Dạo has been working on providing some technical articles for everyone to enjoy. Today, let’s dive into the topic of Message Queue. This component plays an incredibly crucial role and is indispensable in large systems (I bet Facebook, Google, and LinkedIn all have it in their systems), especially in microservices architecture.
However, if you haven’t been involved in large or specialized projects, you might not have heard of this concept. So, what is Message Queue exactly and why is it so widely used? Stick around, and by the end of this article, you’ll have all the answers!
What Exactly is Message Queue?
In simple terms, Message Queue is just like a queue, which contains multiple messages.
Just kidding! You can think of a message queue as a mailbox that allows different components/services within a system (or even multiple systems) to exchange information with each other.
It’s called a queue because it follows the FIFO (First In First Out) mechanism, meaning the messages that are inserted first will be retrieved first.
A system that utilizes Message Queue typically consists of the following components:
- Message: The information being sent (which can be in the form of text, binary, or JSON).
- Message Queue: The place where these messages are stored, allowing producers and consumers to communicate with each other.
- Producer: A program or service that generates information and puts it into the message queue.
- Consumer: A program or service that receives messages from the message queue and processes them.
- A program/service can be both a producer and a consumer.
Real-world Usage of Message Queue
In systems that adopt microservices architecture, we use message queues to enable services to communicate with each other in an asynchronous manner. Service A can send a message to the message queue for service B to process without having to wait for service B to finish its task.
Let’s say I have a website that allows users to download videos from YouTube (definitely not other websites). In this case, we will have the following components:
- Web service: This acts as a producer. It receives the YouTube URLs from users and puts this information into the message queue.
- Processing Service: This acts as both a consumer and a producer. It reads the YouTube URLs from the message queue, starts downloading and encoding the files, and saves them on the server. After the encoding is done, it puts the URLs of the encoded files back into the message queue.
- Uploading Service: Upon receiving messages from the processing server, this service uploads the videos to Google Drive and so on.
In practice, message queues can solve many challenging problems in systems, such as:
- Ensuring Duration/Recovery: Since messages are stored in the queue, even if a service crashes or encounters an error while processing, we don’t have to worry about data loss. We can simply fetch the message from the queue and reprocess it. In a system with multiple consumers, if one or two consumers crash, it won’t bring down the entire system.
- System Decomposition: Message queues help break down complex systems into smaller services, with each service handling a specific function. (For more advantages and disadvantages, please refer to microservices articles.)
- Supporting Rate Limiting and Batching: In many cases, system processing capacity is limited (e.g., handling only 300 orders per second). With message queues, we can gradually retrieve orders from the queue for processing, without fear of falling behind. Instead of sending each email one by one, we can wait for the message queue to accumulate 200 email requests before sending them all at once.
- Easy System Scaling: During peak hours with a high influx of queries, we can increase the number of consumers to handle more messages. When the system is no longer under heavy load, we can simply reduce the number of consumers.
Some Important Considerations
Of course, no technology is a silver bullet. When using any technology, there are certain things to keep in mind:
- Difficult to Handle Synchronization: Not every system requires a message queue. If Service A calls Service B and requires immediate processing results (synchronous), it’s better to use REST or gRPC instead.
- Increasing System Complexity: Adding a message queue will inevitably add complexity to the system. We need to have a clear understanding of which messages go into which queues and who sends and receives them. Debugging locally can also become more challenging.
- Message Format Agreement: To send and receive messages, both producers and consumers need to adhere to a unified message format. Carelessness in this aspect can lead to one side being unable to read the data if the other side makes any changes.
- Monitoring the Queue: It’s essential to have monitoring mechanisms in place to ensure that the message queue doesn’t become overloaded and fill up. The best scenario is to have an empty queue or maintain a consistent number of messages (meaning that all messages sent to the queue are consumed).
As for some popular message queues being used today, they include:
- RabbitMQ
- Kafka (which can do many cool things beyond message queueing)
- Amazon SQS
- MSMQ (Microsoft Message Queuing)
- RocketMQ
- ZeroMQ
In Conclusion
In this article, I’ve shared with you the concept of message queues, which are essential components in large systems, especially those utilizing microservices architecture.
When you reach a senior or software architect level, you will undoubtedly come across this topic during your work or interviews! If you have any experiences to share, feel free to leave them in the comments section!
References:
