RocketMQ Series 1: Introduction to the Basics

★Message Queue 16 articles

1 Getting to Know RocketMQ

RocketMQ is a Java-based distributed messaging middleware , it is widely acclaimed for its high performance , high reliability , high real-time and distributed features .

It supports transactional messages, sequential messages, batch messages, timed messages, message backtracking, and so on. Internet scenarios often use RocketMQ for message routing, subscription publishing, asynchronous decoupling, traffic reduction peaks and other operations to ease the pressure on the system.

2 Comparison of three common messaging middleware

characterization	RabbitMQ	RocketMQ	Kafka
development language	erlang	java	scala
Stand-alone throughput	1w+	10w+	10w+
timeliness	us level	ms grade	Within ms
usability	High (master-slave architecture)	Very high (distributed architecture)	Very high (distributed architecture)
Message Reliability	Basically, no loss.	Parameterized configuration and persistence: basically no loss of	Parameterized configuration and persistence: basically no loss of
Functional characteristics	Based on erlang development, so the concurrency is very strong, the performance is extremely good, the latency is very low; management interface is richer	MQ is more complete, good scalability	Only supports the main MQ features , like some message query , message back and other features are not provided , after all , is prepared for big data , in the field of big data applications wide .
ecologically	Open source, stable, active community	Ali open source, handed over to Apache, low community activity	Apache development, open source, high throughput, active community

Technology Selection Decision Reference:
(1) small and medium-sized software, it is recommended to choose RabbitMQ, small and medium-sized software data volume is not so large, the selection of messaging middleware, should be preferred to more complete functionality, so kafka excluded.
The reason for not considering rocketmq is that rocketmq is an Ali product. If Ali gives up maintaining rocketmq, small and medium-sized companies usually can't spare people to do customized development of rocketmq, so it's not recommended.
(2) large software companies, according to the specific use of rocketMq and kafka between the two choose one. On the one hand, large software companies, with enough money to build a distributed environment, but also with a large enough data volume.
For rocketMQ, large software companies can also spare manpower to rocketMQ custom development, after all, have the ability to change the JAVA source code, or quite a lot of people.
As for kafka, according to business scenarios, if there is a log collection function, it is certainly preferred kafka. Which one to choose depends on the usage scenario. After the introduction of MQ, it will inevitably lead to a reduction in system availability and an increase in complexity.

3 Message Middleware Usage Scenarios

1. Decoupling: For example, system A will be given to the system B to deal with some things, but A does not want to be directly associated with B, to avoid coupling is too strong, you can join the message queue in the middle of the A, B, A will be tasked with things to the message queue, B subscribe to the message queue to perform the task.

This scenario is very common, for example, if A is an order system and B is an inventory system, you can give the job of reducing inventory to system B to handle via message queue. If system A wants to have B, C, D... This advantage is even more obvious when multiple systems are dealing with the problem.

2. Orderliness: First-in-first-out principle, first-come-first-served, for example, a system takes a long time to process something, but when dealing with this thing, there are other people also issued a request, you can put the request in the message queue, one by one to deal with.

Businesses with strong demand for data sequentiality and consistency, such as the same bank card being used by multiple entrances at the same time, need to ensure the sequentiality of incoming and outgoing accounts to avoid data inconsistency.

3. Message routing/data distribution: Send messages from a queue to different other queues according to different rules

Through the message queue will be different color request sent to different services to operate. This achieves the purpose of traffic splitting by service.

4、Asynchronous processing: When dealing with a task, there are 3 steps A, B and C. You need to complete operation A first, and then do operations B and C. The success of the task depends strongly on the result of A, but not on the result of B and C. The success of the task depends strongly on the result of A, but not on the results of B and C.
If we use serial execution, then the cycle time of processing tasks will become longer, and the overall throughput capacity of the system will be reduced (in the same system to do asynchronous is actually a relatively large overhead), so the use of message queues is a better approach.

The login operation is a typical scenario: A: execute login and get the result, B: record the login log, C: write the user information and Token into the cache. After the execution of A can jump from the login page to the home page, B, C let the service slowly to digest, do not block the current operation.

5、Peak Cutting: Cut down the operation during the peak period, for example, the whole operation process of student A contains 12 steps, and the subsequent 11 steps are the data that do not need to pay strong attention to the result, which can be put in the message queue.

4 Concepts, Architecture and Principles of MQ

4.1 Description of the composition

RocketMQ has four main core components: NameServer, Broker, Producer, and Consumer. These roles usually exist in clusters. RocketMQ is based on pure Java.It is characterized by high throughput, high availability and suitable for large-scale distributed system applications.

To start RockerMQ, you first need to start the NameServer, and then start the Brober hosts
The Broker registers the corresponding routes and services with the NameServer.
The Producer performs route discovery, requests Broker routing information from the NameServer, and sends the message.
Consumer should connect to NameServer to get the relevant routing information, so that we can subscribe to the message.
The Broker is primarily responsible for storing messages, whether they are produced or subscribed to, and the source of the messages is the Broker.
Message sending (Producer) will only be sent to the master node, and then the Broker will synchronize the message, synchronized to the slave node, as a consumer (Consumer) will only be prioritized from the Master node, get the message, for consumption
Broker master nodes are unavailable or very busy and will choose slave nodes for consumption

1. Producer：
Responsible for producing messages, typically by the business system. The producer sends the message to the specified Topic by calling the API.

2. Broker：
The message storage center is responsible for receiving and storing messages from the Producer, while the Consumer also gets the messages from here.The Broker also stores metadata related to the messages, including consumer groups, consumer progress offset, queue information, etc. Each Broker can store multiple Topics, and each Topic can also be sliced and diced and stored in different Brokers. Each Broker can store messages from multiple Topics, and the messages from each Topic can be stored in different Brokers in pieces.In the actual deployment, Broker corresponds to a server and is divided into two types: Master and Slave, with Master being responsible for reading and writing, and Slave being responsible for reading, so as to realize data backup and load balancing.

3. Consumer：
Responsible for consuming messages, usually by a backend system. Consumers get messages by subscribing to a Topic and process them according to the business logic. Consumers can consume messages in a clustered or broadcast manner.

4. NameServer：
Acts as a provider of routing messages and is responsible for keeping metadata information about the Broker and making it available for querying by Producers and Consumers.NameServer is designed to be virtually stateless and horizontally scalable with no communication between nodes.

4.2 Basic concepts

4.2.1 Group

Groups are one of the core features of RocketMQ, and are divided into sender-side groups and consumer-side groups.

Sender Group: Allows multiple senders (Producers) to be organized into a sender group. Through the sender Group, batch processing and load balancing of messages can be realized to improve system performance and reliability.
Consumer Groups: RocketMQ allows consumers to subscribe to multiple topics at the same time, and these consumers can be organized into one or more consumer groups. Consumers within a consumer group can share the task of consuming messages to achieve load balancing and fault tolerance. When a consumer fails, other consumers can take over the consumption task to ensure that messages are processed in a timely manner. Consumer groups also support advanced features such as message filtering and sequential consumption.

4.2.2 Topic

Used to distinguish between the types of messages, indicating the logical name of a class of messages, the logical management unit of the message, regardless of the production or consumption of the message, need to execute the Topic. for example, a Topic is dedicated to the user order message to send, a Topic is dedicated to the deduction or increase of the points of the.

A sender can send a message to one or more Topics.
A message recipient can subscribe to one or more Topic messages.

4.2.3 Message Queue

MessageQueue is a data structure used to store and transmit messages in RocketMQ. Each MessageQueue has a unique identifier consisting of a Topic name and a queue number.MessageQueue has the following characteristics:

Unique identification: ensures that each MessageQueue can be uniquely identified and located.
Message Sequentiality: For messages in the same MessageQueue, RocketMQ guarantees that they are consumed sequentially, i.e., first-in-first-out (FIFO).
Load Balancing: RocketMQ achieves load balancing by dynamically adjusting the message distribution policy to evenly distribute messages to all MessageQueues, avoiding the situation where a MessageQueue is overloaded while other MessageQueues are idle.
High Availability: RocketMQ supports clustering multiple Broker nodes. Each MessageQueue can be master-slave replicated on different Broker nodes, providing high availability and data redundancy. Even if a Broker node fails, other nodes can continue to provide services to ensure reliable transmission of messages.

4.2.4 Tag

Tags are used in RocketMQ to categorize and filter messages. A producer can specify a Tag when sending a message, and a consumer can filter and subscribe to messages based on the Tag.The use of Tags can make the management and consumption of messages more flexible and efficient. For example, an e-commerce system may have different Tags based on the categories of goods (e.g., clothing, electronics, etc.), and consumers can subscribe to and process messages of specific categories based on these Tags.

4.2.5 Offset

Offset is used in RocketMQ to identify a consumer's position in the message queue. Each consumer maintains an Offset so that it knows where to start its next consumption, and its use ensures that messages are not lost, avoids duplicate consumption of messages, and supports sequential consumption of messages.

The message queue is an infinitely long array. When a message comes in, the subscript goes up by 1, and the subscript of this array is offset.
Ensure that messages are not lost: by persisting the Offset, even if the consumer is restarted after being down, it can continue to consume from the same position as the last time it was consumed, ensuring that messages are consumed at least once.
Avoiding repeated consumption of messages: consumers update Offset after successfully processing a message, ensuring that each message is only consumed once.
Support for sequential message consumption: for sequential messages, the sequential consumption of messages can be guaranteed by maintaining Offset.

5 Program implementation

5.1 Introducing and Implementing RocketMQ in Java

Introducing dependencies

In Java projects, client-side dependencies for RocketMQ are usually introduced through build tools such as Maven or Gradle. Take Maven as an example.file to add the following dependencies:
```
<dependency>
    <groupId></groupId>
    <artifactId>rocketmq-client</artifactId>
    <version>Latest version number（5.3.1）</version>
</dependency>
```

Realization of producers

In Java, this is accomplished by creating theDefaultMQProducerobject to implement the producer of the message. The producer needs to set the NameServer address and call thestart()method is initialized. Then, you can create theMessageobject and set the subject, label, and message content, and finally call thesend()method to send a message.


public class Producer {
    public static void main(String[] args) throws Exception {
        // Create a message producer and set the message producer group
        DefaultMQProducer producer = new DefaultMQProducer("your_producer_group");
        // Specify the NameServer address
        ("your_nameserver_address"); // Initialize the Producer.
        // Initialize the Producer
        ().

        // Create the message object and set the topic, tag, and message content.
        Message msg = new Message("your_topic", "your_tag", "Hello RocketMQ".getBytes()); // Send the message and get the result.
        // Send the message and get the result
        SendResult sendResult = (msg).
        ("%s%n", sendResult);

        // Close the producer instance
        ();
    }
}

Consumer Realization

In Java, this is accomplished by creating theDefaultMQPushConsumerobject to implement the consumer of the message. The consumer needs to set the NameServer address and consumer group name and call thesubscribe()method to subscribe to topics and tags. Then, register message listeners to handle incoming messages.


public class Consumer {
    public static void main(String[] args) throws Exception {
        // Creating a Message Consumer，and set up message consumer groups
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("your_consumer_group");
        // indicate clearly and with certaintyNameServeraddress
        ("your_nameserver_address");
        // 订阅indicate clearly and with certaintyTopicAll messages under
        ("your_topic", "*");

        // Registering a Message Listener
        (new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                for (MessageExt msg : msgs) {
                    ("Receive message: %s%n", new String(()));
                }
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });

        // Starting a Consumer Instance
        ();
    }
}

6 Summary

This article is just to understand the basic principles and implementation of RocketMQ, in later chapters, we will bring a complete explanation of RocketMQ.