This article is shared from Tianyi Cloud Developer CommunityFault test simulates network packet loss》, author: y****n
1. Simulate network packet loss tool—TC (Traffic Control)
TC (Traffic Control) is a module that comes with Linux and generally does not require installation. TC requires kernel 2.4.18 or above. Traffic controller in Linux operating system TC is used for traffic control of the Linux kernel. It uses queue regulations to establish a queue for processing data packets and defines how the data packets in the queue are sent, thereby achieving control over traffic. The queue regulations used by the TC module to implement the flow control function are divided into two categories: one is the classless queue regulations, and the other is the classified queue regulations. Classless queue regulations are relatively simple, while classification queue regulations introduce concepts such as classification and filters, which enhance their flow control functions.
Classless queueThe regulations are queue regulations that do not distinguish and treat data flows entering network equipment (network card). Queues formed using classless queue specifications can receive packets and re-arrange, delay or discard packets. The queue formed in this type of queue stipulates that the traffic of the entire network device (network card) can be shaped, but it cannot be divided into various situations. Commonly used classless queue regulations include pfifo_fast (first in, first out), TBF (token bucket filter), SFQ (random fair queue), ID (forward random packet loss), etc. The traffic shaping methods used in this type of queue are mainly sorting, speed limiting and packet loss.
Classification queueThe regulations are queue regulations for data packets entering the network equipment to be treated in a classified manner according to different needs. After the data packet enters a classified queue, it needs to be sent to a certain class, which means that the data packet needs to be classified. The tool for classifying data packets is a filter. The filter will return a decision, and the queue stipulates that the data packets are sent to the corresponding class for queueing based on this decision. Each subclass can again use their filters for further classification. The packets are not queued into the queue contained in the class until no further classification is required. In addition to being able to include other queue regulations, most categories of queue regulations can also perform sculpting traffic. This is very useful for situations where simultaneous scheduling (such as using SFQ) and flow control are required.
The basic principle of Linux traffic control is: after receiving packets come in from the input interface, they discard data packets that do not meet the requirements through traffic limit (Ingress Policing), and the input multiplexer (Input De-Multiplexing) will be used to determine and select. . If the destination of the receiving packet is the host, the packet will be sent to the upper layer for processing. Otherwise, forwarding is required and the received packet will be handed over to the Forwarding Block for processing. The forwarding block also receives packets generated by the upper layer of the host (TCP, UDP, etc.). The forwarding block determines the next hop of the processed packet by viewing the routing table. The packets are then arranged so that they are delivered to the Output Interface. Generally, we can only limit the packets sent by the network card, and cannot limit the packets received by the network card, so we can control the transmission rate by changing the transmission order. Linux traffic control is mainly processed and implemented when the output interface is arranged.
2. Common TC commands
(1) Analog delay transmission
# tc qdisc add dev eth0 root netem delay 100ms
This command sets the transmission of the eth0 network card to send a delay of 100 milliseconds;
(2) Analog delay fluctuations
# tc qdisc add dev eth0 root netem delay 100ms 10ms
In more realistic cases, the transmission delay will not be accurate to 100ms, and there will be certain fluctuations. This command sets the transmission of the eth0 network card to send a delay of 100ms ± 10ms (any value between 90 ~ 110ms);
(3) Simulated delay fluctuation randomness
# tc qdisc add dev eth0 root netem delay 100ms 10ms 30%
To further enhance the randomness of this fluctuation, the command sets the transmission of the eth0 network card to 100ms, and at the same time, about 30% of packets will be delayed by ± 10ms;
(4) Simulate network packet loss
# tc qdisc add dev eth0 root netem loss 1%
This command sets the transmission of the eth0 network card to randomly drop 1% of the packets;
(5) Simulate network packet loss success rate
# tc qdisc add dev eth0 root netem loss 1% 30%
This command sets the transmission of the eth0 network card to randomly drop 1% of the data packets, with a success rate of 30%;
(6) Delete related configuration
# tc qdisc del dev eth0 root netem delay 100ms
This command changes the add in the corresponding command to del to delete the configuration;
(7) Simulation package reset
# tc qdisc add dev eth0 root netem duplicate 1%
This command sets the transmission of the eth0 network card to randomly generate 1% duplicate packets
(8) Simulation package is damaged
# tc qdisc add dev eth0 root netem corrupt 0.2%
This command sets the transmission of the eth0 network card to randomly generate 0.2% of damaged packets. (Kernel version must be above 2.6.16)
(9) Simulation package disordered order
# tc qdisc change dev eth0 root netem delay 10ms reorder 25% 50%
This command sets the transmission of the eth0 network card to: 25% of the packets (50% related) will be sent immediately, and the other delay will be 10 seconds.
(10) Check the network card configuration
# tc qdisc show dev eth0
This command will view and display the relevant transmission configuration of the eth0 network card
(11) Check the packet loss rate
# tc -s qdisc show dev eth0
4. Simulate packet loss process
First, you need to determine where to create packet loss, analyze specific scenarios, such as the process of creating private images using the system disk, whether to lose packets in the message queue from cinder-api to volume, or whether cinder-volume needs to write data through storing outward ceph disks Or cinder-volume needs to write data by storing outward object storage disks. After confirming, you need to find the network card used for the corresponding steps. After confirming the network card,
(1) First, it is necessary to determine the node where the operation involves the service: kubectl get pod -n az3|grep cinder-volume |awk '{print $1}'| xargs -I{} kubectl exec -n az3 {} -- grep - rn image-id /var/log/cinder/
(2) Set the network card packet loss rate on the node where the service is located: tc qdisc add dev bond2 root netem loss 10%;
(3) Check the packet loss rate: tc -s qdisc show dev bond2
(4) Verify that packet loss is indeed caused by ping the IP of the corresponding network card service (this example ping object stores IP): ping 10.24.255.1
(5) After the test is completed, delete the packet loss configuration: tc qdisc del dev bond2 root netem loss 10%