Once the Redis server is down, all the data in memory will be lost. Recovering this data from the back-end database puts a lot of pressure on the database, and the performance is certainly not as good as reading from Redis, which slows down the application. Therefore, for Redis, it is important to realize the data objectification, avoiding recovery from the back-end database is critical.
1, AOF log
AOF logging is the first execution of the command to write the data into memory, and then only record the log to save the text form, the following chart: "*3" means that there are three parts of the command, each part of the "$ + number" at the beginning, "$3 set" means that there are three bytes in the part, referring to the "set" command, "$7 testkey" means that there are seven bytes in the part, that is, the "testkey" command, and so on. And so on.
AOF post-write logging will only be logged if the command can be executed successfully, avoiding additional checking overhead and the appearance of logging the wrong command, and not blocking the current write operation. Having said that vantagepersuade exposures, if it goes down just after executing the command before it has time to log, there is a risk of loss. Second, AOF logging is executed in the main thread, which may pose a risk of blocking if there is too much pressure on writing the log file to disk.
The AOF risk is related to writing back to disk, for which three types are provided write-back strategyThis is the three optional values for the appendfsync configuration item:
(1) Always write back synchronously: After each write command is executed, the logs are immediately written back to disk synchronously.
(2) Everysec Write Back Every Second: After each write command is executed, the log is first written to the memory buffer of the AOF file, and the contents of the buffer are written to disk every second.
(3) No OS-controlled write back: after each write command is executed, the log is first written to the memory buffer of the AOF file, and the operating system decides when to write the contents of the buffer back to the disk
The advantages and disadvantages of each of the three strategies are summarized below:
Selecting a write-back strategy is not all well and good; as more and more write commands are received, the AOF file will get larger and larger, causing performance problems. There are three main areas:
(1) The file system itself has a limit on file size and cannot save files that are too large.
(2) If the file is too large, it will be inefficient to add commands to it later.
(3) If downtime occurs, the commands recorded in AOF have to be re-executed one by one, and the file is too large, resulting in the entire recovery process being very slow and affecting the normal use of Redis.
What if the log file is too big? This time, AOF Rewrite mechanismThe AOF file will record the corresponding commands when a key-value pair is repeatedly modified by multiple write commands. When a key-value pair is repeatedly modified by multiple write commands, the AOF file will record the corresponding multiple commands, and rewrite, only according to the key-value pair of the current state of the latest write command for it to generate the corresponding command, so that a key-value pair of the rewrite log in the only command on the line, and in the log recovery, only the execution of this command, you can complete the key-value pair of the write directly. Take a chestnut:
AOF rewriting does not block the main thread, the rewriting process is done by a background thread, bgrewriteaof, which guarantees the integrity of the data through a memory copy and two logs.
2, RDB memory snapshots
Memory Snapshot RDB is the abbreviation of Redis DataBase, compared with AOF, RDB records the data at a certain moment, not the operation, so when we do data recovery, we can directly read the RDB file into the memory, and quickly complete the recovery. But at the same time, we also face two problems:
(1) What data to take snapshots of? This is related to the efficiency of the snapshot execution.
(2) Can data still be added, deleted, or changed when taking snapshots? This is related to whether Redis is blocked and can process requests properly at the same time.
To provide a guarantee of reliability for all data, a full-volume snapshot records all the data in memory to disk, no less. This takes a lot of time, and the more full data there is, the larger the RDB file becomes, and the more time it takes to write the data to disk. For Redis, its single-threaded model dictates that we try to avoid all operations that block the main thread, and Redis provides two commands to generate RDB files, save and bgsave:
(1) save: executed in the main thread, it will cause blocking.
(2) bgsave: creates a sub-process dedicated to writing RDB files, avoiding the blocking of the main thread, which is the default configuration for Redis RDB file generation.
bgsave avoids blocking the main thread and can receive requests normally, however, in order to ensure the integrity of the snapshot, it can only process read operations and cannot modify the data that is executing the snapshot.Redis will then use the copy-on-write (COW) technology provided by the operating system to process write operations normally while executing the snapshot. This is illustrated in the figure:
Simply put, the main thread fork spawns a bgsave child process that shares all of the main thread's memory data. bgsave child processes run to read the main thread's memory data and write it to the RDB file. At this point, if the main thread is also reading all of this data (for example, key-value pair A in the figure), the main thread and the subprocess have no effect on each other. If the main thread wants to modify the data (e.g., key-value pair C in the figure), it generates a copy of that data, and the bgsave subprocess writes that copy to the RDB file, while the main thread can still modify the original data directly in the process.
So far the above two questions "which data to do snapshots", "do snapshots when the data can be modified" are solved. A new question arises, how often is it appropriate to take snapshots? If the machine goes down before the second snapshot, there may be data loss problems, and if it is too frequent, there will be a situation where the first snapshot is not finished and the second one starts. Although bgsave doesn't block the main thread when it executes, if it executes full snapshots too often, it will bring extra overhead to the disk, and the bgsave child process needs to be created from the main thread by a fork operation, which will still block the main thread if it operates too often.
At this point, incremental snapshots on the scene, after doing a full snapshot, subsequent snapshots of only the modified data for the snapshot record, so you can avoid the overhead of each full snapshot. For example, if we take a snapshot of T1 and T2, we only need to write the modified data to the snapshot file.
Although we remember what data has been changed, the "remember" operation requires us to use additional metadata information to record it, which introduces additional space overhead. Sometimes, when the changes are small, the extra space needed to record them is not worth the effort. In this case, we can use a mix of AOF logging and memory snapshots, and in between snapshots, we can use AOF logging to record all the commands in the meantime.
As shown in the figure, T1 and T2 moments of modification, using the AOF log record, in the second time to do the full amount of snapshots, you can clear the AOF log, because the changes have been recorded in the snapshot. This method can enjoy the benefits of rapid recovery of RDB files, but also enjoy the simple advantages of AOF only record operation commands, can be said to be a fish and a bear's paw at the same time.