Comparison and selection of three common File classes in Log4j2

In Log4j2, if the Rolling (supports scrolling and compression) class file Appender is not considered, it contains the following three files Appenders: FileAppender, RandomAccessFileAppender, and MemoryMappedFileAppender. Next, we will introduce the functions, advantages and disadvantages of these three Appenders, and how to choose and use them in actual research and development.

One and three types of Appender Introduction

1. FileAppender

FileAppender is the most basic file Appender in Log4j2. It is based on traditional writing logs to files, which is simple and reliable.

2. RandomAccessFileAppender

RandomAccessFileAppender is based on ByteBuffer and RandomAccessFile implementations, similar to the standard FileAppender, but it always uses memory buffers of size specified by the bufferSize parameter. According to the official documentation, performance can be improved by 20% to 200% compared to using the default configuration FileAppender. However, due to the use of memory buffers, memory overhead is increased and the risk of log loss is also increased. If the program crashes, the logs in the buffer will be lost.

3. MemoryMappedFileAppender

MemoryMappedFileAppender is based on a memory-mapped file. The files are mapped directly to memory, and the logs are written directly to this memory, and the virtual memory manager of the operating system is persisted to the storage device, allowing the logs to be written directly to the disk without passing through the operating system cache, reducing I/O overhead and significantly improving performance. However, the Appender takes up a lot of memory and depends on the underlying operating system and hardware support. If the program crashes, unpersistent logs may be lost.

The default memory mapped area size is 32M, which can be adjusted through the "regionLength" parameter; this Appender does not have a corresponding Rolling class, and cannot support log scrolling switching and backup.

2. Function comparison

characteristic	FileAppender	RandomAccessFileAppender	MemoryMappedFileAppender
Underlying implementation			Memory-Mapped File
reliability	high	Higher (when the buffer is not refreshed)	Slightly lower (when the memory map is not refreshed)
performance	Slightly lower	Slightly taller	high
Platform compatibility	high	high	Slightly lower, relying on operating system and hardware support
Memory usage	Lower	medium	Higher
No GC (Garbage-free) mode	support	support	support
File Rolling	support	support	Not supported
Log output order	Orderly	Orderly	Orderly

3. Synchronous performance comparison

1. Baseline environment

(1) Hardware: Windows notebook, configured as I5-1350P CPU, 32G DDR5 5200 memory, and Samsung MZVL4512HBLU-00BLL 512G SSD (sequential write speed is 2430MB/s).

(2) Software: Based on JDK 1.8.171, using version 1.37 JMH and version 2.24.3 Log4j2.

(3) Configuration: All three Appenders adopt the default configuration (append and immediateFlush are true), and use synchronous Logger for stress testing.

(4) Referring to the actual usage, the output is 100 fixed strings with a length of 100. At the same time, there is a 20% probability that the exception containing a 100-layer stack is output, and the log layout is:%d{yyyy-MM-dd HH:mm:} %-5level %pid %t %C.%M:%L - %msg %ex{full} %n。

(5) Pressure three times and take the average value of the result.

2. Configuration file

<?xml version="1.0" encoding="UTF-8"?>
<Configuration name="log4j2AppenderTest" status="error">
	<Properties>
		<Property name="">
			%d{yyyy-MM-dd HH:mm:} %-5level %pid %t %C.%M:%L - %msg %ex{full} %n
		</Property>
	</Properties>

	<Appenders>
		<Console name="Console">
			<PatternLayout pattern="${}"/>
		</Console>
		<File name="File"
			fileName="target/test-output/">
			<PatternLayout pattern="${}"/>
		</File>
		<RandomAccessFile name="RandomAccessFile"
						  fileName="target/test-output/" >
			<PatternLayout pattern="${}"/>
		</RandomAccessFile>
		<MemoryMappedFile name="MemoryMappedFile"
						  fileName="target/test-output/" >
			<PatternLayout pattern="${}"/>
		</MemoryMappedFile>
	</Appenders>

	<Loggers>
		<Root level="debug">
			<AppenderRef ref="Console" />
		</Root>
		<Logger name="FileLogger" level="debug" additivity="false">
			<AppenderRef ref="File" />
		</Logger>
		<Logger name="RandomAccessFileLogger" level="debug" additivity="false">
			<AppenderRef ref="RandomAccessFile" />
		</Logger>
		<Logger name="MemoryMappedFileLogger" level="debug" additivity="false">
			<AppenderRef ref="MemoryMappedFile" />
		</Logger>
	</Loggers>
</Configuration>

3. Pressure test code

@State()
public class Log4J2FileAppenderBenchmark {

    static Logger fileLogger;
    static Logger randomAccessLogger;
    static Logger memoryMappedLogger;
    static final Random RANDOM= new Random();
    static final double OUTPUT_EXCEPTION_PROBABILITY = 0.2;

    @Setup()
    public void setUp() throws Exception {
        ("", "");
        fileLogger = ("FileLogger");
        randomAccessLogger = ("RandomAccessFileLogger");
        memoryMappedLogger = ("MemoryMappedFileLogger");
    }

    @TearDown
    public void tearDown() {
        ("");
    }

		public void outputLog(Logger logger){
        if (() < OUTPUT_EXCEPTION_PROBABILITY) {
            (Const.MSG_HAVE_100_CHARS, Const.THROWABLE_HAVE_100_STACK);
        } else {
            (Const.MSG_HAVE_100_CHARS);
        }
    }

    @BenchmarkMode()
    @OutputTimeUnit()
    @Benchmark
    public void syncFileLogger() {
        outputLog(fileLogger);
    }

    @BenchmarkMode()
    @OutputTimeUnit()
    @Benchmark
    public void syncRandomAccessLogger() {
        outputLog(randomAccessLogger);
    }

    @BenchmarkMode()
    @OutputTimeUnit()
    @Benchmark
    public void syncMemoryMappedAppendLogger() {
        outputLog(memoryMappedLogger);
    }
}

4. JMH Parameters

The parameters executed by JMH are:-jvmArgs "-Xmx2g -Xms2g" -f 2 -t 4 -w 10 -wi 2 -r 30 -i 2 -to 300 -prof gc -rf json, that is, set the JVM parameter to -Xmx2g -Xms2g (the maximum and minimum heap memory are 2GB), use 2 forks (-f 2), each fork uses 4 threads (-t 4), each warm-up stage runs for 10 seconds (-w 10), pre-warming iterations 2 times (-wi 2), formal test runs for 30 seconds (-r 30), formal test iterations 2 times (-i 2), timeout is 300 seconds (-to 300), GC performance analysis (-prof gc), and output the test results to JSON format (-rf json).

5. Pressure test results

Appender Type	Average throughput	Memory allocation rate (MB/sec)	Number of garbage collections
FileAppender	42.5 ops/ms	1328.1 MB/sec	235
RandomAccessFileAppender	46.6 ops/ms	1498.4 MB/sec	266
MemoryMappedFileAppender	90.9 ops/ms	3162.5 MB/sec	561

6. Conclusion of pressure measurement

Based on the above reference environment and pressure measurement results, the conclusion can be drawn:

(1) The performance of MemoryMappedFileAppender is about twice as high as that of RandomAccessFileAppender and FileAppender.

(2) RandomAccessFileAppender's performance is slightly better than FileAppender, with a throughput of 8.8%.

In addition, from the pressure tests of the remaining CASEs:

(1) The fewer log fields and the smaller the volume of a single log, the more obvious the performance advantages of MemoryMappedFileAppender, which can be up to 10 times higher than FileAppender.

(2) Under the same benchmark environment, the number of pressure measurement threads is increased, and the performance of the three categories of Appenders has been improved, and the improvement of RandomAccessFileAppender and MemoryMappedFileAppender are even more significant. When increasing the pressure test thread to 64, FileAppender, RandomAccessFileAppender and MemoryMappedFileAppender have 46.6, 56.3 and 138.4 ops/ms, respectively, with FileAppender slightly improving. In contrast, RandomAccessFileAppender and MemoryMappedFileAppender's advantages in leading FileAppender are 20.8% and 197% respectively.

(3) When using JDK21 to perform pressure measurement on MemoryMappedFileAppender, an IllegalAccessException exception will be thrown. The reason is: since Java9, the module system (JPMS) has been introduced, which limits the ability of external modules to access internal APIs. Workaround: Add parameters to the JVM at runtime--add-opens /=ALL-UNNAMED --add-opens /=ALL-UNNAMED。

(4) Under the same benchmark environment, it was found that the performance of the three Appenders was significantly reduced by using JDK21 pressure test. The reason was unknown and it has been sent to the community.feedback。

5. Use suggestions

1. How to choose

(1) Under normal circumstances, it is recommended to use FileAppender first. Its performance is good, reaching 42.5ops per millisecond (most of the servers deployed in actual production environments use HD hard disks, and their IO performance is far inferior to SSDs, so the throughput may decline), it can meet most business needs, and is simple and reliable, supporting Rolling (file segmentation, scrolling and backup); RandomAccessFileAppender's performance is only slightly higher than FileAppender, but there is a risk of losing logs, so it is not recommended to use; in actual use, if FileAppender or RandomAccessFileAppender is used, it is recommended to use its Rolling class, namely RollingFileAppender and RollingRandomAccessFileAppender to meet the necessary requirements such as file segmentation, rotation and compression.

(2) If the performance requirements are high and the number of possible log loss can be accepted, instead of considering using MemoryMappedFileAppender, it is better to consider using a combination of asynchronous logs (AsyncLogger) and message queue Appender, which has better performance, small memory usage, and low coupling with the system platform.

(3) Judging from the pressure measurement results, there will be certain fluctuations in performance of different configuration parameters and usage environments. [Special recommendation]: According to the actual business usage scenarios, functional requirements, log configuration, etc., perform performance testing and analysis in advance to select the appropriate Appender.

2. Configuration recommendations

In the actual production environment, [Prohibit] set the append and immediateFlush parameters to false. Because the former will overwrite the original log, the latter may cause the real-time log to be viewed and the log loss.

6. Reference articles

（1）

（2）log4j2 2.