Software Performance Testing Analysis and Tuning Practical Path (2nd Edition) is a book published by Tsinghua University Press, written by Zhang Yongqing, the book is divided into nine chapters, as follows
Book Description:A Practical Path to Analyzing and Tuning Software Performance Testing (2nd Edition)
This article is followed by
The Practical Path to Software Performance Testing Analysis and Tuning (2nd Edition) Readers' Notes (I) General Introduction (above) - Really looking at performance testing from the perspective of performance analysis and tuning
Moving on.
6), performance analysis and tuning practices (indexing and sub-library and sub-table)
When inserting data when splitting a library and a table, you can write data to the library and the table as shown in the following figure.
Query data can be queried as shown in the following figure, the routing table in the following figure is very critical, used to be able to quickly locate the need to query the data is in which the sub-library and sub-table.
7), Performance Analysis and Tuning Practice (Layer Filtering)
The focus of the cascade filter is as follows:
- Filter out all invalid requests that should be filtered at the corresponding level as much as possible at different levels, so that the requests that come into the database at the very end are all valid requests.
- Error fronting and throwing exceptions early. For exception requests, the earlier the exception is thrown, the more favorable it is to alleviate the system's processing capacity and save resource consumption.
- Avoid duplicate requests as well as malicious requests via bots, thus reducing the processing pressure on the system and better protecting it. The Practical Path to Software Performance Testing Analysis and Tuning (2nd Edition) Readers' Notes
7. Performance analysis and tuning practices (common performance problems)
(1), the performance index curve frequently appear large-scale jitter
- (1) The system may be experiencing frequent Full GC. Full GC is a mechanism for garbage collection in Java applications, and generally if Full GC occurs, the application will stall for a short period of time. For an introduction to Full GC, you can refer to the introduction in subsection 5.1.7 of the paperback. At this time, you can first go to the GC log of the application, if Full GC is very frequent and there is no memory leak, then you can refer to the paper book 5.4.1 subsection on how to reduce GC to solve the problem. A Practical Path to Software Performance Testing Analysis and Tuning (2nd Edition) Reading Notes
- (2) The system takes a long time to query, modify or delete data at a certain time, which leads to unstable overall performance. For example, in the performance pressure test query, most of the parameterized incoming parameters, the query results in very little data, but may be a certain number of parameters to query a very large amount of data, resulting in the system in the processing of these large amounts of data time-consuming. A Practical Path to Software Performance Testing Analysis and Tuning (2nd Edition) Readers' Notes
- (3) When the system is querying, sometimes it may hit the cache, sometimes it may not hit the cache. Hit the cache, the query will be very fast; can not hit the cache, you need to query the database, but the time to query the database is certainly longer than the cache, so it will cause system performance instability. Usually the database will also have a cache, if you hit the database cache, the query will be faster; if not, the query time will certainly become longer. The Road to Practicing Software Performance Testing Analysis and Tuning (2nd Edition) Readers' Notes
- (4) If the system is deployed in a distributed manner, then you can check whether the processing power of each node in the distributed processing system is consistent; if it is not consistent, it may also cause the system to jitter frequently ref. A Practical Path to Software Performance Testing Analysis and Tuning (2nd Edition) Readers Notes
- (5) The server connection is not enough to cause the connection batch release and then suddenly batch connection, once the batch release connection, the system TPS will immediately rise, because at this time can establish a connection. When the connection is full, the request can not be processed, thus having to wait, which in turn causes a sudden drop in TPS.
2), when increasing the number of concurrent users, the system's TPS and average response time have not been able to be improved
Usually, after checking the application server will find that the CPU, memory and other resources have not reached the upper limit of the use of the system, but the system has a bottleneck in the processing, then the system must be somewhere "blocked". At this time, you need to continue to do the following checks:
- (1) Whether the click-through rate actually comes up during performance pressure testing. If the click-through rate or the number of requests per unit of time does not come up, it means that the pressurized testing machine can not provide more pressurized testing capacity. Especially in large distributed systems, a single pressure testing machine is often not enough, because a single pressure testing machine, whether it is the network connection, or bandwidth, as well as their own CPU, memory and so on there will be a great deal of limitations, the performance of the pressure test is not only the server resources will be a great deal of consumption, to provide the ability to pressure test the pressure testing machine will be a great consumption of resources.
- (2) Check the network bandwidth usage and troubleshoot whether the bottleneck is caused by network bandwidth limitation. At this point, the links that need to be checked for network bandwidth include all the places where there are network request interactions, such as the pressure test machine to the Web server, the Web server to the application server, and the application server to the database server. As shown in the following figure. (The Road to Software Performance Testing, Analysis and Tuning Practice (2nd Edition), by Zhang Yongqing, reprinted with permission)
- (3) Refer to subsection 5.3.2 of the paper book to use the jstack command line tool to view the thread stack of the Java system, and analyze the current system bottleneck directly from the thread stack because of what resource is waiting for, and the resource may be an invisible and not easy to find resource.
- (4) If you are not familiar with the application of point (3), you can use the most stupid way is according to the request processing link process, from the top to the bottom or from the bottom to the top of the order to check. At this point, you need to firmly believe that the system must be "blocked somewhere", carefully check through the checklist to check, you will be able to find the "blocked" location. This is just like the water supply system, if a user suddenly feedback that my tap water pressure is very small, the water pressure has not been up, then the water company's maintenance staff to the door, certainly from the user's home as a starting point, and then on the water supply link in each link to investigate, until they find which link is congested. (The Road to Software Performance Testing, Analysis and Tuning Practice (2nd Edition), by Zhang Yongqing, reprinted with permission)
- (5) If you still can not find the cause of the problem according to the previous four points, then you can try to reduce the intermediate links to reduce the impact of the uncertainty factor, and then conduct a pressure test comparison, first to determine the possible scope of the problem, and then in a clear range to find the specific cause. For example, as shown in the figure, the Web server is removed, so that the pressurized test machine request directly to the application server for pressure testing. As shown in the figure below.
3）、When increasing the number of concurrent users, the TPS of the system decreases slowly and the average response time increases slowly.
When the system appearsIf the TPS is decreasing and the average response time is slowly increasing, it is possible that the system has reached an inflection point in performance and has reached its maximum processing capacity. At this point, you need to do the following checks
- （1) Application server resources such asCPU, RAM,Whether IO etc. has reached its usage limit.
- （2) Whether the resources of the database server and the number of links to the database, etc. have reached the upper limit of utilization.
- （3) If paragraph (1) point or point (2) The resource usage in the point has reached the upper limit, you can expand the server resources and then continue the pressure test again. Usually, when the performance inflection point occurs, a resource in the server has also reached the upper limit of its usage.
(4), the performance of the pressure test process, the server memory resource utilization has been gradually and slowly rising, as the performance of the pressure test continues, there is never a decline or within a certain range of small fluctuations, and at this time the TPS is also slowly declining
When this happens, it is likely that there is a memory leak, in which case you can do the following checks:
- （1) Check the system log for any memory overflow error messages.
- （2) Refer to the use of paperbacks during performance pressure testingin subsection 5.2.1jconsole orin subsection 5.2.2jvisualvm to further localizeDoes the Java JVM have a memory leak. (The Practical Path to Software Performance Testing, Analysis, and Tuning (2nd Edition), by Yongqing Zhang.)
- （3) If there isJVM Memory Leaks, you can refer to the use of paper booksin subsection 5.3.3MemoryAnalyzer tool to further analyze where in the code the memory leak is occurring.
- （(4) performance pressure test process, when the number of concurrent users and click-through rate is unchanged, the server's resource consumption should be in a stable range, or within a certain range of constant small fluctuations, which is more normal.
- （4) If paragraph (3) Points can't be lined up for specific problems, refer to paperbacksThe hierarchical analysis in subsection 1.6.1 is used to locate the problem.
(5), in the distributed deployment of the performance of the environment during the pressure measurement of each application server CPU or memory resource consumption is too large a difference.
When this occurs, the following troubleshooting can be done:
- （1) Check whether the hardware configuration of each application server is consistent. (The Practical Path to Software Performance Testing, Analysis, and Tuning (2nd Edition), by Yongqing Zhang, reprinted with permission)
- （2) Check the operating system, application software, and database software for each application server,JDK software and other versions and whether the configuration information is consistent. (Software Performance Testing, Analysis, and Tuning: A Practical Approach (2nd Edition) by Yongqing Zhang, please cite the source)
- （3) If paragraph (1) point and paragraph (2) Points are fine, checkWhether the load balancing of web servers forwarding requests to application servers is even. For exampleAre there inconsistent forwarding weights in the Nginx configuration, or are thereConfiguration limits for ip_hash, etc., can be found in the paper bookIntroduced in subsection 3.1.1.