Recently found a web site on the server is always very high CPU, the site is deployed on IIS, I have multiple sites on IIS, each site a process pool, each process pool name are based on the site to take, so it's easy to see which site ate the CPU, the site has been running for more than ten years, is based on the web site written in the .net 4.8 framework ( Ten years old project refactoring words do not need to mention, the new project we still use .net core plus a front-end framework, front-end and back-end separation , really fragrant)
When I found that the CPU was very high, my first reaction was to use windbg to analyze the dump file of the process pool (i.e., create a .dmp file, windbg is loaded .dmp file to analyze), because I have not done this before, it really took a long time to step on the pits (2 days, so this article is a summary of my stepping on the pits for 2 days)
First of all, the server system I use is windows server2019 64 bit operating system, my web site's IIS process pool -> advanced settings to enable "enable 32-bit applications" (because a read/write excel component is based on 32-bit), so my process pool is running in 32-bit mode, but I did not know at first. The process pool is running in 32-bit mode, but I didn't know that at first, I found the process pool directly in the operating system's task manager, and then right-clicked on "Create Dump File", but you must pay attention to, there is a big pit here, because the 64-bit operating system's task manager is 64-bit (the task manager's process is 64-bit), and the process pool is running in 32-bit mode, and then right-clicked on "Create Dump File". (task manager process name is ), so I am using the 64-bit task manager, to create a 32-bit process pool dump file, here must not do so, if you do so, the next no matter what you do what efforts, will get a lot of inexplicable feedback (windbg can load, but there are always inexplicable problems, visual studio can analyze, but also can not get the right results, I will later on the windbg can load, but there are always inexplicable problems, visual studio can analyze, but also can not get the right results, I'll later on the windbg can be analyzed, and then click "create dump file". can not get the right results, I will say later what is missing in the results, in short, you remember this pit is I spent more than a day to step out 。。。。。) If you want to know whether your process pool is 64-bit or 32-bit, you can open the server's task manager and click on the "Details" tab, then right-click on the title of the task manager list, and "Hide Columns, Select Columns" will appear. If you click on "Select Columns", then drag the scroll bar and pull down to find the "Platform" check box (Figure 1), the Task Manager will have an additional column of platforms, which will show "64-bit or 32-bit "as shown in Figure 2 below
Figure 1
Figure 2
From Figure 2 above, it can be noticed thatI have PID=11720 and it's 32-bit, but my task manager is 64-bit, and that creates a problem with the dump file.The correct way to create the dump file is to use the 32-bit task manager to create the dump file for the 32-bit process, the 64-bit system actually comes with a 32-bit task manager, the path is as follows: C:\Windows\SysWOW64\ (the path of the 64-bit task manager is: C:\Windows\System32\), open the 32-bit task manager. Before you open the 32-bit task manager, you have to close the 64-bit manager that has been opened before, open the 32-bit manager, the same to check the "platform" column (to ensure that you do not open the wrong task manager), you can see from the following figure 3, I PID = 9056 This process pool is shown as a 32-bit, but also shows a 32-bit (and the above picture of the process pool). (It doesn't match because I just picked a random screenshot, it's just to prove that the bit counts of the task manager and the process pool are the same)
Figure 3
This operation of creating a dump .dmp file of a process pool must be noted!!!! : use and their own process pool bit number consistent task manager to create dump file .dmp is very critical (32-bit process pool with 32-bit task manager to create a dump file, 64-bit process pool with 64-bit task manager to create a dump file, how to open a different task manager above there is an introduction)! Most of my time is wasted because of the creation of the wrong dmp file led to the analysis results have not been correct (windbg can be loaded, visual studio can be loaded (visual studio can analyze the .dmp file, and it is very good to use!) The two tools can be analyzed, but there is no way to give accurate results! (This is very fatal, because I have no doubt that the .dmp file will have problems), such as visual studio dead give "thread CPU utilization summary" this key data, other data to give, so I mistakenly think that my .dmp file is not a problem, so fall into the pit inside the fall! So long 。。。。 (If it simply won't load, I'd also suspect that my .dmp file is the problem)
You have to go and manually create the dump file with task manager when your CPU is high, you can also use other tools that automatically create .dmp files when the CPU exceeds a certain threshold (search for yourself, I'm not good at presenting my experience without using tools)
If you have created a .dmp file according to the above operation, and want to quickly know where the CPU is high in the cause (after all, now you are online environment is a problem, which have time to think about how to use windbg, you are now concerned about where the problem is, if you are like this, please pull the article a little bit behind the location of the use of windbg directly skip the problem, the next paragraph is to say that windbg problem, if you just want to quickly solve the problem first, directly to the end of the article visual studio analysis.dmp file introduction)
==================================dividing line=================utilizationWinDbganalyze.dmpfile=============================================================================================
Here's how to analyze the problem using windbg (don't bother reading this paragraph if you're on fire right now, scroll down to is analyzing the .dmp file with visual studio to read it)
Because when I wrote this blog post, I was really very busy, I do not write a blog post and I am afraid that I forgot this pass, next time I have to step on the pit, so simply write a little bit, with windbg also stepped on a lot of pitfalls, in order to save time (now really busy), simple to say a little bit:
1, make sure you use the windbg version is new enough (because the new version will automatically download from Microsoft's symbol server and this dmp file match the relevant dll), I installed a total of three versions of windbg, from the old version to the new version are: 6.12.0002 (the oldest), 10.0.26100.1, the latest version: 1.2407. 24003.0 (Debugger client version), 10.0.27668.1000 (Debugger engine version). 24003.0 (Debugger client version),10.0.27668.1000 (Debugger engine version)
I found that it seems that the old version of windbg is divided into 64-bit and 32-bit windbg, but the latest version, I look at the installation directory of the file, it is compatible with both, because some people say that you have to use 32-bit windbg to analyze the 32-bit .dmp file, and 64-bit analyze the 64-bit .dmp file, in fact, I use the latest version of windbg, it is possible to load the 64-bit .dmp file and analyze, I use the latest version of windbg to try to load 32-bit .dmp file. I used the latest version of windbg to try to load the 32-bit .dmp file, I found it from the task manager Debug Engine Host Process this process shows 32-bit, and then the task manager to the file location is C:\Program Files\WindowsApps\Microsoft. WinDbg_1.2407.24003.0_x64__8wekyb3d8bbwe\x86 This folder under the exe , to see that it calls the x86 folder under the exe file, that it should be correctly recognized the .dmp file type, and then used the 32-bit exe to open the .dmp file, and then I tried to use it to load a 64-bit .dmp file, and found that there is one more Debug Engine Host Process in the task manager, which does not show 32-bit, and the path of the file it loaded is shown on the task manager as C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2407. 24003.0_x64__8wekyb3d8bbwe\amd64 , obviously loaded different types of analysis tools according to the .dmp file. The task manager screenshot is below:
Conclusion: you don't bother with 32-bit or 64-bit windbg, install the latest version and it will automatically recognize the .dmp file!
Attachment: the latest version of windbg installation tutorial, first of all, Microsoft's address /zh-cn/windows-hardware/drivers/debugger/, click on "Download WinDbg", and then will be downloaded to a " ", double-click this file, if you can successfully complete the installation, then you can. ", double-click this file, if you can install everything successfully completed, it can be, if you can not install the prompt "can not open the sub-application package", then use Notepad to open the "", there is a Uri =" /dbazure/prod/1-2407-24003-0/", copy this Uri to download and install, that's how I installed it, I don't know if it's feasible for you.
Screenshots of the latest version are below:
After the above, you should have the latest version of WinDbg installed (Do not use the old version of WinDbg, cherish life, I tossed 3 versions, the latest version is the best to use (can automatically download the analysis of your dmp file things needed, 4k display resolution display adapted to the old version is not adapted to the old version, can automatically identify the .dmp file is 32-bit or 64-bit, automatically load the correct engine to analyze your .dmp file, the three reasons enough for you to use! The latest version））
After installation, windows bottom left corner search "windbg", and then right-click administrator privileges to run (this is my habit, this tool I like to administrator privileges to run for fear of permissions problems of all kinds of unknown errors, in fact, whether or not you need to administrator privileges to run I'm not sure), and then click on the "File->open Source file", and then select the .dmp file you just.
Normally, it should automatically download the required .dll files, if not, you can run: .reload /f (This process is very long, sometimes it takes ten minutes, if you have a ladder to hang a ladder will be much faster）
After the above steps, it is reasonable to say that you should now have the required Dll, you can execute the following command: .cordll -ve -u -l (This command allows WinDbg to uninstall and load the core dll again, and show the loaded situation.Microsoft WinDbg using the following documents: /zh-cn/windows-hardware/drivers/debugger/getting-started-with-windbg , the related meta-commands Microsoft instructions address: /zh-cn/windows-hardware/drivers/ debuggercmds/meta-commands）
The result of executing the command .cordll -ve -u -l is shown below:
0:000> .cordll -ve -u -l CLRDLL: C:\Windows\\Framework\v4.0.30319\:4.8.9261.00 f:8 doesn't match desired version 4.8.4739.00 f:8 CLRDLL: Loaded DLL C:\ProgramData\Dbg\sym\mscordacwks_x86_x86_4.8.4739.\665A9716849000\mscordacwks_x86_x86_4.8.4739. Automatically loaded SOS Extension CLR DLL status: Loaded DLL C:\ProgramData\Dbg\sym\mscordacwks_x86_x86_4.8.4739.\665A9716849000\mscordacwks_x86_x86_4.8.4739.
As you can see from the picture, at first WinDbg is loaded directly from the address C:\Windows\\Framework\v4.0.30319\ (hereinafter referred to as the local dll), but the version is not right, because the version of the local dll is 4.8.9261 (you can find the dll, right-click on Properties to see the details), and then WinDbg is again Then WinDbg started to load from C:\ProgramData\Dbg\sym\mscordacwks_x86_x86_4.8.4739.\665A9716849000\mscordacwks_x86_x86_x86_4.8.4739. The path was loaded, and it finally loaded successfully. This file is what the latest version of WinDbg automatically downloads from Microsoft's symbol server.
The reason why the DLL version is not correct: the .net framework installed on my local development machine is 4.8.9261, while the .dmp file I created on the server at the time of creation of its process runtime environment is 4.8.4739.00, so there will be a mismatch. It's because: one version on the server, another version of .net is used locally for analysis. When this version is not right, I used the old WinDbg from the Internet a variety of search articles, that is copied from the server down to the local, and then use the .cordll -lp D:\ your DLL path, a variety of get, dead can not be loaded correctly, this pit is also fell into a long time, after downloading the latest version of this problem is not a problem, so it has been emphasized that with the latest version of the old version of the need to study too much! commands, wrong easy to draw the wrong conclusions (mainly because I'm not familiar with this, familiar with the old version of the estimated use of the hand, but I believe you can see here at the moment, it is estimated to be a newbie WinDbg, and I'm the same as the honest use of the latest version of the time to save a lot of tossing environment)
What version is used on the server, you can execute the command !analyze -v in WinDbg, when it finishes, there is a: which is the version number of the server. After the command is executed, the result is as follows (there are too many results, I only copied the key one, you can see it by Ctrl+F searching for this "" string in your WinDbg)
0:000> !analyze -v
******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* KEY_VALUES_STRING: 1
Key :
Value: 4.8.4739.0
The point is: I used the old windbg version 6.12, then I searched a lot of articles on the Internet, then I copied the articles from the server to my local area, and then I learned to specify the path of the three Dlls (I think this is the right theory), and then I used the old windbg version 6.12.
In fact, it took a really long time to get it to work, but it didn't work, and the tip was something like the following
0:000> .cordll -I clr -lp E:\IISW3WP\core86server
CLR DLL status: No load attempts
But with the latest version, are automatically according to the .dmp file to download the required dll to a specific directory, and then load it downloaded this dll, and do not need to specify the directory, which is like a tool produced by Microsoft, foolproof operation, I initially is to look up all sorts of articles, according to the article inside the various commands to try, and never loaded a successful, so cherish life.
Use the latest version of WinDbg!
After executing the .cordll -ve -u -l command, make sure that you see a prompt like the one I wrote above, and you should be able to execute the relevant commands at this time
The first command: !runaway (the official Microsoft explanation of the command: /zh-cn/windows-hardware/drivers/debuggercmds/-runaway ), not surprisingly will get the following results:
Then you can see that there are many threads listed, for example, if I select thread 44, then you can use the command ~44s to enter into this thread (after entering the command and pressing Enter, the following will change to 0:044), and then execute !clrstack to see more detailed information, you can see that the
You can also list all thread heap information with the command: ~*e !clrstack
Then it will list a bunch of information, from the information inside the general can see which page may be a problem. For example, I can see that the above is a problem with the ordersend_orderoutlistfishy_aspx page, but you have to analyze what the problem is.
However, I found that Visual studio analysis of this .dmp file that is quite excellent, the results of which directly to me which line, I directly look at this line of code on the suspicion of the problem, do not want to toss the WinDbg is strongly recommended to use Visual Studio directly to analyze try!Because this page displays very, very, very much content, and then there is a control label on the page, and then with +=xxxx, spelled out very, very, very much, so I directly suspected on here, and then changed to StringBuilder, reducing the time by three-quarters (from the technical solution, from 2 minutes to 30 seconds), and then later on from the business continued to go to the communicated with colleagues who use this page, directly change the function (summarized in the historical legacy of this page, this page of this data pressed no one to see, to see this page of people, pressed do not need so much data, only show a very small part of the data on it)
But I am using WinDbg now have the following questions, need WinDbg gurus to help me out:
1, How can I show exactly which line of code like Visual Studio? I've copied all the pdb files from my webform root directory to my local area and added this directory to my symbol directory with E:\IISW3WP\pdb , then with .sympath+ E:\IISW3WP\pdb , and with
.reload /f re-force loaded.
2, I use !runaway command to show the thread, then this 24th thread is the longest time, then I use the command ~24s to enter the thread, and then use !clrstack command to want to see the relevant information, the result is found to report an error, the content is as follows,,, why can't I get it? Please see this post, know what the problem is, or have any suggestions please leave me a message, I want to learn this WinDbg, I've searched and searched without finding a solution. chatgpt also did not answer right. According to my understanding Visual studio can display, WinDbg certainly can, I suspect that my symbol settings + Source Link problem, in the end, how to set up, and Visual studio and so clearly indicate which line of code problem? I used WinDbg to find only which page is the problem, I want to continue to find the problem I still have to go to the page code carefully, and hit the log to know.
0:024> !clrstack
OS Thread Id: 0x6d40 (24)
Child SP IP Call Site
GetFrameContext failed: 1
00000000 00000000
==================================dividing line=================utilizationVisual Studioanalyze.dmpfile===========================================================================================
Here's what happens when you use visual studio to analyze a .dmp file. If you're on fire right now, don't bother with windbg yet, just open the visual studio tool!
I do not know your problem, can not directly use visual studio to analyze the .dmp file to find the problem (I believe it can be), anyway, mine is directly with visual studio to find the problem, very intuitive, do not want to pour windbg a bunch of commands and the environment, is really the universe's first editor (do not carry this point of view, the bar is that you are right), directly open the visual studio (I use visual studio version 2022), open visual studio and then click on "File -> open the file", select just this .dmp file. visual studio (I use visual studio2022 version), open visual studio and then click on "file -> open file", select just this .dmp file, such as Figure 4, click on the right side of the operation tab under the "Run Diagnostic Analysis" button.
Click on "Run Diagnostic Analysis" in Figure 5, check all those options, click the "Analyze" button, there will be a waiting dialog box, wait for it to complete the implementation, you can see the results, as Figure 6
Figure 6
Then mouse click on the "thread CPU utilization summary", the right scroll bar scroll down a little, you can see which threads have problems, such as Figure 7
Pitfalls: If you analyze the results of the "Thread CPU Usage Summary" is not in the analysis results, and you are sure that this .dmp file must be eating a lot of CPU, then you are likely to be the problem of the .dmp file, because I use 64-bit manager to create 32-bit process pools created by the dump file . dmp, visual studio can't analyze this "Thread CPU Usage Summary".
Then according to the above lines of code to take a closer look at the code of this page, good guy, this is a web site to deal with the problematic orders of the page, when the initial development of the customer service mode is to look at all the orders shipped overtime, and then notify the shipper to ship in a timely manner, and now because of overtime orders too much too much (every year during the peak season are this way), this page needs to be Now, because there are too many timeout orders (this is what happens every year at peak times), the page needs to display very, very much content in one breath, which is output using a control, and then the code is written in such a way that splicing the content is += xxxxx, so the page is a perfect computationally intensive thread, which is accessed by just a few shippers, and then it is just a GG. Assuming that this page is just a few more visits to the database, or query data for a longer period of time this IO-type calculations, the delayed response, but also not to kill so much (at best, the people who visit this page to wait a little longer), the key to the bottleneck of it Ya is splicing strings 。。。。。
Subsequent optimization is simple, asked the customer service, according to the logic of the current work of the customer service, directly do not show all the timeout (customer service is not concerned about the timeout, each shipper's own attention to their own, the shipper's own problem orders are not many, and then += this writing is also very dilettantish, changed to a StringBuilder object to splice up.
So far, the problem is solved, this problem has cost me almost two days of time, the biggest pit is the creation of the .dmp file as mentioned earlier, if the beginning of the creation of the right .dmp file, and then analyze it directly with Visual Studio, it is estimated that the time to find the problem as long as 1 hour. However, programmers in the problem solving, stepping on the pit is also a kind of learning, stepping on the pit enough, the more experience accumulated, the next time you encounter a problem the faster the solution, for stepping on the pit, I'm not chagrined at all.