Debug distributed web application (3)

zhaozj2021-02-08 238

Frequently asked questions during debugging

The most common issues in distributed web applications are:

ASP 0115 - Access conflict ASP server is too busy to 100% CPU problem

In this section, we will analyze the possible causes of these issues, discuss how to debug and identify code rows that cause each problem.

ASP 0115 - Access Conflict

Access conflicts are mainly caused by stacks and stack failures.

If you do some operations, if you assign a memory and try to perform read / write operations outside the memory block, you will cause access to your conflict.

E.g:

Char * psztemp = (char *) Malloc (Strlen ("Bomb this");

STRCPY (PSzTemp, "Bomb this");

In this example, the user assigns 9 bytes of memory, but does not contain NULL endors ("/ 0"). When you try to copy a string to this memory location, an error is generated because the function attempts to access the part other than the assigned memory segment.

Another scheme attempts to access an illegal memory location.

E.g:

Functiona ()

{

m_PAV = NULL;

FunctionB (m_PAV);

}

FunctionB (CACCESSVIOLATE * PAV)

{

PAV-> BombThis ();

}

In this example, FunctionB does not check its legitimacy using the PAV pointer passed to it, resulting in access conflicts.

Most access conflicts occurs when the ASP page is trying to access a component, and this component is trying to make mistakes. For simplicity, we assume that you have installed a component to the library package. When you try to access an ASP page using this component, the ASP page may return the following information:

ERROR 'ASP 0115'UNexpected Error / / .aspa trappable error Occurred in An External Object. The script cannot Continue Running.

This error message indicates that there is an error in a particular ASP page, which is caused by an exception in an external object. It indicates that you have tried to access a failed component in the ASP page. You can further isolate questions, remove all references to any components in the ASP page, and then add them one by one until you find the cause of the failure.

If the isolated process is too tedious, or you did not get a clear error message indicating an ASP page, you can connect Windbg, User Mode Process Dump, or Iis Exception Monitor, and start debugging.

Find out the code line that causes access to conflict

Suppose you have connected to the InetInfo process using the USER Mode Process Dump tool and created a dump file. Now start to check the dump file in Windbg, and try to sketch what happened.

Open a failed dump file in Windbg, in the View menu, click Call Stack.

Figure 4. Windows Debugger dialog box showing the stack

In the above figure, even if the top function in the stack is a function in the Kernel32 DLL, this does not necessarily mean that the error is in this function, and most of the cases are not the case. The next step is to determine if there is a non-standard Windows system component in the list and determine what they are doing when there is a fault. If you read the stack in the figure from the down to the chart, you will find that the 8th and 7th lines indicate that the ASP uses the Microsoft VBScript engine to launch the components, line 6 and 5th lines indicate that the VBScript engine uses the IDispatch interface method to call Methods inside the CraShatlapp). Because the IDispatch interface is used, Oleaut32.dll appears naturally in the figure. The fourth lines and third lines indicate this. The second line in the stack indicates that the blowoff method is called, and the Blowoff calls the OutputDebugstring API in the Kernel32 DLL. Assume that the standard Windows component Kernel32 DLL, OleAut32.dll, and the VBScript engine have no problem (from the perspective of problem isolation), this is a secure assumption. Next, only the unique non-Windows components listed in the remaining stack are listed, CraShatlapp. The OutputDebugstring API is called in the Blown, where the CraShatlapp is called. By entering KB in the command window, we can check the value of the block of Blowoff to OutputDebugString.

Inside the Kernel32 DLL, the Blowoff method passes NULL as a parameter to the OutputDebugstring method. It is best to check what action has been carried out inside the blowoff method so that it passes a null value. If you have the source code for the CraShatlapp component, you can double-click on the second line in the stack.

Figure 5. Windows Debugger dialog box showing the stack

This will call the source code of the blowoff method.

Figure 6. Windows Debugger dialog box showing the source code for the blowoff method

The source code clearly shows that NULL is passed as a parameter to OutputDebugString.

Although this problem will not be encountered in the product code, the intent of this example is that there is a concept that makes you step on the procedure involved in debugging and determining the cause of a conflict error.

ASP server is too busy

The ASP server is too busy. It is usually caused by the ASP request queue. Very large queue can be generated when the IIS web server is high load or waited for a blocking thread. This usually occurs when the frequency of the IIS web server is waiting for a frequency exceeding the frequency that can be affected by the component. All access requests are placed in the ASP Request parallel and processed in the previous order they enter. If the blocking is only a short period of time, the queue mechanism will eventually solve the problem and start a timely response request. However, if the blocking is long, the request in the queue will continue to increase, and finally reach the peak in the AspRequestQueueMAX metadata setting, the default value is 3000. If the queue size is below 3000, when the 2999 request is in the queue, and wait for all other requests to be filled, the requested user will see an hourglass sign. Even at this time, if the queue is clear enough, this request will still be received and processed. Conversely, if the queue size reaches the value of AspRequestQueuemax, there is a new request to add, then the IIS server returns the ASP server too busy.

AspProcessorthReadmax and AspRequestQueuemax settings

AspProcessorthReadmax and AspRequestQueueMAX metadata settings have a great impact on the performance of the site. The AspProcessorthReadmax parameter specifies the maximum number of threads that can be affected by the processor. The AspRequestQueuemax parameter specifies the maximum value of the request queue. These two settings can be changed by the following command: adsutil.vbs set w3svc / askPROCESSORTHREADMAX NEWVALUE

Adsutil.vbs Set W3SVC / AspRequestqueueum NewValue

The aspprocessorthreadmax value should be low for write a good script. The main purpose of adjusting AspprocessorthReadmax is to make the processor's utilization in maximum load reaches 50%.

Since the physical configuration of each site is very different, it is difficult to predict the optimal value of these parameters. Performance monitors should be used to monitor critical counters to make a decision on these parameters. Statistics using Performance Monitor should include:

Processor Time:% Processor Time

ASP: Requests Per Second, Requests Rejected, Requests Queued

You can monitor these values when a product site reaches the maximum load. Typically, you will observe the start of the number of requests in the queue at high-capacity sites. If the utilization rate of the processor is very low, the total queue length does not rise, indicating that your site is much greater than your actual needs. If the total queue is up and down, the processor utilization is below 50%, then some requests are blocked, and you will benefit from the increase of the number of threads. If the number of requests in the queue is observed after increasing the thread is worse than before, and the processor utilization is reduced, this means that there is a serious blocking problem. A typical blocking problem may be caused by a SQL query that needs to perform a long time. Increasing the number of threads allows for more requests to execute the page of the query; this will cause greater pressure to make the database response time longer. To get an optimal value of AspRequestQueuemax, select an acceptable longest response time, calculate the size of the queue that can be emptied during this time, then set the value of AspRequestQueuemax to this size.

Debug example

We have discussed the asprocessorthreadmax parameters and AspRequestqueuemax parameters in detail, follow we will simulate the ASP server too busy, and try to debug the thread of which component occurs, and why occlusion. Assume that there is an ASP page, and this page is calling a blocking component.

To simulate this problem, we set the AspRequestQueuemax parameter to 20 by the following command:

Cscript c: /inetpub/adminscripts/adsutil.vbs set w3svc / askREQUESTQUEMAX 20

To initiate a performance monitor, perform the following steps:

Click Start, point to Programs, point to Adminstrative Tools, and then click Administrative. Right-click on the chart, select Add Counters. In the Add Counters dialog, select Active Server Pages from the Performance Object list box. In the Select Counters from list, select Requests Executing and Requests Queued. Click Add and click Close.

Assume that a component is called in your ASP code, and the component is blocking the request.

After adding these counters, the value of the counter is observed. You will find that when the request in the queue reaches the value (20) of the AspreQuestQueuemax, the server returns too busy error. Figure 7. Performance Monitor for Request Counters in Show Queues

To debug the INetInfo process to determine which thread occurs and why occlusion occurs, start WindBG and connect to the InetInfo process. The most common reason that causes thread blocking is dead lock. You should try to identify the key part of the lock count greater than 0, and the thread ID owned by this section. This thread is the problem. Once the thread ID is obtained, the stack of the thread can be debugged, which is found to be blocked. You can do this in the command window to complete this step:

! LOCKS

At this point, the unlocking count is greater than zero. The output will be similar to the following:

Critsec? G_cs @@ 3u_rtl_critical_section @@ a 0 at 100355a0

Lockcount 24

Recursioncount 1

Owningthread 6e8

EntryCount 18

ContentionCount 18

*** locked

Find the part of the lock count greater than zero and pay attention to the thread ID it owns.

Sometimes! Locks command does not provide the information you need, which is usually because NTDLL.DLL's symbol arrangement is incorrect. In this case, you can use the following comparison of the original but effective method.

In the command window, type ~ * KB to display the stack of all threads. Find the thread stack that contains NTDLL! RTLENTERCRITICALSECTION. After you identify the thread containing this item, note the value of the param1 in the line where NTDLL! RTLENTERCRITICALSECTION is located. This value is the object that is sent to the EntercriticalSECTION API. Use this value, enter the following command in the Command window:

Critsec 100355A0

With any of the above methods, it should be able to identify the thread ID of the locked count greater than 0. Convert this thread ID from hexadecimal to decimal.

Please check this thread now and identify the method of blocking the thread. In the Debug menu, click Threads.

In the Threads dialog box, click the thread ID to be found in the above steps, click Select.

In the View menu, click Call Stack. The stack of this thread is displayed at this time. Just as starting to debug the stack as in the previous example (access conflict).

100% CPU problem

When a thread takes up all the time of the processor, a 100% CPU problem occurs. One possible reason is that a program loop exclusively CPU resources. Performance Monitor is an ideal tool that solves 100% CPU usage problem. First, you need to create a performance monitor log.

Preparation log

To initiate a performance monitor, perform the following steps:

Click the Start menu, point to the program, point to Administrative Tools, and then click Performance Monitor. Right-click COUNTER LOGS, and then click New Log Settings. Enter a log name and click OK. On the General tab, click Add to add the counter to the log. Add the following counters:

ASP: All Counters Memory: All Counters Process: All counters and all instances Processor: All counters and all instances Thread: All counters and all instances Web Service: All counters and all instances After adding all counters, change Sampling Interval to Every 1 SECOND, then click OK. Now all content has been set, you can start logging data. Right-click this log name, then click Start. Make sure the log captures all the data that has increased to 100% of the processor utilization. Start WindBG and connect to the InetInfo process. Once the processor utilization is returned to the normal level, you can stop the record, the method is to right-click the log name, and then click STOP.

Check log

The log is now created, we can start carefully checking it to identify the process that consumes most of the CPU time, and further identifying threads that deplete CPU time inside the process, and the reason why it does.

The first step is to identify which process consumes most of the CPU time. To open a log file, click the System Monitor folder under the root directory. In the Details pane, right-click on the chart and click Properties. In System Monitor Properties, click the Source tab. Select Log File and select the log file you just created by browsing.

Right-click on the chart, and then click Add Counters. In the Add Counters dialog, select Process from the Performance Object list. Select Select Counters from List and select% Processor Time from this list box. Select All Instances, click Add, and then click Close.

This will provide us with a list of all processors and their CPU usage. Now you need to find out which process has caused 100% CPU issues.

Make sure the Highlight tool button (with a spherical icon) is selected. Now you can scroll through these counters, all deleted all processes other than the process of the closest 100% tag. Make sure that only those counters with you are selected. For example, if you are commissioning 100% of the CPU utilization problem, you may find INetInfo, other offline processed COM package / applications (MTX.exe / DLLHOST.exe) or IIS applications.

Once you are isolated from the process level, you can focus on the thread that is still active within the process within the process, and tries to isolate the thread that causes problems. To do this, we add more counters to display the CPU usage of all threads in the process. For this example, the counter is added to all threads in the INetInfo process. Right-click on the chart, and then click Add Counters. In the Add Counters dialog, select Thread from the Performance Object list. Select SELECT Counters from List and select% Processor Time from the list box. Select SELECT Instances from List and select all instances starting with inetinfo / *. Click Add and click Close.

At this time, the CPU usage of all threads in the INetInfo process is displayed in the chart.

Now you can scroll the counter, which will be removed all threads other than the thread that is closest to 100% tag. This way we accurately find that thread that caused problems. Figure 8. Performance Monitor Displaying the Counter of the CPU Usage

We only retain the thread of the instance ID of 34, which takes up the longest CPU time. You must now find this thread ID and then find this thread in Windbg and debug it.

To find a thread ID, right-click on the chart, and then click Add Counters. In the Add Counters dialog, select Thread from the Performance Object list. Select Select Counters from List and select ID Thread from this list box. Select SELECT Instances from List and select Instance INetInfo / 34. Click Add and click Close.

In this way, we get the ID of the thread that caused problems. Select the ID THREAD counter. Record the last value, maximum, minimum value, and average value of the counter. Usually these four numbers are the same. Record this value (764 in this example) and switch to the WindBG application.

Figure 9. Performance dialog box display ID thread counter

On the Windbg's View menu, click Threads. This will display a list of threads running within the INetInfo process when a 100% CPU issue occurs. Find the thread ID number you record in the Performance Monitor. If you find this ID number in the list, click it and click Select. Click OK.

On the View menu, click Call Stack. At this time, the stack of the thread that causes problems. This stack isolates problems. Once we find this method, we can return to the original code, see which row code causing this problem.

in conclusion

By using the IIS Exception Monitor, Windbg, User Mode Process Dump Tools, and Performance Monitor, most of the common problems in the distributed web application can be effectively resolved. These tools are of course no problem with all answered and solutions, but they can point out a direction for you finally resolve problems.

Reference

Method: Debugging the high CPU utilization in Internet Information Services (English)

Information: Debugging the abnormality in the Internet Server product (English)

9cbs

New Post(0)