LINUX standard test design process
Level: primary
Robert Williamson
Robbiew@us.ibm.com)
Software Engineer, Linux Technology Center, IBM
Automatic software test allows you to run the same test in a period of time, ensuring that the content you compare has truly comparability. In this article, members of the Linux Test Project team share the methods, principles, and scripts and tools for the tests used by the Linux® kernel.
In the test of the stability of Linux kernel, it is necessary to clearly declare and prove why versions are stable or unstable. However, it has not been proven to be proven to test the pressure test within the currently existing system that can test the overall stability of the Linux core. This article gives a method of creating a system-wide Linux pressure test and proves the correctness of its results. Different Linux developers, users and distributions use their own way to test the stability of the kernel. However, there is no release of the basic information that they decided to run which tests, covered code, and the pressure level, etc., which have not been released, which greatly reduces the value of the results.
Using the laboratory machine and test from the Linux Test Project test kit, we develop a test combination based on system resources, and provide sufficient pressure to the system. We analyzed this combination test to determine which parts of the Linux kernel were used in test execution. Then, we modified a combination test that increased the percentage of code coverage while maintaining the desired high-intensity system. The final resulting pressure test covers a sufficient number of Linux kernels, which helps stability declaration, and system usage and data over-core data to support it.
The four steps of this combined test method are: test selection, system resource utilization evaluation, kernel code coverage analysis, and final pressure test evaluation.
Select test
Test selection includes testing tests to reach two perspectives:
- Test should be able to get a high level of resource utilization in major kernel regions such as CPU (S), memory, I / O and networks.
- Test should fully cover the kernel code to help support the stability declarations generated from their results.
As long as it is possible, it is possible to use automated or easy to modify the test to support auto operation. Automatic operations can make testing faster and repeatedly, and help reduce human errors. Another aspect that needs to be considered when choosing a suitable test is that the application that can be freely published the result. It is best to choose to resolutely support the open source method and / or GPL test and test suite to help ensure the purity of the release process.
Evaluation system resource utilization
The combination of the selected test must bring enough pressure to the resource of the system. The four main aspects of the Linux kernel can affect the system's response and execution time:
- CPU: Time for processing data on the CPU (S) of the machine.
- Memory: The time used to read and write data from the real memory.
- I / O: The time used to read and write data from the magnetic disk memory.
- Networking: The time used to read and write data from the network.
Test designers should use the two famous and widely applying open source Linux resource monitoring tools to evaluate the level of resource utilization. (See the References later in this article to get links to these tools.)
- Top: An open source tool maintained by Albert D. Cahalan, which is included in most Linux releases, available for current 2.4 and 2.6 kernels.
- SAR: Another open source tool; it is maintained by Sebastien Godard. This tool is also included in most Linux release versions available for current 2.4 and 2.6 kernels.
The system resource utilization evaluation phase in the method usually requires multiple attempts to obtain a suitable test combination and obtain the utilization of the desired level. Over-utilization is always a crucial problem when it is determined. For example, if the selected combination is too limited by I / O, it may cause the CPU's test results, and vice versa. This part of the method is mainly a large number of experiments and error until all resources achieve the expected level. TOP tools can be used to quickly determine which resource (CPU, memory, or I / O) affects each test, and shows how much resources they use in real time. The SAR tool is used to collect network utilization statistics within a period of time and record all snapshots of all utilization data to a file.
When a combination is selected, the test must run for a long time to accurately evaluate the utilization of resources. The test run is long depends on the length of each test. If multiple tests are running simultaneously, the time must be longer so that the longest one in these tests can be completed. During this evaluation process, SAR tools should also be running. In the conclusion of evaluating operation, you should collected and evaluate the utilization level of all four resources.
The following example shows the CPU, memory, and network utilization of SAR output:
Listing 1. Sample of SAR output
10:48:27 CPU% User% Nice% System% iowait% iDLE 10:48:28 all 0.00 0.00 0.00 0.00 100.00 10:48:29 ALL 3.00 0.00 1.00 0.00 96.00 10:48:30 All 100.00 0.00 0.00 0.00 0.00 10 : 48: 31 all 100.00 0.00 0.00 0.00 0.00 02:27:31 kbmemfree kbmemused% memused kbswpfree kbswpused% swpused 02:29:31 200948 53228 20.94 530104 0 0.00 02:31:31 199136 55040 21.65 530104 0 0.00 02:33:31 198824 55352 21.00 02:35:31 199200 54976 21.63 530104 0 0.00 02:27:31 iFace RXPCK / S TXPCK / S RXBYT / S TXBYT / S 02:29:31 Eth0 738.79 741.66 76025.55 136941.85 02:31:31 Eth0 743.30 744.97 76038.82 136907.77 02:33:31 ETH0 744.80 745.02 76135.53 136901.38 02:35:31 Eth0 742.3 5 744.34 75947.45 136864.77
Analysis of kernel code coverage
Get enough kernel coverage is another responsibility for system pressure testing. Although the selected test combination fully utilizes four major resources, it is also possible to just perform a small part of the kernel. Thus, you should analyze the coverage to ensure that the combination can be a system pressure test, not a system load generator. Currently, two open source tools can help the Linux kernel code coverage analysis: - GCOV: A open source tool for maintenance by Linux Test Project. This tool analyzes the coverage of kernel code and report which rows, functions, and branches are overwritten and how many times they have been accessed.
- LCOV: An Open Source Code Tool developed by IBM, maintained by Linux Test Project. This tool consists of a set of Perl scripts that are built on the text-based GCOV output to implement HTML-based output. Output includes a percentage of coverage, chart, and an overview page that quickly browsses overlay data. You can find these two tools from the Linux Test Project (LTP) home page (see Resources to get link).
After the GCOV module is loaded, all tests in the system pressure test combination must be executed. Although the original system pressure test can be executed simultaneously, it should be executed at the same time, but this run should be looped. Each test should run once until the end, run one by one, cannot run any tests again. Single, looping, is to reduce the unpredictable and non-purpose kernel code execution caused by balancing their loads when running multiple system pressure tests while running multiple system pressure. You should perform GCOV analysis after the last test run. Since it is ultimately to format data for analysis, run the LCOV tool and carry the GCOV module.
The LCOV tool generates a complete HTML tree, which contains each row of the code in the core and how many times the number of times per line is performed (if any). This tool quantifies coverage data and generates a percentage number about each part of the kernel and a file coverage. The following example shows an exemplary code coverage output:
Figure 1. GCOV output example
The LCOV maintainer defines "sufficient coverage" (green), so this LCOV example is just an evaluation. However, the original data included makes any browsing can make him or her own judgment. After browsing the coverage analysis, the test creator can now modify the combination of tests to change and / or increase the amount of code covered.
Evaluation final pressure test
The last step to perform the method is to verify the system pressure test. In a kernel that is considered to be a stable, a pressure test is performed; in general, the kernel in the release can meet this requirement, but not always. To perform a pressure test for a long time (recommended for at least 24 hours), run the SAR tool at the same time, the following two points:
- Long-term operation helps to find all the problems in the combination, otherwise, these issues in a short time "Sniff Test" may be ignored.
- The data generated by SAR is composed of the baseline for comparison in the test run.
After running over a long run, you can now determine if all data collected is whether this test combination is a suitable candidate for system pressure testing.
Figure 2. Summary of design process
This design method is used when designing Linux kernel pressure test scripts ltpstress.sh. This application combines multiple tests and memory and network transport load generators from different aspects of the LTP test suite. Before execution, the test will adjust its total memory usage based on how much physical and virtual memory exists in the system. This test script can be obtained from the LTP test suite (see Resources). To ensure the accuracy of the results, this script is created under controlled laboratory conditions. The IBM Linux Technology Center Test department uses this pressure test and other tools and tests as a relatively fast and easy way to confirm the stability of Linux kernel release. To help ensure a sufficient coverage, the test is carried out under the mandatory user in the laboratory conditions.
Reference
- in
Linux Test Project Home Download Pressure Test Shell script and other practical tests.
-
The mission of IBM Linux Technology Center is to cooperate with Linux development community to jointly promote the success of Linux.
- OSDL
Linux Kernel Scalable Test Platform (STP) provides a framework that can test kernel patches by using it using an online performance and scalable kit.
- Tools used by LTP pressure test include TOP (
Part of the Procps package) and SAR
Part of SYSTAT).
- In addition, the LTP pressure test has also taken advantage of procedures for GNU test coverage.
GCOV and its PERL-based tool (HTMLIZER) LCOV that converts GCOV results to HTML.
-
Core comparison: Improvements in 2.4 to 2.6 core development (DEVELOPERWORKS, February 2004) introduced better tools, tests, and techniques for any kernels than previous.
-
Nucleuclear comparison: Web services on 2.4 and 2.6 (DeveloperWorks, February 2004) describe the results of Web service testing of IBM Linux Technology Center.
- in
Improve Linux kernel performance and scalability (DeveloperWorks, Jan 2003), Linux Technology team discussed how to quantify Linux performance to compare test results for a period of time.
-
Test Linux reliability (DevelOperWorks, December 2003) provides file information for the test results and analysis of the Linux kernel and other core OS components for IBM Linux Technology Center.
-
The Linux kernel debugger insider (United States, June 2003) shows you how to track the execution of the kernel and check its memory and data structure.
- in
The developerWorks Linux zone can find more reference materials for Linux developers.
- Buy developer bookstore Linux area
Discount Linux books.
- pass
DeveloperWorks Subscription uses the latest IBM tools and middleware to develop and test your Linux application: You can get IBM software from WebSphere?, DB2?, Lotus?, Rational? and Tivoli to get IBM software, and one can be used within 12 months All licenses of these software, all costs are lower than what you imagine. - Self-developerWorks
For your Linux app Download no runs on Linux through selected developerWorks Subscription products free test version, including WebSphere Studio Site Developer, WebSphere SDK for Web services, WebSphere Application Server, DB2 Universal Database Personal Developers Edition, Tivoli Access Manager and Lotus Domino Server. To get started faster, see how-to articles and technical support for each product.
About author
Robbie Williamson is a full-time software engineer for IBM Linux Technology Center. He graduated from Texas State University in 2000, a bachelor's degree in computer science. In his career, he used technical support staff, verify engineers, and developers realized by various UNIX. Robbie is currently one of the maintaors of Linux Test Project,
Robbiew@us.ibm.com Contact him.
Note: This article only represents the author's point of view, does not represent the point of view of IBM. The foundation of the results discussed in this article is a solution that created and tested in a laboratory environment. These findings may not be obtained in all user environments, and additional steps, configuration, and performance analysis are required in those environments. It is given here that the original stunned information is not guaranteed, described or implied. This information is not an explanation or component of any IBM product guarantee. The implementation and proof of the solution is responsible for the implementation team.
Full article:
IBM DeveloperWorks China website
