Two types of classic software test methods before describing Microsoft's software testing methods, I would like to understand what is software testing from the concept, or the idea level, and the purpose is to export the theoretical root cause of Microsoft test methods. Traditionally, the method of software testing is considered to be two categories. The first type of test method is to try to verify the software "work", so-called "work" means that the software function is implemented in advance; the second type of test method is to try to prove software is "not working" . Representatives of the first type of approach are the pioneer Dr. Bill Hetzel in the field of software testing ("The Complete Guide to Software Testing), he organized the first in the United States in Northern Carolina University in the United States in June 1972. Subject to the forum for software testing. He first tests a such definition in 1973: "It is to establish a confidence that the program can operate on the expected idea." That IT IS supposed to do. "Later in 1983 he will Definition revision is: "Evaluate the characteristics or capabilities of a program and system and determine if it reaches the expected result. Software test is any behavior of this. Any Activities AIMED AT EVALUATING An Attribute or Capability of a Program OR System. "In his definition" ideas "and" expected results "are actually the user needs or functional design we are now mentioned. He also defined the quality of the software as "conformity". The first type of test can simply describe such a process: running the software in the design specified environment, comparing its results with user needs or design results, if the test is passed, if not, it is considered to be bugs. . The ultimate goal of this process is to run all of the functions of the software in all design specified, and pass. The first type of method is generally mainstream and industry standards in the software industry. The IEEE / ANSI standard in 1990 made such a definition of software testing: "Under the established condition conditions, run a system or formation, observe the results, and evaluate some aspects .the process of operation a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component (IEEE / ANSI, 1990 [Std 610.12-1990] "here, the term" predetermined condition "is also understood to Demand or design. Despite this, this method is still questionable and challenged by many autonomous authority. Representatives are Glenford J. MYERS (representative "The Art of Software Testing"). He believes that test should not focus on verification software is working In the opposite should first determine that the software is wrong, then discovers as much error as possible. He also argued as possible from the perspective of human psychology, "verifying software is working" as the purpose of testing, very bad for testers Find the software's error. So he proposed his definition of software testing in 1979: "It is the process of running the program with the purpose of discovering the error .The process of executing a program or system with the intent of findinger." This is The second type of software test is simply that verification software is "not working", or there is a mistake. He even extremely believed that a successful test must be a test of Bug, otherwise it is worthless.
This is like a patient (assuming that this person is not sick), go to the hospital to do a medical examination, and the various indicators are normal, that the medical examination is not worth the patient's condition, it is failed. I don't fully agree with this view. The second type of software test method is also very popular in the industry and is supported by many academic experts. Everyone is familiar with Ron Patton in "Software Test" (Chinese version is published by Machinery Industry Press, "said this book is a clear and simple definition:" The software tester is to find software defects, as early as possible, and ensure that it is repaired. "Some software companies use bugs as an indicator for assessment test personnel, in fact, this is a method. The advantages and disadvantageous comparison of the two types of methods Although the total purpose of software testing is for the quality of software products, it is obvious that these two types of test methods are trivial, or the instead of guiding ideology. This also determined that they had a lot of differences in ideas, process and priorities, and their aphasic. The first type of test method is based on demand and design, so it is conducive to defining the category of testing, and facilitates the side focus of deployment tests, and strengthens targeted. This is especially important for large software testing, especially in limited time and human resources. The second type of test method is not necessarily related to demand and design. If the planned management is not properly managed, the test activity is easy to lose its focus, and go to astray. The first type of test method can cooperate with the planning of the software architecture and software development, so that software test activities will be launched one by one, so that software function and quality have planned gradually improve and improve (on the level of testing, I It will be specifically introduced in the future discussion). The second type of test method does not have the graduality of this process. The disadvantage of the first type of test method is the lack of flexibility, which is not conducive to the subjective initiative of the test personnel. It is like Mr. MYERS, which is not easy to find the error (bug). And this is the strength of the second type of test method. Microsoft's strategy is because of the various types of test methods, Microsoft combines two types of methods in software testing activities, based on the first type of test method and the main clues, and use second types of test methods. . Microsoft's first type of test Microsoft's first type of test is generally divided into three steps: audit requirements and design -> Design test -> Implement operation test. The foregoing test is the correctness of the software to verify the software with demand and design. Everyone is naturally, the demand and design itself also has the right problem. The test will also be futile based on incorrect demand and design. Therefore, verification requirements and design are the first step in Microsoft for the first type of test. It is necessary to point out that the needs and design described herein generally include: (1) demand text written by the project manager according to user requirements (information from market sectors, user support, etc.) (Requirement Specification ); (2) Functional Design Specification, which is written by the project manager; (3) Implementation Design Specification, which is written by the developer according to the function text. Microsoft's testers should participate in all of these text audits. As a tester, the audit focus is the detection of the integrity, rigid and functional design of the text on user needs. At the same time, this review is a warm-up activity for testers to enable them to enter technology and business. In the second step, the tester should design test cases based on the review and design of the reviewed demand and design.
In the three text mentioned earlier, the functional design text is the main basis. The reason is very simple, this type of test is concerned about whether the software can implement the function correctly, not how these functions are implemented. From here you can see this is a typical "black box test." It is true that Microsoft's test is mainly black boxes from the perspective of the user. The completion of this step means the completion of the "test plan" and "test case design". The "Test Plan" text mainly describes the category, field, method, tool, resource and plan schedule, etc. of the test. "Test Case Design" text To list test cases, the settings, execution steps, and expected results of each use case. These two texts tested should also be reviewed by project managers and developers. In this way, through various mutual audits, everyone has formed a basic consensus on the project. The implementation of the operation of the third step is a longest and most complicated phase throughout the development process. Overall, the process of testing the previous design is implemented in accordance with the process of implementation. This includes writing an automated test program, repeatedly running the automated test programs, and also includes a phased execution manual test case. This phase of the test must be carried out under the careful plan, I have mentioned in front, this is the characteristics and strengths of the first type of test. This plan is first reflected in the development and testing of mutual coordination, according to the relying on the structure of the product, according to the overall plan of the project, according to the overall plan of the project. From the process of testing, you always run or perform simple use cases, then repeat the use case; first verify a single basic function, then the end-to-end function; first find the solution to the surface, the influential BUG, and then deep It is not easy to reproduce the bug. Therefore, with the process of project development and testing, the function of the product is constantly improving, and the quality is constantly increasing. There is a little more specifically that there are many test cases to be repeated, especially basic automation tests, each build is running. Although most of these tests are passed, rarely discover new bugs, but their value is obvious, in order to prevent quality regression. It can be seen that MYERS is not applicable here. At this stage, the tester has a cumbersome but very important job, which is the maintenance of existing test cases. For example, in the following two cases, some test use cases are added. First, for the field of test design, the second is that the external bug (such as report from the Beta customer) is not covered by the existing test example. When the function design of the product changes (in Microsoft this is the common thing), the test case involved is of course modified accordingly. Microsoft's second type of test Microsoft's second type of test is phased, often with randomness and assault as needed. For such tests, there is a special name in Microsoft: "BUG Bash (BSH Sweeping)". BUG Bash usually happens at the end of the project development (Microsoft Milestone), such as the beta release, draw a special time period (usually 1-3 days), all participating in the project during this period, concentrated all energy To use all aspects of knowledge, do full wisdom to search the project's bug. This is a very interesting event, but it is not easy to organize such an activity. Generally there are the following points: (1) Although this is a test activity, participants are not limited to testers. Project managers, developers even managers should participate, as they mobilize them. The purpose is to brainstorm; (2) to encourage departments, field cross search, because new ideas and perspectives usually help find more bugs; (3) To mobilize enthusiasm, enhance fun, can introduce competition mechanisms, For example, when the activity is over, the discovery bug is discovered, and the most serious bug is discovered, gives substances and spiritual rewards.
(4) You can expand, such as security, user interface availability, internationalization, and localization. Microsoft's second type of test In addition to BUG Bash, there are often some professional tests, most typical is for security attacks. Generally, the company is invited to search for the company to search for the safety vulnerability of the product. The above, from the perspective of traditional software test concept, I introduced Microsoft's strategy and the specific practices of two types of traditional test methods, and their focus. This is actually just a foundation, a very original basis. Software Tests The role of Microsoft software product development, is far from these original methods, nor is it a letter of traditional software test concepts. Microsoft has a lot of unique practices in software testing, and conceptual breakthroughs, such as "software test information service function", "user-centered macro quality system", "grading test", "quality management system" , "BUG trial", "test automation" and "software test hardware - sectors, teams, people and infrastructure", etc. These I will introduce them in the next discussion.
Historical review In order to better understand microsystem testing in methods and ideas, I want to review the development history of software development and software testing, and reveal their inevitability. Edward Kit In his bestseller "Software Testing in The Real World: Improving The Process (1995, ISBN: 0201877562)" The entire software development history is divided into three phases: the first stage is the 1960s and its previous, then When the size of the software is small, the complexity is low, the process of software development is free. The developer's Debug process is considered a unique test activity. In fact, this is not a software test in the modern sense. Of course, there is no special tester in a stage. The second stage is the 1970s, and the software developed at this stage is still not complicated, but people have begun to think about the development process, and propose the concept of "software engineering software engineering". However, this phase of people's understanding of software tests is limited to basic functional verification and bug search, and the test activity only appears in the later stage of the entire software development process, although testing is undertaken by specialized testers, but testers are industries and Software professional entrance newcomers. The third stage is in the 1980s, and the software and IT industry have entered the great development. The software tends to be large. In contrast, people have designed a variety of complex and precise processes and management methods for software development, and integrate the concept of "quality" into it. Software Test has an industry standard (IEEE / ANSI), it is not a one-time, but only the post-development activities, but integrates with the entire development process. Software tests have become a professional, need to use specialized methods and means, special talents and experts need to be borne. The integration of testing and development is most worth noting in this historical development process. The trend of testing and development process fusion. People may not be unfamiliar with this integration. For example, early development of test activities, allows testers to participate in the verification of user needs, participate in functional design and implementation review. For example, testers work closely with developers, with gradual implementation of unit testing, module function testing and system integration test with development progress. It is indeed the form of expression and development fusion, and the initial fusion is only reflected in this level. After the 1990s, the size and complexity of software increased rapidly, and this form of integration also quickly went to a deeper level, more practical. Specifically, this fusion is the dependence of the entire software development activity. Traditionally, only software quality control depends on testing, but the practice of modern software development proves that not only software quality control is dependent on testing, and the development itself will not be able to advance. Project management has left the test. in accordance with. In Microsoft, the test does have such status and role. That's why Microsoft has such a large investment in the software test. Developing modern software development for testing, especially large software development, usually encounters two issues: (1) In the initial stage of development, how to launch a large-scale team, the group is in turn, while maintaining the order of development. Thus efficiently utilize resources and shorten the development cycle. (2) In the late development, how to solve deep BUGs, how to face design changes, and ensure that the quality of the product does not appear or falls. For small and simple software, these two problems also exist, but do not protrude, and easy to solve. But for complex large software development, these two problems often become difficult to overcome. Usually the function of large projects is rich, but the architecture, and the level will be quite complicated. Sustained development method is that a small number of people have been investing in one time.
However, this way is clear that the resource utilization is low, the development cycle is long, and the modern software and the high-speed development of the IT industry are unable to develop. So large projects require large teams. In Microsoft, product development teams (mainly including development, testing and project management) generally have more than 100 people, some products, even thousands of people (more than 3,000 of Windows2000 development departments). Such a large-scale human resource acts in a dynamic, internal interconnected system, if there is no effective synergy, its confusion is inevitable. Imagine that there are two developers, which are developed two different functional modules, which are dependent on each other. In order to coordinate each other, they can discuss all at any time. If this relationship occurs between five developers and five functional modules, this coordination can only be done through a regular meeting. And a large project will have a lot of such relationships, and many times this relationship has uncertainty and unpredictability. When a developer writes a new code or modifies the existing code, he often does not determine, or cannot fully determine which related modules will be affected, and what should be What results will be brought about. Because the complexity of the system is far exceeding the ability of people's logical thinking, skills, and experience. Therefore, this traditional coordination means is far from needing. In Microsoft, this coordination is achieved by testing. Specifically: daily construction automation test. With regard to the daily compilation and automation test, I will introduce it in the future. It is simple to build a new version every day. Each version is running through a certain automatic test case to test the quality of work on the day. The quality mentioned here is of course a concept of quality in general, but it also reflects the overall coordination of the project during the development process. The maximum advantage of automatic test is its height repeatability. An ideal automatic test system can operate a lot of test cases at any time, convenient and quickly run. Therefore, a developer can analyze the quality of the code before the day of automatic test results (afterward check), or before the time is deposited in the day, run automatic testing to further ensure the quality of the code (prior inspection). In Microsoft, every day construction is starting at midnight, and then the completion is a comprehensive automatic test, it will automatically send the results automatically through E-mail before morning. The first thing to develop after work is often checking the test results. If there is no problem, you will start a new job. If there is a test use case, the developer must cooperate with the test personnel to find out the cause and solve the new code. Sometimes a small mistake will cause a large-scale test case failure, a large part of the development team will be affected. To avoid this, the developer is required to run a certain amount of automatic test on his personal build version before depositing the code, all of which are passed after depositing. If the developer does not follow such a request, the unauthorized test fails, this irresponsible behavior is to be severely criticized. As can be seen from this process, developers depend on tests to ensure the quality of development work, so that the development is integrally coordinated forward. When the development enters the later stage, although the project has been overall, developers will encounter some technical challenges from time to time. For example, some bug addresses the adjustment of the deep-hierarchical structure of the project; for example, the design is modified due to the opinion of customer feedback. Every such modification and adjustment is actually destroying a stable system. If the improper handling is often a bug modification, there will be a lot of new bugs, just like a series of interlocking vicious circles. Many project duration delays are caused. To avoid or at least reduce this damage to a minimum, the developer first needs to know the influence of this damage. The logic thinking, skills and experiences here, skills and experience are far less than, and automatic testing has once again become an effective tool.
