Software test automation practice
Author: chris
Why is the software test automation? Through my work in Microsoft in these years, I can feel that it should be automated and can define software systems. In theory, it should be a functional definition software system, how is the software system works from functional specifications. But if you don't have a set of test programs that can be automatically tested, there is no way to verify that this software system is modified. This software system is not exactly the same as the previous software system. You have no way to verify, only one set of automatic The system can verify that the software system is still in the original state. I think the automatic test is a very important thing. If a software system, a software system is not followed by an automated system, this software system basically has no vitality, and there is no way to update, and there is no way to do other services. .
The complexity of the code cannot be seen. Now testing a code, all coverage is also impossible, each if ... else ... or switch statement is doubled, and many exception handling code will not encounter normal use. Many errors related to timing, dead lock, resource conflict, multithreaded are difficult to capture. In addition, each of our versions each with SP and QFE, each SP is combined, you can get an astronomical number. If there is no automatic test system, you want to test the entire product, in all different circumstances, Almost a impossible thing. If you have a set of automated test systems, there will be a half-time effect on the future.
Microsoft has a variety of tests, first developing teams with their own DRT or Check-in Suites. No need to reload the machine, the speed is fast, and the requirements are simply reported, 100% passed, so there is no need to analyze the ability. Close to the relationship between the development team, in their source control system. The test team has a test LAB BVT, all testers are together, the BVT requirements are needed to reinstall the machine and can exclude other old factors to interference above. The speed is fast, only 0 level test, BVT is also automatically running BVT after the end of the daily build, the result report requirements are also relatively simple, generally need 100%, if not 100% is always looking for problems, the result, The results of the big day should be 100%. Close to the test team, in the source code control system of the test team, automatically trigger by the daily product compilation system, fully automatically run.
It also needs to reload the machine and simulate the form of usage of end users. Need to use daily compile results, large size, number of tests, and the environmental requirements are also complex. At this time we will introduce the testers just mentioned, and you may need to test different versions, or different languages. Reporting to test results is high, because all tests must be 100% this time, there may be such a problem day before yesterday, today this question reappears, is it ignored? If this report results tools can't do this, you must re-do every day. The test team is generated as needed. The personal private pass of the test team is generated by the test, and does not necessarily need to reload the machine. It is high for the test results. The relationship with the test team is close to the source code control system of the test team.
A specific automated test system analysis. There are many sets of automated test systems in Microsoft, such as Oasys, is a system for office, as well as Maddog, Bruce, and more. These systems evolve to the end, the overall structure is similar. If you have developed a new automation test system independently, I estimate that the result is like this. Take the Oasys system as an example, there is a web server, which is the web ui of the front desk. SQL Server background server, control program, client program, customer program is installed on machines, some result reports and analysis programs, file servers, save daily buildings, Log File each time, etc. Other, in addition, all tests are carried out in the machine pool, a series of almost the same machine, call a group or some of some at any time in one test, or a machine for use. The function of the specific module:
The file server is a daily result to store each Run's logfile. For example, what you need to install, what kind of background background.
SQL Server stores the relevant setup parameters, stores all the parameters related to the test, the case of all Client machines, what kind of machine, how much memory, system time, what is now doing, all machines exist Also store all Run Pass's overall results, running, pass rate, running machine name, etc. ...
The web server is a web front desk, which is equal to a web configuration. The test team uses the entrance to the background server, the advantage is that only IE can run, zero installation, easy to set up. You can query and modify all parameters stored in SQL Server, you can generate, query and trigger Lab run, you can query the LAB RUN progress, overall results, etc. You can also query and adjust the status of the client, the installed OS, program, etc.
The machine pool is dozens of or relatively close light machines, with an operating system or file system's mirror program, you can use self-update.
The controller program, reads work data from SQL Server, generates work scripts, and assigns jobs to the client according to client parameters. When running, get the result and update SQL Server from the client, divided into machine room controller Personal controller, etc. Lab Controller is primarily used to control the Machine Pool in Lab, and Private Controller is primarily used to control the machine of the test team member in Office. The controller program should have the function of dynamically assign tasks. The controller program should have the function of Load Balance. The controller program should be able to work according to the priority of the Lab Run and the optimized allocation of the machine resources. For example, you have an automatic test, an automated test is not only one test at Run, and there may be three to four simultaneous exercises at the same time, and some Lab Run will progress very slow, some priority is high. Dynamic allocation features are very important, according to your Lab Run's priority, decide how many resources are allocated to run, which is a very important system. How much time is also a more important function, if you have previous data, after the last Lab run, you know how much time each test needs Run, add the overall time, you can see it today. How much is left.
Client program. This program is a very small, very lightweight program, which is running on each small client, almost zero installation, is a small program, very light, because he rely on any module, can run on the bare metal, All modules in Microsoft may be a module that needs to be tested. If the program running on the client needs ADO, we need to test this ADO. It can be used to complete the task given to the controller and record the whole process, very stable, and will not crash, and can intercept most of the abnormal conditions. Screen capture, Timeout, all systems are running automatically in Lab, all of which are some machines, no people. The automated running system is very fast, equal to 24 hours a day. Result report and analysis procedure, installed on the host of the test team member. Query the database and display the status of the required RUN, pass the rate, and it takes time. Simple operation control, such as re-run, specify the host run, etc. The most important thing is to automatically analyze the result files of running, compare the Test Case that improves and deteriorate. Every day I have said that there may be a thousand-made outcomes need to analyze, and the result of thousands of results may have a big one has been analyzed. If you don't want to look at these results, it is necessary to compare. A variety of situations can be read, and the results of the deposits are compared, and the new need to analyze measures, you have seen these measures. After this program analyzes the error, you can enter the reason for failure errors. Provide links to results files, Bug Database links, providing a tool for further analysis results. Record the reasons for the failure of Test Case, and directly input and track the BUG number. When the results are analyzed, it can be recorded as a baseline result in order to compare the results of the future.
How do we use this system. First, a test manager gets a test task, such as Windows 2000 tasks, he can tell someone, this person should be a member (LRF) of the test team (LRF), this person is born according to these things Some runs, know what state this run enters, is not very successful, etc., assisting the entire Run can be done very peaceful. This person's test team does not have a lot of people who are willing to do this. Everyone turns this thing. LRF generates corresponding Run according to the demand in the web ui template, fills in the OS requirement, language requirements, run priority, completing time, etc., put these things into it. This RUN is obtained by the controller program, and after obtaining the number, the number of machines, the number of machines, the total number of machines, the total number of machines, etc., and the number of tests, etc. are allocated. The computer room client gets a task, cleans itself, the program and module of the installation requirement, and the test run from the file server, performs the test and copy the result.
LRF tracks Lab Run progress through the web ui, checks if there is a major problem, check whether the operation progress can be completed and timely adjustment, when Lab Run is close to an end, the Email test team verifies its respective results. It can be said that the tester itself does not monitor Lab Run. The test team member received Email to open the results report and analysts we just said, the most important thing, new mistakes, such as bug, producing a small code, reproduce this error.
The problem of the test program, the problem on the test program itself, you want to modify your test programs, etc.. If it is a problem with the client installation, the entire system is a problem for each of the entire system, for example, today XP is all passed, relatively simple. Finally, the test manager summarizes all the results, but also in the results report, after the results are analyzed, the test manager automatically generates a total report, which will determine how our products will be made by this data, not just in the final summary Time to use. To make a quick review, for a set of test automation systems, it is very important for a mature test team. If a test team does not have a complete test system, there is no way to test. A newly developed test automation system, trusts to be common. Our test system can do anything, you can watch the module of his system design, not to test what is designed, you can test anything, so the office is such a big product, anything inside is tested on the test system, Test system requirements are very simple. For the automated test itself, it is a very simple thing, nothing knows, only knows a result, and it is over. There are also some controls that control the problem of dynamic allocation resources I just said. If you are not enough, this is the automated test control, but what you want to test, it is completely determined by the test person, all automatic scripts are all edited, you just save it on the file server. Each piece of test automation system has focused on each piece, does not have to be large. From the test automation system, the beginning of the beginning of the beginning may be large. You can see that you have to establish a complete test automation system. Each system will find ways to automate, modular, just started to invest very much, long-term advantageous. If you are hand-tested, there are so many different applications in the background, with a variety of XP, in different languages, if you use manual to do hard to imagine. If there is a test automation system, all other machines are automatically run as long as it is developed, and he doesn't have to take him. So he is very important to encourage the morale of the entire team. A tester works a day, it is very simple to test the test, just look at the new situation every day, the other things do not use. Automated testing is also very important for later software maintenance. The automatic test system can test, no matter what you do, you can test all the tests of your product, if only one version may not work, Microsoft all product Office already has twelve versions, so many versions Program, if there is no automatic test, it is difficult to imagine. If there is an automatic test, you can use it synchronously, so you can achieve the effect of halving, soon you can verify, you can avoid duplication of labor. Automatic test definition software system, if a software system does not automatically test, later maintenance, the issuance of the next version is an unimaginable thing.
