Software test new model
Author:
Brian march
translation:
Blueski
Under normal circumstances, a software model description mainly includes, what questions should you take into account during the test process, how to plan the test, test what target is to achieve, when start, what information you want to use in the test? . A good model can guide you to think about the problem, and the bad model can only make you misuse astray.
Here I am going to claim that most software test models are in a bad model. This is because these test models are only some of the decorations and supplements of the software development model.
People have been looking for a model of software development. After creating a new model, the test is placed in the back section of the model as a stage. Therefore, the test is always active, and the test is always driven by the development. Overall, we are testing their finished products. However, as a test, the driving method is incorrect. In fact, it is obviously related to various behaviors in the development process, and the test does not play a balanced role. Such tests only detected what developers did, and did not detect what they did in accordance with rules. This practice cuts the behavior of this close contact, and the remaining only those thoughts of hastping. s damage.
The result of this is that the effect is very poor, the efficiency is very low. Tests that have poor effect will result in many bugs that have not been discovered, and the cost is wasted with cost.
In this article, I have to do 2 things, one, I want to deny a bad model, that is, the V model. I hope that by discussing, "Unit Test" and "Integrated Test" These 2 words can be canceled from our vocabulary. Second, I will describe a better model. But first, I think that it is necessary to have a full and reasonable model. I just describe some important requirements that should meet with some new models. These requirements will be enumerated at the end of this article.
1. What is the problem with the V model?
In this article I want to analyze the V model as a typical model of a bad model. I have selected V models as a typical typical because the V model is the most widely known test model.
The most typical V model version generally describes the software development process in its start part, as shown in the following figure:
Figure 1 - VIE development stage
This is an old waterfall model. As a development model, it has many problems, but it does not discuss it. Although its various status is the basis of our most familiar V models to be discussed. My criticism also also targets other decorative test models above some better development models, such as spiral models [Boehm88].
In the V model, the test process is added to the second half of the development process, as shown below:
Figure 2 - V Model
The unit test is detected, whether the development of the code meets the requirements of the detailed design. The integrated test is detected that the previously tested components can be combined together. The system test is detected that it is integrated with the requirements of the system specifications. The acceptance test detects whether the product meets the needs of end users.
For test design, it is obvious that users of the V model tend to treat execution tests and test design. After the development document is ready, you can start related test design. As shown in the figure below, the corresponding test design covers the relevant development process:
Figure 3 - Cover the test design over the V model after the development process
The V model has very attractive symmetrical shape and bringing many people to astray. This article will focus on issues caused in unit testing and integrated testing.
For the convenience of explanation, the following pictures are made here, including a separate unit, and a unit group, I call the subsystem.
Figure 4 - an imaginary subsystem
For a unit, how much is the most appropriate problem, there have been a lot of discussion, exactly a single function, a class, or a collection of related classes? These discussions do not affect what I want to explain here. We are right and think that a unit is a minimum code block that developers can discuss independently. The V model considers that people should first test each unit. When all units in the subsystem have been tested, they will be collected together to test to verify that they can form a run.
So how do I test for the unit? We will view the definition of the interface in the detailed design, or view the source code, or simultaneously see the input data that meets the relevant criteria in some test design, then check the result, see if it is correct. Since each unit generally operate independently, we have to design the pile module (STUB) and drive module (DRIVER) as shown below.
Figure 5: Unit and external drive modules and pile modules
The arrow in the figure represents the execution trajectory of the test. This is the "unit test" that most people said. I think this method is sometimes a bad way.
The same input can also be provided with other units in the same sub-system, so that other units play a pile module and play a drive module. As shown below:
Figure 6 - Test execution trajectory between each unit in the subsystem
Which method is selected, this requires a compromise and trade-off. How much is the design pile module and a drive module? How do these modules maintain maintenance? Will the subsystem cover some faults? How big is the difficulty of tap over the entire subsystem? If our test is really started until the integration test, then some bugs may be discovered later. What is the price caused by the price of the design pile module and the drive module compare? and many more.
The V model does not consider these problems. When the unit development is completed, the unit test is executed, and the integrated test is performed when the system is set together, this is only. What is strange and frustrating, people never do some weighing, they have been subject to their model.
Therefore, a useful model should allow testers to consider saving and postponed the possibility of testing.
One test, if the bug in a particular unit is to be found, it is preferable to perform independently in the unit, and thereby plus the specific pile module and the drive module therebetween. Another method is to test it as part of the subsystem, which is mainly to find integrated issues. Since a subsystem itself also requires a pile module and a drive module to simulate the contact of the subsystem and other subsystems, the unit test and integration test may be postponed to at least the entire system is already partially integrated. In this case, the tester may simultaneously perform unit testing, integrated testing, and system testing through the external interface of the product, the same, its main purpose is to reduce the cost of the overall lifecycle, test the test cost and extension, and thus Causes extension discovery of the cost cost of bugs. Accordingly, the difference between "unit test", "integrated test" and "system test" have been greatly weakened. The result can be refer to the figure below:
Figure 7 - New method: unit testing and integration testing in some stages
In the square on the right of the figure, it is best to change to "perform some appropriate tests and get the corresponding result."
What happens to the left? Consider the system test design, its main dependent and source of information is specification. Suppose you know that there are 2 units in a specific subsystem, they are interconnected at runtime and to perform a specific declaration in the specification. Why not test the declaration in this specification when it is integrated, it is like the design of the test immediately after the design is completed. If there is no relationship if the sub-system outside the declaration and subsystems, why should I wait until the entire system is completed? The sooner, the lower the BUG cost is wrong? In the previous picture, we used the arrows (more effective, but there were delays in time). Here you can also put the arrows down (in advance):
Figure 8 - New method: Test design in advance at different stages
In this case, it is best marked as: "any test design that can be made in the current information conditions and any test designs". Thus, when the test is designed from a description of a component in the system, the model must allow such tests to be executed before the component is assembled. I have to admit that my picture is very ugly, these arrows are all, I have 2 points for this:
1. What we discussed is not to create beauty, but want to find as much serious mistakes as possible while reducing costs as much as possible.
2. Difficult part of the reason is also because the results must be performed in accordance with certain order, that is, developers first provide system description documents, and then test and associate these documents. These documents are like a solid old oak tree, and the test design is like a fine branch wrapped around the tree. If we use different principles to organize, the picture may become better. But complexity is still inevitable, because the problems we have to discuss are very complicated.
The reason for the V-model failed is that it divides the system development process into different stages with fixed boundaries, which makes it difficult to cross these boundaries to collect information needed. Some tests should be made earlier, and some tests need to be delayed. Moreover, it also hinders to achieve information from the different stages of the system. For example, some organizations sometimes perform such practices, that is, the completed work is signed. Such a provision also expands to the design of the system test. The signing indication has been evaluated, the test design has been completed unless the corresponding design document changes, otherwise it will not be revised. If the information related to these tests is later re-excavated, for example, architecture design indicates that some tests are redundant, or the detailed design indicates that there is an internal boundary that can be tested together with existing system tests, then It actually needs to continue to adjust the original system test design.
Therefore, the model must allow individual testing designs for integrated information from different sources. In addition, the model should also allow for a re-test design after new information sources.
2. A different model
Let us see the second content of this article, a different model.
Many times people hand over the code to others and say: "I hope you can accept and like it." This not only happened to put the entire project in a CD, also happened inside the project.
For example, a group said to another group: "We have completed the support for XML for the Comm librarism. Source code is now placed in the Master library, the executable library has been added to the integrated and created environment .xarg There is no hindrance to the work of the group, let's take it. "
A programmer checked the BUG's modification and sent a message: "I have modified the bug in the bug list, I am sorry!" At this point, you can continue to handle it.
In these cases, people have to hand over the code to other people, which may have some impact. Test staff need to intervene this process. Before handling, the tester should perform these code, discover the bugs, and ask questions: "Do you really submit these?" So, the content of the handover may be delayed until BUG is repaired. Although you have to do other tests, this test is still very basic testing. If you don't do this, you can't be a qualified tester.
Our test model must include this important reality requirement: Test for code handover. Thus, the test model should be prompted for testing for each code handover.
Let me work as an example with the XML's Comm library. There is a group where a group is handed over to the Xarg team to perform the remainder of the project. So who will affected?
The Xarg team to directly use these support XML code may be affected; this may affect market staff later, they have to provide product demonstrations and propaganda for the "Partner release" version at an industry display meeting. And XML support is an important part of their sales; and it may harm the partners who adopt our products.
Now we can make some interesting questions about the test plan. The simplest thing to do is to perform the complete test of XML support immediately when you hand over. ("Complete" meaning is to design as much test as possible for this purpose), but maybe some XML features are not required, so they can put them in partner version sealing ("Partner Release" in the picture below. The test is going. This means that some XML related tests can be placed in a later handover process. Or based on other reasons, for example, there are other test tasks to be executed in the near stage. The Xarg team delays a short period of time due to the bug repair in XML.
The progress stated in our test plan can be expressed by the way of annotation on the development of the time line, as shown below:
Figure 9: Adding test plan above the development plan
We should strictly be tested around the time period of XML support. Test design and test support work is taken as early as test. In addition, additional XML tests are delayed to "code completion" based on the entire project ("Code Complete" milestone in the figure, or wait until all subsystems are concentrated together, and the entire product is passed through the industry conference. The version was created after stabilization ("Partner Release" milestone in the figure).
Obviously, two contents are not included in the code completion milestones:
There are also a large number of other testing (including design, tool selection). These work may be postponed due to the handover of subsystems other than CommM. Moreover, there are also tests for the completion of certain risks specified in the milestone, for example, there may be a set of demonstration Demo scripts for running marketers, including the deviation that she may cause inadvertently. Its purpose is to avoid this, that is, when she stands in front of the audience of 1,000 people, she is only the first time to enter data in a particular order. Some XML tests in the first handover need to be executed again on the code completion milestone.
My point is that the test plan is composed of many difficult decisions. These decisions include personnel organization arrangements, machine resource allocation, time positioning of the test design, and the number of test support code, which tests should be automated, and so on. These decisions should be made according to the content information in separate hands. If only one handover, then you can be more smooth. The test plan should continue to consider the following questions:
1. Risk analysis, who will be damaged, what way?
2. Select a test path to locate specific risks.
3. Estimate the period and cost of test design and execution.
4. In the specific location on the project progress, the plan will be incorporated into the implementation: a. Start the design of the test ...
B. Design and create some of the support tests ...
C. Testing the section before all tests is completed, because there may be no only one handover, there may be some potentially complex interactions in each handover test plan. Work arrangements have to be adjusted to achieve a balance between each other. Test support code and tools need to be shared in each task. You must also take into account what extent to let the tests designed for earlier handover, and so on.
This looks very complicated. It seems that there seems to have too much content needs to be tracked, and too much content may be ignored. Maybe you think I am asking you to execute the requirements of the test plan in IEEE 829 [IEEE98] for each handover, then combine them into a test plan for handover throughout the project.
Yes, nor. Thinking about the problem is always taken. Too many plans may reduce the output of results. Sometimes, what you need to do is to stop planning and start actions. For example, you can't think and describe each bug fix, although bug fix is also a handover.
But bug repair is a problem in reality in actual work. BUG repair should be included in the overall project plan. It is important to emphasize that you should have a default bug modification process standard process, which should include the verification process running on each submitted bug fix. You also need to think hard to think about the problem. Many times, each verification is placed together simultaneously and complete.
More realistically, a model should allow some mechanical behavior, for example, "Do not do the following operations, whether it is the X-type handover." At the same time, we encourage the implementation of specific handover implementation, and the smaller the risk, the more mechanical test behavior can be used.
A model that clearly takes into account basic test reality will definitely be better than ignoring these reality or completely abstract your work complexity. The document is another example.
I have not mentioned the needs and specifications, or design documents. A series of changes arising in a certain handover will cause many disputes. What is the role played by these documents? They are often used:
Figure 10: Utilization of development documents in the test
Documents can guide you how to respond during a handover change. If you have a good demand document, it may be a description of the problem solved by the product, although it may not be very straightforward. It can help you analyze risks. A good specification describes the behavior of the system. This will help you transform the test method into specific tests. A good architecture design can help you understand a few different situations that may cause changes: what is the impact of other parts of the system? What test needs to be done again?
I don't often see a good document. Demand documents are often like a list of systematic features for market sales. Specifications Sometimes it is like a user manual file submitted after the code is completed. Design documentation often does not exist.
Ok, I have separated the test process and software development process by a centralized discussion of the change caused by the handover. If the documentation is weak about "XML Support Features Join", I will do my best to perform as much test design. Then, if you have a post-related user documentation in the later period of the project, I can have related tests that have been submitted again. If the market demand changes, they often do this, I will also increase or remove some tests thereafter. All this looks like this:
Figure 11 - Test under the incomplete document and testing in the later period
In this way, although the project document is always not in place, it is often delayed, and the effect of the test is often reduced, and we still have to avoid the test of the project document.
The flexible test personnel of the mind are not too believed in documents. After all, people are always mistaken. So, isn't people writing these documents?
Since the "official" document is weak, the test model must explicitly encourage a variety of different information sources outside the project documentation during the test. Test staff must communicate with programmers, users, marketers, technical authors, and any people that may provide clues to achieve better testing. Test staff should also work hard to immerse yourself in the atmosphere built by certain techniques. For example, I hope that the tester often visits the XML address of W3 organization when doing XML testing.
http://www.w3.org/xml/) and other XML sites, mailing lists, and even more special DAVENET / scripting news (http://www.scripting.com/). These resources are not called "auxiliary channels", but can be included in the resources of the plan and schedule.
In addition, the test it is also a useful information for useful information. Good testers will read the BUG report carefully because these reports tend to the weakness of the system. In particular, these reports also implied some strategies on the architectural strategy provided by some formal architectural design. The act of performing tests should produce some new test ideas. If the model does not take into account these, then it is a backward model.
Therefore, the test model should include feedback loops, allowing test design to take into account that more test content can continue when running testing.
In our work, true complexity comes from all programs execution in an uncertain, easy to ignore. The code is not the only thing that is constantly changing. The plan schedule is also changing. New features expansion will bring new milestones. Some features will be removed from the current version. During the development process, everyone-market staff, developers and testers will gradually understand the problems such as "what products provide". In these cases, how can we say that the first version of the test plan will be completely correct?
Therefore, the model should require testing planners to formulate clear provisions, which will be responsible for the transfer of handover content, new handover, and handover content.
3. Summary
The V model has the following fatal defects, and other models are actually similar to this:
1. Ignore this fact that software development is composed of a series of handover, and each handover content has changed the previous handover behavior.
2. Depending on the existence of the development document, and the accuracy, integrity of the document, and no time limit.
3. Determine a test design is based on a separate document without including the revision of the documentation according to the previous rear stage.
4. Determine these tests that depend on a separate document must be implemented together.
I roughly describe an alternative model, but it is not fine enough. It takes into account the code of handover and milestones. The following is clearly descriptated for test cost control:
The target design is to define the test input that may find bugs, and the target execution target is to add this data in a variety of ways, and check the results, thereby reducing the cost of the entire lifecycle.
Our model assumes that software products are always imperfect. There are many changes in the development process, and the testing of products is also a process of continuous learning.
In the past, I rarely considered the model. I have been still in the surface of the surface. Although I plan to make a plan, I always spend a lot of extra energy and time to consider the aspects not mentioned in the model. In other words, the model has hindered, so I need to study this.
For a new model, the requirements for the model must be very clear, which is very important to product development. I hope that the description of the models of the model they advocated herein can be as accurate as the description in the V model and have the same guiding significance.
The ideal test model should include the following requirements:
1. Make the test respond to each code in the project.
2. Requires testing planners to formulate clear regulations, and be responsible for the transfer of handover content, new handover, and handover content.
3. In the test design, in addition to the use of project documents, you should also expressly encourage other information, which have different sources. 4. In fact, the project document is always not in place, and often delays, the effect of the test is often reduced. But we still need to avoid testing of project documents.
5. Allows some independent tests based on the integrated information provided by a variety of sources.
6. Let the test are redesigned to perform in the form of new information.
7. Contains the feedback loop, allowing the test design to take into account that more test content can be found when the test is running.
8. Let the tester recognize that the delay of the test can save costs.
9. Perform the execution of the components before the component is assembled into the program.
thank
It was earliest to realize that James Bach realized that handover should be considered in the test plan. NOEL
NYMAN and Johanna Rothman have provided some helpful comments in a draft. Kamesh Pemmaju and Carol Stollmeyer have finally not given up the defense and elaboration of principles, not only in the details, but also in actual life, and in every actual project. They gave me a lot of promotion and made me repeatedly think about the problem.
reference
[BOEHM88] Barry W. Boehm, "A Spiral Model of Software Developments and Enhancement," IEEE Computer, May 1988
[IEEE 98] "IEEE Standard for Software Test Documentation," IEEE Standard 829-1998, Electronic and Electrical Engineer Society, 1998
[Marick98] Brian Marick, "WHEN Should A Test Be Automated?" International Quality Weekly, May 1998 (
ftp://ftp.rstcorp.com/pub/papers/automate.pdf)
