How to implement Benchmark standard test
--- Proposal of the problem and TPC-C standard
Last Updated:
Monday, 2004-10-18 16:36
Eygle
1. Problem
No matter what kind of system you implement, you may have considered such a series of questions: What kind of equipment I should purchase? How is my system performance? How many users can my system? My system can withstand how much concurrency? Performance problem When will you appear? When will I upgrade? Obviously, answer these questions is not that simple, more people give an experience value based on experience, but in the large system design process, experience shows Is not enough, you have to take enough evidence to explain your architecture, your system must be theoretically to meet the needs of users. Here we have seen here, one of the most important content is introduced, that is: User needs. This is in line with economic rules, there is a need for development, and demand promotes everything. So there will be the standard test of today we want to discuss. In recent time, it has been involved in a large-scale program design, planning and testing Now write some experience, for your reference.
2.TPC organizations About Benchmark test, the most famous is TPC (Transaction Processing Performance Council, Transaction Performing Performance Council). You can get more detailed information on its official site www.tpc.org .TPC organization was established in 1988 In the year, some original manufacturers with software and hardware systems or related solutions in the computer are developed from the eight companies at the beginning of the establishment to more than 50 companies, IBM, NCR, HP, Oracle, Microsoft and other internationally renowned companies. It is its member. Headquartered in the United States, it is open to the world, but so far, the most members of TPC organizations are beautiful, day, Western Europe's computer software and hardware.
As a non-profit organization, the Transaction Performance Committee (TPC) is responsible for defining transaction processing and database performance benchmarks such as TPC-C, TPC-H, and TPC-W reference testing, and publish objective performance based on these reference test items data. The TPC benchmark test uses a very stringent operating environment and must be carried out under the supervision of the independent audit institution. Commission members include most major database product vendors and server hardware system vendors.
The TPC does not give the code code, and only gives the Standard Specification of the Benchmark. Note that here we propose the most important content in Benchmark test: Standard specification. You may hear a lot of people saying this, TPC-C value does not have any practical meaning, any testing cannot simulate the load in real production environments Happening. This is always the truth, but it is unfamiliar, or this statement actually misinterprets the original test of TPC-C. We say that the most important thing in Benchmark test is the standard specification, that is, the tool, etc. The factor is not important, as long as everyone uses the same standard specification, the same tool is systematically tested, then the test results have more significance.
From this significance, Benchmark test actually became a stage that various manufacturers show their technical strength. Any manufacturer or other tester can optimally construct its own system based on the standard standards announced by the TPC organization, optimally use the optimal Platforms and the most efficient applications. So you may listen to Oracle to attack DB2, Oracle said: IBM TPC test uses an Oracle database, not their own DB2. In fact, this statement is also one side, TPC-C Testing the performance of OLTP, and we know that DB2 is really powerful in data warehouses.
In order to ensure the objectivity of the test results, the testers (usually manufacturers) must be submitted to a complete report of the TPC, including the detailed configuration of the measured system, the classification price, and the total cost of five-year maintenance fees. Price. The report must be verified by the TPC authorized auditor (TPC itself does not do audit). Now there are only a few auditors in the world, all in the United States. TPC has launched four sets of reference procedures, known as TPC-A, TPC -B, TPC-C and TPC-D. The A and B have been out of time, no longer used .TPC-C is a reference program for online transaction processing (OLTP), and TPC-D is a reference process for decision support (DecisionSupport). TPC later tried to launch TPC-E, TPC-S, and finally gave up.
3.TPC-C
We focus on TPC-C test.
TPC-C is an industrial standard benchmark test project designed to measure online transaction processing (OLTP) system performance and scalability. This benchmark item will test a wide database function including queries, updates, and queue small batch transactions. Many IT professionals treat TPC-C as a valid indicator that measures the "true" OLTP system performance.
TPC-C benchmarks measured an analog order entry and sales environment measurement per minute business (TPMC) throughput. It is particularly worth mentioning that it will specifically measure the number of new order transactions generated every minute when performing other four transaction types (such as payment, order status update, delivery, and securities level). The Independent Audit Agency will be responsible for notarizing the benchmark test results, and TPC will report a comprehensive test report. This test report can be obtained from the TPC Web site (http://www.tpc.org).
TPMC definition: TPC-C throughput measurements measurement of average transaction per minute during effective TPC-C configuration, at least 12 minutes
3.1TPC-C specification summary
TPC-C is specifically for online transaction processing systems (OLTP systems), in general, we also refer to this type of system as a business processing system.
A TPC-C test specification simulates a more complex and representative OLTP application environment: Suppose there is a large commodity wholesaler, which has a number of commodity libraries distributed in different regions; each warehouse is responsible for 10 points for 10 points The goods; each point of sale provides services for 3,000 customers; each customer has 10 products; about 1% of the products in all orders are not in stock directly to the warehouse, requiring warehouses in other regions. goods.
The system needs to process the following types:
New-ORDER: Customer Enter a new order transaction; payment: Update the customer account balance to reflect the payment status; Delivery: Delivery (analog batch transaction); ORDER-STATUS: Query the status of the customer's recent transaction; stock-level : Query the inventory status of the warehouse in order to be able to replenishment in time.
For the first four types of transactions, the response time is required within 5 seconds; for the inventory status query transaction, the response time is required within 20 seconds.
Logical structure diagram:
flow chart:
3.2. Evaluation indicator
After two years of development, the TPC-C test specification was released in July 1992. Almost all vendors that provide hardware and software platforms in the OLTP market have released the corresponding TPC-C test results. With the continuous development of computer technology, these test results are also refreshed.
TPC-C test results mainly have two indicators:
● Traffic indicator (THROUGHPUT, TPMC)
According to the definition of TPC, the traffic indicator describes how many new-Order transactions can be handled every minute while performing Payment, Order-Status, Delivery, Stock-Level. The response time of all transactions must meet the requirements of TPC-C test specifications.
The bigger the traffic indicator value, the better!
● Price / Performance, Exquisite TPMC, is the ratio of test system prices (referring to the US quote) and traffic indicators.
The smaller the price, the better!
About this part, you can find a detailed instructions in the official document of TPC-C, you can get the official documentation of the TPC organization in the following link:
http://www.tpc.org/tpcc/spec/tpcc_current.pdf
On the official website of IBM, you can also find some description:
Http://www-900.ibm.com/cn/products/servers/pseries/tech/tpcc.shtml
The current version of TPC-C is 5.2.
Author: eygle, Oracle technology followers, Oracle technical forum itpub.www.eygle.com from China is the biggest author's personal site you may contact the author by Guoqiang.Gai@gmail.com welcome to explore technical exchanges and links. exchange.
Original source:
http://www.eygle.com/special/how.to.implement.benchmark.test.01.htm
