Source: http://www.smth.edu.cn/pc/pccon.php? Id = 860 & nid = 25758 & s = all
NINE LANGUAGE Performance Round-Up: Benchmarking Math & File I / O
Type: Dell Latitude C640 NotebookCPU: Pentium 4-M 2GHzRAM: 768MBHard Disk: IBM Travelstar 20GB / 4500RPMVideo: Radeon Mobility 7500 / 32MBOS: Windows XP Pro SP 1File System: NTFS Here's what happens in each part of the benchmark:
32-bit integer math: using a 32-bit integer loop counter and 32-bit integer operands, alternate among the four arithmetic functions while working through a loop from one to one billion That is, calculate the following (while discarding any remainders). :
1 - 1 2 * 3/4 - 5 6 * 7/8 - ... - 999, 999, 997 999, 999, 998 * 999, 999, 999 / 1,000,000,000
64-Bit Integer Math: Same Algorithm As Above, But Use A 64-Bit Integer Loop Counter and Operands. Start At Ten Billion and End At Eleven Billion So The Compiler Doesn't Knock The Data Types Down To 32-Bit.64- bit floating point math: same as for 64-bit integer math, but use a 64-bit floating point loop counter and operands Do not discard remainders.64-bit floating point trigonometry:. using a 64-bit floating point loop counter, calculate sine, cosine, tangent, logarithm (base 10) and square root of all values from one to ten million. I chose 64-bit values for all languages because some languages required them, but if a compiler was able to convert the values to 32 BITS, I let it go ahead and perform That Optimization.i / o: Write One Million 80-Character Lines to a Text File, Then Read The Lines Back Into Memory.
RESULTS
Here are the benchmark results presented in both table and graph form The Python and Python / Psyco results are excluded from the graph since the large numbers throw off the graph's scale and render the other results illegible All scores are given in seconds;.. Lower is better.TimeintmathlongmathdoublemathdoubletrigI / OTOTALVisual C 9.618.86.43.510.548.8Visual C # 9.723.917.74.19.965.3gcc C9.828.89.514.910.073.0Visual Basic9.823.717.74.130.785.9Visual J # 9.623.917.54.235.190.4Java 1.3 . 114.529.619.022.112.397.6java 1.4.29.320.26.557.110.1103.1python / psyco29.7615.4100.413.110.5769.1python322.4891.9405.747.111.91679.0
Analysis
Let's review the results by returning to the five questions that motivated these benchmarks. First, Java (at least, in the 1.4.2 version) performed very well on most benchmark components when compared to the .NET 2003 languages. If we exclude the trigonometry component, Java performed virtually identically to Visual C , the fastest of Microsoft's languages Unfortunately, the trigonometry performance of Java 1.4.2 can only be described as dismal It was bafflingly bad -..! worse even than fully interpreted Python This was especially puzzling given the much faster trigonometry performance of Java 1.3.1, and suggests that there may be more efficient ways to code the benchmark in Java. Perhaps someone with more experience with 1.4.2 can suggest a higher-speed workaround.
Java performed especially well (when discounting the strange trigonometry performance) compared to Microsoft's syntactically equivalent Visual J #. This discrepancy may be due to the additional overhead of the CLR engine (as compared to the overhead of the JVM), or may have something to do with Visual J # implementing only version 1.1.4 of the Java spec.Second, Microsoft's claim that all four .NET 2003 languages compile into identical MSIL code seemed mostly true for the math routines. The integer math component produced virtually identical scores in all four languages . The long math, double math, and trig scores were identical in Visual C #, Visual Basic, and Visual J #, but the C compiler somehow produced impressively faster code for these benchmark components. Perhaps C is able to make better use of the Pentium 4's SSE2 SIMD EXTENSIONS for Arithmetic and Trigonometry, But this Is Pure Speculation ON My Part. The I / O Scores Fell Into Two Clusters, with Visual Basic And Visual J # APPA Rently Using Much Less Efficient I / O Routines Than Visual C # Or Visual C . This is a Clear Case WHERE Functionally Identical Source Code Does Not Compile Identical Msil Code.
Third, Java 1.4.2 performed as well as or better than the fully compiled gcc C benchmark, after discounting the odd trigonometry performance. I found this to be the most surprising result of these tests, since it only seems logical that running bytecode within a JVM Would Introduce Some Sort of Performance Penalty Relative To Native Machine Code. But for Reasons Unclear To ME, this Seems Not to Be True for these Tests.
Fourth, fully interpreted Python was, as expected, much slower than any of the fully compiled or semi-compiled languages - sometimes by a factor of over 60. It should be noted that Python's I / O performance was in the same league as the fastest languages in this group, and was faster than Visual Basic and Visual J #. The Psyco compiler worked wonders with Python, reducing the time required for the math and trig components to between 10% and 70% of that required for Python without Psyco. This was an astonishing increase, especially considering how easy it is to include Psyco in a Python project.Fifth, Java 1.4.2 was much faster than Java 1.3.1 in the arithmetic components, but as already mentioned, it lagged way behind the older version on the trigonometry component. Again, I can not help but think that there may be a different, more efficient way to call trigonometric functions in 1.4.2. Another possibility is that 1.4.2 may be trading accuracy for speed relative to 1.3. 1, WITH New Routines That Are Slower But More Correct.
What lessons can we take away from all of this? I was surprised to see the four .NET 2003 languages clustered so closely on many of the benchmark components, and I was astonished to see how well Java 1.4.2 did (discounting the trigonometry score ). It would be foolish to offer blanket recommendations about which languages to use in which situations, but it seems clear that performance is no longer a compelling reason to choose C over Java (or perhaps even over Visual J #, Visual C #, or Visual Basic ) - especially given the extreme advantages in readability, maintainability, and speed of development that those languages have over C. Even if C did still enjoy its traditional performance advantage, there are very few cases (I'm hard pressed to come up with a single example from my work) where performance should be the sole criterion when picking a programming language. I would even argue that that for very complex systems that are designed to be in use for many years, maintainability ought to Trump All Other Considances (But That's An Issue To Take Up Another Article). Perxpanding The Benchmark
The most obvious way to make this benchmark more useful is to expand it beyond basic arithmetic, trigonometry, and file I / O. I could also extend the range of languages or variants tested. For example, testing Visual Basic 6 (the last of the pre-.NET versions of VB) would give us an idea how much (if any) of a performance hit the CLR adds to VB. There are other JVMs available to be tested, including the open-source Kaffe and the JVM included with IBM's SDK (which seems to be stuck at version 1.3 of the Java spec). BEA has an interesting JVM called JRockit which promises performance improvements in certain situations, but unfortunately only works on Windows. GNU's gcj front-end to gcc allows Java source code to be compiled all the way to executable machine code, but I do not know how compatible or complete the package is. There are a number of other C compilers available that could be tested (including the highly regarded Intel C compiler), as well as a host of other popular interpreted L . Anguages like Perl, PHP, or Ruby So there's plenty of room for further investigation.I am by no means an expert in benchmarking; I launched this project largely as a learning experience and welcome suggestions on how to improve these benchmarks Just remember the. Limited Ambitions of My Tests: I am NOT Trying to Test All Aspects of a System - Just A Small Subset of The Fundamental Operations On Which All Programs Are Built.
About The Author: Christopher W. Cowell-Shah Works in Palo ALTO A Consultant for The Accenture Technology Labs (The
Research
& Development wing of Accenture). He has an A.B. in computer science from Harvard and a Ph.D. in philosophy from Berkeley. Chris is especially interested in issues in artificial intelligence, human / computer interaction andsecurity
His Website is www.cowell-shah.com.
