The Data Abstraction Penalty (DAP)Benchmark for Small Objects in Java

• By Argyn Kuketayev
• August 1, 2001

How does one measure the performance penalty imposed by a compiler and execution environment on a program for using the data abstractions? In this article, I present the Data Abstraction Penalty (DAP) Benchmark for small objects in Java. The DAP Benchmark consists of 13 tests that perform the same calculation. Test zero is written in the lowest level Java, while other tests use different features of Java in various combinations. As in the Stepanov Kernels benchmark for C++,¹ the penalty is a geometric mean of ratios of test execution times to Test zero's execution time. I also present the results of a DAP benchmark run on leading Java 2 Runtime Environment (J2RE) implementations.

Every developer knows performance degrades when higher levels of data abstraction are used in programs. On the other hand, data abstraction is a desired feature of modern languages. That's why it is important to help programmers balance performance with abstraction. One way to achieve this goal is to study performance-abstraction characteristics of the compilers and execution environments.² In this article, I describe the DAP Benchmark for Java I wrote in attempt to measure performance degradation of Java programs while using various data abstraction features of the language.

The structure of the benchmark
The DAP Benchmark for Java was inspired by the Stepanov Kernels benchmark in C++.¹ I searched for a similar benchmark for Java. Failing to find one, I wrote the DAP Benchmark myself. It consists of 13 separate tests that perform the same operation in different ways—they add N double values M times. Ratio of execution time of a test to execution time of Test zero is a degradation factor. Following Stepanov, I call the geometrical mean of all 13 factors the abstraction penalty or DAP. I claim the DAP represents the penalty imposed by Java compilers and virtual machines on programs using abstract data structures and containers. Ideally, it should be equal to one. In reality, it is much more.

• Tests 0, 001, 002, 101, and 102 use simple for loop to sum an array of values.
• Test 0 uses an array of primitive double values.
• Tests 001, 011, 021, 101, 111, and 121 use Double object wrapper class to hold values.
• Tests 002, 012, 022, 102, 112, and 122 use a custom object wrapper class to hold double values called SimpleDouble.
• Tests 011, 012, 111, and 112 use custom iterator classes over an array that are similar to Java Collections Framework's iterator.
• Tests 021, 022, 121, and 122 use the Java Collections Framework Iterator class to traverse a Vector.
• Tests 101, 102, 111, 112, 121, and 122 use a custom plus method to sum values.

Results
I used my home computer to run the DAP benchmark. It has an Intel Pentium III 733 MHz CPU, 256MB SDRAM, Asus CUV4X Motherboard. It runs MS Windows 2000 SP1. In order to compare results of the benchmark for different Java virtual machines, I selected three Java virtual machines from the Volano Report³ that show the best results using Windows. They are:

• IBM J2RE 1.3.0 (build cn130-20010329),
• Sun J2RE, Standard Edition (build 1.3.1-rc2-b23), and
• JRockit (build 2.0.7-excelsior80, Thin Threads).

JRockit does not have its own class library, so I compiled the benchmark with Sun J2SDK 1.3.1-rc2-b23, and used Sun's Java Runtime Environment (JRE) class libraries to run the benchmark on JRockit's Java virtual machine. In order to run the DAP benchmark on IBM's Java virtual machine, the source files have been compiled with IBM JDK 1.3.0 build cn130-20010329.

Figure 1 shows the individual execution times for the three Java virtual machines I used with M=10,000 and N=4,000.

Figure 1. Data Abstraction Penalty Benchmark results.

As you can see from Table 1 and Figure 1, IBM's Java virtual machine has the largest DAP; however, it sharply outperforms the other two Java virtual machines in the first five tests. The tests 0, 001, 002, 011, and 012 do not use iterators and plus methods; the sum is accumulated in a primitive double variable and values are stored in an array. It seems IBM's Java virtual machine is not optimized for handling small objects in Java Collections Framework's data structures.

JVM	Abstraction penalty	Geometric mean time	Absolute total time
IBM 1.3.0	18	2.5	87
Sun 1.3.1-rc2	6.4	3.7	69
JRockit 2.0.7 (Sun)	9	4	66

Table 1. JVM comparison results.

Conclusion
The DAP benchmark allows the measurement of the performance of small objects in Java, giving developers a clue as to the abstraction penalty imposed by the Java compiler and runtime environment. However, there is a significant variance in results between vendors of different JVMs. See: the code section for the DAP source code.

References

1. Alexander Stepanov's benchmark. For source, see http://developer.apple.com/tools/mpw-tools/compilers/benchmarks/mrc-3.0f1/stepanov-3.0f1.html.
2. Robison, Arch D. "The Abstraction Penalty for Small Objects in C++," POOMA '96. For source, see http://www.acl.lanl.gov/Pooma96/abstracts/robison.html.
3. The Volano Report from April 2001. See http://www.volano.com/report/2001-04-13/.