In my last post, I presented some simple calibration results for the Sun Wireless Tookit against the JBenchmark 2 benchmark test.
In this post, I present extend the previous analysis and post calibration results using varied heap sizes and VM execution speed.
Results
The following plot shows JBenchmark 2 scores derived from the Sun Wireless Toolkit for emulation speeds of 100, 250, 500, 750 and 1,000 bytecodes / second for heap sizes of 500, 1,000 and 2,000 KB.
The trend line for a 1MB heap size coincides with the one that I presented in an earlier post except at 1,000 instructions / second where my machine had a score of 130 versus 120 on Doug's box (the make, model and CPU are different).
Based on this sample of two machines, we can conclude that the WTK VM emulation provides by the WTK will provide similar results across all machines. This is the expected result and it is important because it means that the emulator can be calibrated to perform in a similar way to real handset terminals by setting the VM emulation speed. Furthermore, the emulation speed will will be accurate to within about 10% regardless of the machine specification.
The graph also shows that heap size does not affect the JB2 score. To prove this, I lowered the emulated heap while keeping a constant emulation speed of 1,000 bytecodes/second until the JBenchmark test failed with an out of memory exception. JBenchmark scores varied between 135 and 142 for memory configurations in the range 500 to 100 KB - similar values to those for 1MB in the plot, above.
This result could indicate that either the JBenchmark 2 test creates very little garbage or that the WTK VM speed emulation does not affect garbage collection. I suspect the later is true but need some more benchmark tests that do create lots of garbage in order to test this.
What Next?
I have written an Applet that provides a simple way to visualise handset performance over time by vendor. I just need somewhere to host this which I hope to get around to any day now.
