Performance testing tip - assumptions are dangerous

I've been doing a lot of research over the past couple of weeks with a goal to understand how cryptography performs on different platforms, with different algorithms, with different data characteristics. In addition, I was also interested to learn how crypto accelerators perform compared to software based crypto.

I can't share all the findings with you, but I can tell you I was quite surprised by what I found. Once again, I was reminded that assumptions regarding performance are invariably wrong. The reason for this is there are so many subtle factors involved that can affect performance. Also, technology is constantly moving forward. Results and conclusions found today may not necessarily hold true tomorrow. As an example, does anybody remember how people used to chastise Java for being slow. That may have been the case in 1996, but in 2008 it most certainly is not.

A major assumption I had before I conducted my research was that I was convinced that the crypto accelerator would outperform s/w based crypto. This, it turned out was not the case. There were some cases where the s/w based crypto outperformed the crypto accelerator significantly. I also assumed that the SPARC chip would perform pretty well compared to AMD/Intel chips - it didn't really perform well at all in terms of raw speed/performance for the kinds of crypto operations I was performing.

The graph to the left shows the fastest and average performance for performing a SHA-1 hash on a credit card number on various platforms. The units are in nanoseconds. Look how fast Java performs on Intel on both Linux and Windows.