Whenever I see claims like “Breakthrough Performance” or “Leadership Performance”, or some such claims from vendors, my antennas go up and I reach for that bottle of salt. Typically these so called whitepapers or reports are full of marketing cruft carefully sidelining the real information which is also present in the report. Today I came across this report on VMware’s site: http://www.vmware.com/files/pdf/techpaper/vmw-hp-sap-vmware-performance-brief.pdf
If you go through that brief carefully you notice something strange at the very end. In the x86 space they are comparing a HP Proliant BL 460c G7 with a Fujitsu PRIMERGY RX300 S5 and show that tthe HP system scales to 32125 SAP SD users while the Fujitsu one can only scale to 16000 users. In the non-x86 space they compare with a IBM System p550 (Power6). Now I am not familiar with the specifics of IBM Power systems so, leaving that aside, let us focus on the x86 space. If we look at the two systems and configs side by side some things become striking:
|HP Proliant BL 460c G7||Fujitsu PRIMERGY RX300 S5|
|X5675 Westmere EP||X5570 Nehalem EP|
|2 CPU/12 Cores/24 Threads||2 CPU/8 Cores/16 Threads|
|12MB Cache||8MB Cache|
|ESX 5.0||ESX 4.0|
MS Windows Server 2008 R2
|SuSE Enterprise Server 10|
|APP Tier: 10 Servers||APP Tier: 7 Servers|
Now one can’t help but say WTF! This is in no way a balanced comparison. There are too many variables here. These folks are comparing a Nehalem EP 8MB Cache with a Westmere EP 12MB Cache and more cores. The game is half over there. Next being thing in the ESX version. HP’s purported leadership performance G7 is running ESX 5.0 while the competitor’s older server is running ESX 4.0. The game is lost here. Finally what made HP run SuSE on the Fujitsu box and not Windows Server 2008 R2 ? Why is a Dell box not included in the “Leadership Performance” tests ? Do the great benchmarkers expect people reading the doc to be smoking Ganja ?
Now there is one situation where a vendor’s boasts with their latest and greatest systems compared to the competition’s older systems can hold water. This is only under the condition that the competition does NOT have a comparable system on offer. That eminently is not the case here. The comparison could have been done with a Fujitsu PRIMERGY RX300 S6 which is a similar spec Westmere EP system and should have been running Windows. Now there may possibly be some other factor, I am not aware of, that made the benchmarkers choose the older PRIMERGY system. If that is true why is that not mentioned in the document ?