News

NEWS ANALYSIS: Lies, damned lies and benchmarks?

• By Gary Barnett
• July 15, 2002

There are few things more pitiful than to listen to a vendor whining about how benchmark ''Y'' is ''fixed,'' then to hear the same vendor use the next breath to proclaim its superiority in (you guessed it!) benchmark ''X.''

Curiously, there's one set of benchmarks that no one seems to want to talk about anymore -- The TPC (www.tpc.org). These performance benchmarks were once the darling of vendors like Oracle and Sun. The problem now is that the top TPC benchmark products are dominated by Microsoft.

Naturally, it stands to reason that if Microsoft's technology can take, when I last looked, positions 1, 2, 3, 5, 6 and 9 in an all-out performance benchmark (TPC-C), then there must be quite a problem if you're Oracle or Sun. It gets even more unpleasant if you look at the price performance rankings for the same benchmark -- OOPS! The boys from Seattle scoop every single one of the top 10 price performance slots. Well, naturally you'd smell a rat wouldn't you? But the real shocker is that Microsoft didn't submit these results -- IBM, Compaq and Dell did.

But there's great news for those Java app servers, and for the rest of those non-Microsoft databases feeling a little left out!

ECperf (http://ecperf.theserverside.com/ecperf/) is on hand, specifically designed to test J2EE application servers. There's also a comprehensive document explaining ECperf entitled Eight Reasons ECperf is the Right Way to Evaluate J2EE Performance (http://ecperf.theserverside.com/articles/article.tss?l=Why_ECperf).

There is a ninth reason why vendors might want you to look at ECperf, which I am surprised wasn't mentioned. Perhaps it goes without saying the ninth reason is ''Microsoft doesn't have a J2EE app server.''

Another interesting tidbit is found in this document. It seems database vendors that brag about their ECperf benchmark scores are misleading us just a little bit; ECperf isn't actually designed to measure database performance! The ''Eight Reasons'' document explains in Section 3.3 (and I quote verbatim): ''ECperf is not designed to test DBMS performance and scalability. These are adequately measured by other standard workloads such as TPC-C, TPC-D and TPC-W[2].'' Now I'm sure that this is an oversight on the part of the braggers. After all, they would never seek to deliberately mislead us would they?

Don't you think that if the Java app server and database vendors were so sure they could beat Microsoft in price/performance they'd pick a field on which Microsoft could play? Unless of course ... but no, that alternative is too awful to consider.