NEWS ANALYSIS: Lies, damned lies and benchmarks?
- By Gary Barnett
There are few things more pitiful than to listen to a vendor whining about
how benchmark ''Y'' is ''fixed,'' then to hear the same vendor use the next
breath to proclaim its superiority in (you guessed it!) benchmark ''X.''
Curiously, there's one set of benchmarks that no one seems to want to talk
about anymore -- The TPC (www.tpc.org). These
performance benchmarks were once the darling of vendors like Oracle and Sun. The
problem now is that the top TPC benchmark products are dominated by Microsoft.
Naturally, it stands to reason that if Microsoft's technology can take, when
I last looked, positions 1, 2, 3, 5, 6 and 9 in an all-out performance benchmark
(TPC-C), then there must be quite a problem if you're Oracle or Sun. It gets
even more unpleasant if you look at the price performance rankings for the same
benchmark -- OOPS! The boys from Seattle scoop every single one of the top 10
price performance slots. Well, naturally you'd smell a rat wouldn't you? But the
real shocker is that Microsoft didn't submit these results -- IBM, Compaq and
But there's great news for those Java app servers, and for the rest of those
non-Microsoft databases feeling a little left out!
is on hand, specifically designed to test J2EE application servers. There's also
a comprehensive document explaining ECperf entitled Eight Reasons ECperf is
the Right Way to Evaluate J2EE Performance (http://ecperf.theserverside.com/articles/article.tss?l=Why_ECperf).
There is a ninth reason why vendors might want you to look at ECperf, which I
am surprised wasn't mentioned. Perhaps it goes without saying the ninth reason
is ''Microsoft doesn't have a J2EE app server.''
Another interesting tidbit is found in this document. It seems database
vendors that brag about their ECperf benchmark scores are misleading us just a
little bit; ECperf isn't actually designed to measure database performance! The
''Eight Reasons'' document explains in Section 3.3 (and I quote verbatim):
''ECperf is not designed to test DBMS performance and scalability. These are
adequately measured by other standard workloads such as TPC-C, TPC-D and
TPC-W.'' Now I'm sure that this is an oversight on the part of the braggers.
After all, they would never seek to deliberately mislead us would they?
Don't you think that if the Java app server and database vendors were so sure
they could beat Microsoft in price/performance they'd pick a field on which
Microsoft could play? Unless of course ... but no, that alternative is too awful
Gary Barnett is IT research director at Ovum Ltd., a United Kingdom-based consulting firm.