Sloppy Test Survival Guide (Part I)

• By Matt Stephens
• November 14, 2005

I just ran across this neat (if rather short) secure code test for Java programmers. It isn’t security of the "What’s the difference between a keystore and a trust store?" variety; but instead is about the basic aspects of Java language security that it’s sometimes easy to forget (remembering to make variables private; using CloneNotSupportedException and so on).

While the test is worth the few minutes needed to run through it, I did have a couple of (minor) reservations about it, which then got me thinking about the role of such tests in the industry.

It was obvious that the test was written by someone for whom English is not their first language. This made one or two of the questions slightly ambiguous. It was possible to work out what he meant, but for me it highlighted one of the problems I’ve always had with tests like these: if a question has odd wording, trying to work out if it’s a trick question or if the author just wrote the question carelessly can mean the difference between a right and a wrong answer. When the test generally has the feel of being sloppily written, sometimes you just have to guess one way or the other... which lessens the test’s value overall.

The situation is made worse when a test is for something that really matters, like an IT certification exam, or worse... imagine you’re at a job interview, made to sit in a room for an hour and do a multiple-choice programming aptitude test. Instead it becomes a frustrating hour of multiple-choice guesswork because the questions are so poorly worded. (The "survival guide" alluded to in this blog entry’s title is, simply, this: If faced with a situation like this... stand up, leave the room, and exit the building. You can virtually guarantee that any potential employer who puts you through such an ordeal of ambiguity and sloppiness really won’t be worth the angst of working for. Trust your early warning detector, cut the interview short, and get home in time to catch the last 10 minutes of Dora the Explorer).

Imagine that one question discusses checking whether two objects are "the same". In Java, you can have two references to the same object, or two objects which are "equal" (entirely different concepts). If you have confidence in the test’s author, you can happily surmise that he really does mean "same" and not "equal to". But if he’s already managed to undermine your confidence, it’s easy to assume that, although he might have written "the same", he might really have meant "equal to"... or did he?

More about this topic in a couple of days (stay tuned!)