Columns
Road testing for the year 2000 -- a tale of two tools
Searching for useful year 2000 tools can be like
Diogenes looking for an honest man -- too many promises.
One product that promises much, but does not deliver, is OnMark 2000
Workbench from Viasoft Inc., Phoenix. According to Viasoft, the product will
"help answer" four questions: What will it cost to bring my code into
compliance? How long will it take? When will I be done? and What do I give my
auditors?
The documentation claims that "the Impact Analysis section of the tool
parses C/C++ program source code to identify suspected date variables based on
user-modifiable variable name lists." However, this is not accurate based
on the common use of the term "parse" among computer scientists. The
core of the Impact Analysis tool is a pattern matcher, and not a very good one
at that.
The Workbench text-searching engine includes a list of strings Viasoft
deems as dangerous code. For example, OnMark 2000 Workbench considers the SQL
command "update" a year 2000 problem -- no matter where it occurs.
This is because the pattern matcher makes no attempt to preprocess the source,
and apparently cannot tell a literal code string from a variable code string.
For power, the text-searching engine is not as good as those found in free
packages such as grep, awk and Perl. Ease of use is also questionable, as
patterns are edited directly with little UI support. OnMark 2000 Workbench also
takes hours to search even moderately large source code trees. Other search
engines can perform similar searches in minutes.
The documentation further claims that "[OnMark] also tracks any
variables that are dependent on these known or suspected date variables. The
tool stores the code in its source-code database and generates statistics
describing the scope of the year 2000 problem." As noted earlier, OnMark
2000 Workbench cannot parse the program reliably; therefore the program cannot
track variable dependencies. Dependencies in C/C++ demand dynamic analysis, as
many features (like casting) conspire to thwart complete static analysis.
OnMark's source-code database and statistics are classic examples of
"garbage in, garbage out." As far as describing the scope of the year
2000 problem, OnMark routinely reported that we had more "suspect date
variables" than our project had variables.
The tool's Project Estimator is a simple spreadsheet: provide the cost of
labor and how much labor is required, and OnMark 2000 Workbench does the
multiplication and reports how much money the project will cost. The Compliance
Editor is also lacking in functionality. Using color highlighting, OnMark
quickly convinces users of how little help it is in identifying true year 2000
problems.
The Project Plan Generator represents the most significant piece of
programming in the entire package. Yet, it amounts to a Microsoft Project
"year 2000 wizard." Information entered in other modules can be used
to jumpstart a Microsoft Project document that describes the projects.
Unfortunately, the Project Plan Generator also suffers from weak analysis. Bad
numbers, but nicely formatted.
Finally, OnMark 2000 Workbench provides a small (1,700 lines of code) C
conversion library for converting dates from two- to four-digits based on a
"sliding window" conversion (years after 80 are 20th century, years
before 80 are 21st century). I give Viasoft credit for providing this in source
code -- you can at least verify what the software does by looking at it.
However, most C++ programmers won't give it a second look, and C programmers
will have to make it fit project standards.
IBM'S CMTT
In many ways, IBM's C and C++ Maintenance and Test Toolsuite (CMTT) is just
the opposite of OnMark 2000 Workbench -- it provides more content than it
promises. Although its component Y2K tool is a disappointment, its solid set of
tools could be useful for testing well beyond the year 2000.
CMTT comprises the following:
- ATAC -- a powerful code coverage analyzer.
- xregress -- analyzes code coverage traces to determine the minimal test
sets needed to achieve desired coverage.
- xvue -- computes dynamic slice information for test maintenance purposes
based on ATAC-generated coverage information.
- xslice -- computes dynamic slice information for debugging based on
ATAC-generated coverage information.
- xprof -- performance profiler.
- xFind -- pattern match-based search tool; the main claim to Y2K test
functionality.
- xdiff -- text-comparison tool for finding differences among programs.
Each of these tools can be used from the command line or from the graphical
shell. But NT users be warned: CMTT offers little concession to Windows
programming conventions. Maximize the application and you will think you've
been warped to a Solaris box. And while CMTT documentation writers like to laud
their "state-of-the-art graphical user interface," I personally think
competitive products have advanced beyond CMTT.
The heart and soul of CMTT is Automatic Test Analysis for C (ATAC). If you
buy this package, buy it for ATAC. If you do not buy it, tell your code
coverage analyzer that you want all of these features anyway.
Code coverage analyzers answer a simple question: When I run a program, how
much of the program was actually executed? This is a vital question when
evaluating a test plan. A test plan that executes 100% of a program is not
necessarily good, but a test plan that covers less than 100% of a program is
bad. This is because any line of code that is not tested, or proven correct,
can cause catastrophic failure of the program (just think of the line
"exit(0)" randomly placed through the program). In my experience,
testing efforts that do not incorporate coverage analysis rarely exceed 20%
coverage of the program under test. Is it any wonder that these programs fail
in the hands of users?
 |
| xvue, part of IBM's CMTT product, computes dynamic slice
information for test maintenance purposes based on ATAC-generated coverage
information.OnMark 2000 |
Like many code coverage tools, you must adjust your make files to use it.
ATAC must also be invoked during compilation and linking in order to use all of
its features. It has no way (as some newer Windows-based coverage tools do) to
analyze programs that have not been specially prepared in this way. Personally,
I prefer the instrument-and-go approach of tools like Rational Software's
Visual Coverage.
One thing that distinguishes CMTT from the competitive coverage analyzers I
have used is its support of a kind of code-coverage algebra. Most coverage
analyzers say, "There's your trace; you figure out what it means."
CMTT lets users manage and analyze traces in interesting ways. For example,
traces can be named and costs assigned (such as the length of time needed to
run). CMTT can then compare the traces and tell you what set of tests will give
you the maximum coverage at the lowest cost. This lets users prune a runaway
test set. Personally, I have never seen this problem. In fact, I would love to
have this problem. When your testing efforts generate so many tests that you
cannot run them all, CMTT is ready.
Another slick feature is the ability to compute and show trace differences
graphically. By comparing the traces of tests that use a particular feature
(called invoking tests) and other tests that do not use that feature (called
excluding tests), a user can quickly focus on code that is used by that
feature. This is useful for maintaining large systems where the objective is to
modify a feature without breaking the system in some other way.
Trace comparisons can also be used to drill into complex debugging issues.
If you have some test runs that demonstrate a difficult to understand defect,
and some test runs that do not exhibit this defect, CMTT can compare the traces
and quickly show you the difference between the test runs.
If you do not use a code coverage analyzer, you need to. Compared to other
coverage analyzers, ATAC has all the steak but comes up short on sizzle. If you
are using a command line, character-mode coverage analyzer, CMTT will look
pretty cool. If you are already using Visual Coverage, I doubt you are going to
switch.
CMTT'S XPROF, XFIND AND XDIFF
xprof is CMTT's profiler. It does not attempt to use time as a measure, but
instead focuses on counters. Where coverage generally asks the question
"Was this done at least once?" xprof provides the answer to the
question "How many times was it done?" This can be precisely the
information that is needed to hone in on the best optimization strategies.
xprof is a competent, but not compelling profiler. As a "pack
in," it provides a crucial capability that may be enough for some testing
projects. As a standalone program it would not fare too well against
competitive profilers from Rational and NuMega. The execution counts are
useful, but not to the exclusion of timing information. The graphical display
is helpful, but it does not provide a fast way to sift through mountains of
information to focus on key issues.
According to the documentation: "xFind is a Year-2000 static analysis
tool in the Toolsuite. Find supports the identification of date-sensitive
objects through a simple transitive relation. Date-sensitive patterns are
initially specified which are common across the application. The atoms (groups
of characters, surrounded by white space) in the code that match each pattern
are then either accepted or rejected. Acceptance or rejection can be selected
in either local file or global scope. Once the initial atoms are selected, the
transitive relation is invoked, which identifies more candidates for
date-sensitive objects."
While the honest appraisal is refreshing, the end results are only a little
more satisfying than with OnMark 2000 Workbench. Text searches are inherently
limited and often require complex ad hoc reasoning beyond the capabilities of
xFind. In my experience, a full programming language is needed to make any
sense of such searches (for example, Perl). xFind is at least a well-written
search engine -- it can troll through million-line projects in a matter of
minutes. Given that xFind does not "understand" the syntax or
semantics of C++, its application of transitive relationships often ends up
marking large, irrelevant sections of code. It is disappointing that while ATAC
appears to be capable of parsing C++ code, xFind does not take advantage of it.
xdiff is a useful source difference analyzer, but by now these tools are
commonplace and there is little in xdiff to compel a user to switch tools. The
two programs are displayed side by side, with differences called out in color.
The user is allowed to help the program maintain synchronization in the
comparison. If you are still using a command line diff, you might find this
very helpful.
The designers of CMTT have a clear understanding of testing problems and
useful ways to attack these problems. What they do not understand is the
Windows interface and how to design for it. Ultimately, CMTT's tools are not
likely to enjoy much success among Windows programmers until the user interface
is revamped. Unix users will feel more at home with the command line and
Solaris look and feel. However, the tools are reasonably priced for the value
they provide; you may want to acquire a license if only to supplement your
mainstay tools.
Serious testing projects, for the year 2000 and beyond, demand the kind of
serious test tools found in CMTT. While CMTT deserves to make your short list
of considered tools, it doesn't (yet) make the top of the list.