Columns

Road testing for the year 2000 -- a tale of two tools

Searching for useful year 2000 tools can be like Diogenes looking for an honest man -- too many promises.

One product that promises much, but does not deliver, is OnMark 2000 Workbench from Viasoft Inc., Phoenix. According to Viasoft, the product will "help answer" four questions: What will it cost to bring my code into compliance? How long will it take? When will I be done? and What do I give my auditors?

The documentation claims that "the Impact Analysis section of the tool parses C/C++ program source code to identify suspected date variables based on user-modifiable variable name lists." However, this is not accurate based on the common use of the term "parse" among computer scientists. The core of the Impact Analysis tool is a pattern matcher, and not a very good one at that.

The Workbench text-searching engine includes a list of strings Viasoft deems as dangerous code. For example, OnMark 2000 Workbench considers the SQL command "update" a year 2000 problem -- no matter where it occurs. This is because the pattern matcher makes no attempt to preprocess the source, and apparently cannot tell a literal code string from a variable code string.

For power, the text-searching engine is not as good as those found in free packages such as grep, awk and Perl. Ease of use is also questionable, as patterns are edited directly with little UI support. OnMark 2000 Workbench also takes hours to search even moderately large source code trees. Other search engines can perform similar searches in minutes.

The documentation further claims that "[OnMark] also tracks any variables that are dependent on these known or suspected date variables. The tool stores the code in its source-code database and generates statistics describing the scope of the year 2000 problem." As noted earlier, OnMark 2000 Workbench cannot parse the program reliably; therefore the program cannot track variable dependencies. Dependencies in C/C++ demand dynamic analysis, as many features (like casting) conspire to thwart complete static analysis. OnMark's source-code database and statistics are classic examples of "garbage in, garbage out." As far as describing the scope of the year 2000 problem, OnMark routinely reported that we had more "suspect date variables" than our project had variables.

The tool's Project Estimator is a simple spreadsheet: provide the cost of labor and how much labor is required, and OnMark 2000 Workbench does the multiplication and reports how much money the project will cost. The Compliance Editor is also lacking in functionality. Using color highlighting, OnMark quickly convinces users of how little help it is in identifying true year 2000 problems.

The Project Plan Generator represents the most significant piece of programming in the entire package. Yet, it amounts to a Microsoft Project "year 2000 wizard." Information entered in other modules can be used to jumpstart a Microsoft Project document that describes the projects. Unfortunately, the Project Plan Generator also suffers from weak analysis. Bad numbers, but nicely formatted.

Finally, OnMark 2000 Workbench provides a small (1,700 lines of code) C conversion library for converting dates from two- to four-digits based on a "sliding window" conversion (years after 80 are 20th century, years before 80 are 21st century). I give Viasoft credit for providing this in source code -- you can at least verify what the software does by looking at it. However, most C++ programmers won't give it a second look, and C programmers will have to make it fit project standards.

IBM'S CMTT

In many ways, IBM's C and C++ Maintenance and Test Toolsuite (CMTT) is just the opposite of OnMark 2000 Workbench -- it provides more content than it promises. Although its component Y2K tool is a disappointment, its solid set of tools could be useful for testing well beyond the year 2000.

CMTT comprises the following:

  • ATAC -- a powerful code coverage analyzer.
  • xregress -- analyzes code coverage traces to determine the minimal test sets needed to achieve desired coverage.
  • xvue -- computes dynamic slice information for test maintenance purposes based on ATAC-generated coverage information.
  • xslice -- computes dynamic slice information for debugging based on ATAC-generated coverage information.
  • xprof -- performance profiler.
  • xFind -- pattern match-based search tool; the main claim to Y2K test functionality.
  • xdiff -- text-comparison tool for finding differences among programs.

Each of these tools can be used from the command line or from the graphical shell. But NT users be warned: CMTT offers little concession to Windows programming conventions. Maximize the application and you will think you've been warped to a Solaris box. And while CMTT documentation writers like to laud their "state-of-the-art graphical user interface," I personally think competitive products have advanced beyond CMTT.

The heart and soul of CMTT is Automatic Test Analysis for C (ATAC). If you buy this package, buy it for ATAC. If you do not buy it, tell your code coverage analyzer that you want all of these features anyway.

Code coverage analyzers answer a simple question: When I run a program, how much of the program was actually executed? This is a vital question when evaluating a test plan. A test plan that executes 100% of a program is not necessarily good, but a test plan that covers less than 100% of a program is bad. This is because any line of code that is not tested, or proven correct, can cause catastrophic failure of the program (just think of the line "exit(0)" randomly placed through the program). In my experience, testing efforts that do not incorporate coverage analysis rarely exceed 20% coverage of the program under test. Is it any wonder that these programs fail in the hands of users?

xvue, part of IBM's CMTT product, computes dynamic slice information for test maintenance purposes based on ATAC-generated coverage information.OnMark 2000

Like many code coverage tools, you must adjust your make files to use it. ATAC must also be invoked during compilation and linking in order to use all of its features. It has no way (as some newer Windows-based coverage tools do) to analyze programs that have not been specially prepared in this way. Personally, I prefer the instrument-and-go approach of tools like Rational Software's Visual Coverage.

One thing that distinguishes CMTT from the competitive coverage analyzers I have used is its support of a kind of code-coverage algebra. Most coverage analyzers say, "There's your trace; you figure out what it means." CMTT lets users manage and analyze traces in interesting ways. For example, traces can be named and costs assigned (such as the length of time needed to run). CMTT can then compare the traces and tell you what set of tests will give you the maximum coverage at the lowest cost. This lets users prune a runaway test set. Personally, I have never seen this problem. In fact, I would love to have this problem. When your testing efforts generate so many tests that you cannot run them all, CMTT is ready.

Another slick feature is the ability to compute and show trace differences graphically. By comparing the traces of tests that use a particular feature (called invoking tests) and other tests that do not use that feature (called excluding tests), a user can quickly focus on code that is used by that feature. This is useful for maintaining large systems where the objective is to modify a feature without breaking the system in some other way.

Trace comparisons can also be used to drill into complex debugging issues. If you have some test runs that demonstrate a difficult to understand defect, and some test runs that do not exhibit this defect, CMTT can compare the traces and quickly show you the difference between the test runs.

If you do not use a code coverage analyzer, you need to. Compared to other coverage analyzers, ATAC has all the steak but comes up short on sizzle. If you are using a command line, character-mode coverage analyzer, CMTT will look pretty cool. If you are already using Visual Coverage, I doubt you are going to switch.

CMTT'S XPROF, XFIND AND XDIFF

xprof is CMTT's profiler. It does not attempt to use time as a measure, but instead focuses on counters. Where coverage generally asks the question "Was this done at least once?" xprof provides the answer to the question "How many times was it done?" This can be precisely the information that is needed to hone in on the best optimization strategies.

xprof is a competent, but not compelling profiler. As a "pack in," it provides a crucial capability that may be enough for some testing projects. As a standalone program it would not fare too well against competitive profilers from Rational and NuMega. The execution counts are useful, but not to the exclusion of timing information. The graphical display is helpful, but it does not provide a fast way to sift through mountains of information to focus on key issues.

According to the documentation: "xFind is a Year-2000 static analysis tool in the Toolsuite. Find supports the identification of date-sensitive objects through a simple transitive relation. Date-sensitive patterns are initially specified which are common across the application. The atoms (groups of characters, surrounded by white space) in the code that match each pattern are then either accepted or rejected. Acceptance or rejection can be selected in either local file or global scope. Once the initial atoms are selected, the transitive relation is invoked, which identifies more candidates for date-sensitive objects."

While the honest appraisal is refreshing, the end results are only a little more satisfying than with OnMark 2000 Workbench. Text searches are inherently limited and often require complex ad hoc reasoning beyond the capabilities of xFind. In my experience, a full programming language is needed to make any sense of such searches (for example, Perl). xFind is at least a well-written search engine -- it can troll through million-line projects in a matter of minutes. Given that xFind does not "understand" the syntax or semantics of C++, its application of transitive relationships often ends up marking large, irrelevant sections of code. It is disappointing that while ATAC appears to be capable of parsing C++ code, xFind does not take advantage of it.

xdiff is a useful source difference analyzer, but by now these tools are commonplace and there is little in xdiff to compel a user to switch tools. The two programs are displayed side by side, with differences called out in color. The user is allowed to help the program maintain synchronization in the comparison. If you are still using a command line diff, you might find this very helpful.

The designers of CMTT have a clear understanding of testing problems and useful ways to attack these problems. What they do not understand is the Windows interface and how to design for it. Ultimately, CMTT's tools are not likely to enjoy much success among Windows programmers until the user interface is revamped. Unix users will feel more at home with the command line and Solaris look and feel. However, the tools are reasonably priced for the value they provide; you may want to acquire a license if only to supplement your mainstay tools.

Serious testing projects, for the year 2000 and beyond, demand the kind of serious test tools found in CMTT. While CMTT deserves to make your short list of considered tools, it doesn't (yet) make the top of the list.