Compuware's NuMega DevPartner for Java

Short Line   MugMugMugMugMug
Version Reviewed: 2.0
Current Version: 2.0
Rating: 5 cups

Cup rating system:
5 Outstanding
4 Outstanding
3 Good
2 Acceptable
1 Poor

Vendor Info
Compuware Corp.
31440 Northwestern
Farmington Hills,
MI 48334

Product Review Product Review

Beginning with this issue, we are adjusting our product rating system. Both Reader and Evaluator feedback has lead us to conclude that four cups is just not enough. So, with this issue, we introduce a 5-cup rating system.

The new rating system is as follows:

5 Cups = Outstanding: A "must-have" product that not only does very well with what it is supposed to do, but in many ways goes above and beyond all expectations.

4 Cups = Very Good: A top-notch product that meets expectations and does very well with what it is supposed do.

3 Cups = Acceptable: A product that does its job adequately.

2 Cups = Has Potential: This product has the right idea or has its heart in the right place, but falls short on execution.

1 Cup = Poor: This product fails to do its job appropriately and would not be recommended for use.

John K. Waters
Product Editor

Numega IF YOU'VE EVER used program analysis and debugging tools, you probably know that many of them don't live up to their promises. In particular, these tools tend not to scale beyond simple, sample programs and may become close to unusable when confronted with an actual project of even moderate size. Simply put, this was not the case with DevPartner for Java.

DevPartner for Java is a suite of program analysis tools aimed at assisting in the testing, performance tuning, and debugging of Java programs. The suite comprises three tools: TrueTime, a performance analysis tool; TrueCoverage, a coverage analysis tool; JCheck, a tool for analyzing program execution and thread activity. The tools provide friendly GUIs, as well as command-line versions of TrueTime and TrueCoverage. The tools can integrate themselves into the development environments of Microsoft Visual J++ 6.0 and Symantec Café.3.x (integration was not evaluated). DevPartner works with Sun JVM version 1.1.7 or later (including Java 2.0/1.2.x), the Microsoft JVM version or later, and the VM shipped with Symantec Café 3.x. At press time, the tools were not yet compatible with the HotSpot performance engine recently released by Sun.

NuMega's DevPartner for Java is a suite of tools for the analysis of the execution of Java programs. The suite includes TrueTime; a performance analysis tool; TrueCoverage, a coverage analysis tool; and JCheck, a thread and program event analysis tool.

Price: The suite sells for $689. Printed documentation and support by phone are available separately for a fee.

System Requirements:
  • 32 MB of RAM on Windows 9x, 64 MB on Windows NT.

  • 24.5 MB of disk space for the complete suite, and an additional 200 MB free disk space for running JCheck.

  • Sun Microsystems' Java Virtual Machine v 1.1.7A, 1.1.7B, 1.1.8, 1.2, or 1.2.1; or the Microsoft VM versions 4.79.2339 through 5.00.3167. The VM bundled with Symantec Café 3.0 is also supported.

  • Optional: Microsoft SDK for Java 2.01 or later for displaying full symbolic information when using the Microsoft VM.

  • Optional: Microsoft J++ 6.0 or Symantec Café 2.0 for development environment integration.

Installing the suite was, for the most part, simple and painless. One heads-up: Each tool must be installed separately so that TrueTime and True Coverage end up in the same directory. Installable copies of Sun's JDK versions 1.1.7B and 1.2.1, the Microsoft SDK for Java version 3.2, and the Microsoft VM build 3167 come bundled with the suite.

The printed documentation that comes in the package consists of quick reference booklets for each of the tools and a "Getting Started" booklet for the Java suite. User manuals are included on the CD in the form of Adobe Acrobat PDF files. Printed manuals are available from NuMega at $15-$20 apiece. The documentation is the same as that for the Visual Basic and Visual C++ versions of the tools, and only small sections are dedicated to Java topics. There are no printed or printable reference manuals.

To put the DevPartner suite through some real-world paces, I tested the tools against a project of some complexity: the Jetty Web server from Mortbay Computing Pty. Ltd. (, a complete Web server with support for multiple hosts, multiple virtual addresses, Java Servlets, and Java Server Pages. At 30,000 lines of Java source code, the Jetty Web server is a sizable (medium-to-large) OO program. Another reason I chose the Jetty Web server was that it was "open source," so I could make changes to the source code and evaluate their impact with DevPartner.

I conducted the evaluation on a Pentium II 300 MHz, with 128 MB of RAM and several gigabytes of available IDE disk space. I used the Sun 1.2.1 VM for most of the evaluation; I used the 1.1.7A and 1.1.8 VMs in a few of the tests. For comparison purposes, I repeated some of the tests on a Pentium 133 MHz laptop, with 48 MB of RAM.

The core of the test consisted of stressing the Jetty Web server with a small stresser program contributed by Kent Johnson of Transparent Language Inc. The program launches a specified number of threads, each of which requests a list of documents from a given base URL. I ran the stress program on the same machine and over a LAN, running between 8 and 16 threads each time, each requesting 40 or so documents.

The DevPartner tools behaved well on all the tests, even on the lesser of the machines used.

Figure 1
TrueTime shows times and runtime percentages for each function used, and lets you quickly filter by Java package or module.

Modern JIT compilers have brought the performance of Java code close to that of traditionally compiled languages. Yet, Java programs may still present performance problems because of the complexity beneath the high level of abstraction of the Java class libraries, or the complexity of the programs we attempt to develop with the platform. Experienced programmers know that performance bottlenecks often occur in the least expected places, and that to really know where our code is spending its cycles we need the assistance of a profiling tool.

NuMega describes TrueTime as a "performance analysis tool," a title that fits the software much better than just "profiler," especially when you compare it to bare-bones tools like JPROF, which comes bundled with Sun's JDKs.

Among its most useful features is what NuMega calls "Quantum" technology, which lets TrueTime distinguish between time spent in threads of your application and time spent in threads of other running applications. TrueTime generates reproducible timing results even under the most diverse conditions. The timings produced by my tests with the Jetty Web server were all within a few percentage points of each other, even under a range of CPU load conditions, LAN traffic, and modem use.

TrueTime collects information about the number of calls and about minimum, maximum, average, and real (elapsed) execution times, not only for every Java method executed by your program, but also for each entry point to the DLLs and ActiveX controls used by the application. This functionality proved useful while testing the Jetty Web server. For example, the session data showed that, even though the time spent in the Java part of calls to java.lang.Object.wait() was insignificant, the time spent in the underlying call to the WaitForSingleObject() Windows API amounted to 24% of the total. This insight suggested that a new way to approach synchronization in the server's thread pool was required.

I found TrueTime very easy to use. To start a profiling session, you open the .class file that contains the main entry point for the application or the HTML file that runs the applet and tell TrueTime to start the program. TrueTime will instrument the program on-the-fly, run it, and start collecting information. While the application is running, a log window reports events like the start of system threads and the load of DLLs used by the application.

Once the application ends, TrueTime presents the gathered information in a two-pane "Session Window." The left pane of the window contains a tree with entries that serve as filters for the information shown. Four of the options filter the results to contain only the worst 20 performers or the 20 most-called Java and/or system functions. Another item contains the complete tree of Java packages used by the application. Filtering the data to a set of packages or to a single class is as simple as clicking on the tree.

The right pane of the Session window contains three tabs. The first tab shows function-by-function profiling information. The second tab shows the source code for the currently selected item (when available), along with line-by-line profiling information. The third tab provides a summary of the session, including information about which Java packages and DLLs were profiled, and the number of functions present in each. For the first two tabs, the user can choose what information to view by turning data columns on and off. Changing the sorting criteria is as simple as clicking on the column headers.

Figure 2
TrueTime's function details view shows how each function spent its time, and what the impact of a function has been on its callers.

Double-clicking on a function in the session pane pops up a details dialog for that function. The dialog shows which functions are called by the one on focus, which functions call it, and the number of calls and percentage of time spent in each case. Navigating to any of the functions shown is as simple as double-clicking on it. You can always back up and revisit already viewed information by pressing the "back" button, much how it's done in Web browsers. With the help of the Session window, I found that the Jetty Web server was spending 35% of its time inside the very inefficient java.lang.FilterOutputStream.write(byte[]). Using the Function Details dialog, I was able to quickly find which functions were calling methods of FilterOutputStream, how many times, and at what cost. With that information on hand, I was able to change a dozen or so lines in the Jetty source and improve overall server performance by 23%!

TrueTime lets you associate multiple runs (sessions) with a single "project." Selected session information can be exported to comma-delimited ASCII files for further processing with other tools. You can also exclude specific Java packages, directories, and system modules from instrumentation by listing them in the Program/Exclude dialog.

The tool can be used with several different JVMs, but the required configuration changes must be done by hand each time, which is an annoying and error-prone chore. One can work around this limitation by using the command-line version of the profiler along with batch files, but it would be nice to see a more developed version of the Project concept in future versions of the product.

DevPartner's TrueCoverage coverage analysis tool takes much of the pain out of checking for test coverage by hand.

For each run of a program, TrueCoverage records which functions were called and which lines in each function were executed. The program lets you accumulate coverage data from several executions of your application by merging the session files. This facility allows for simpler testing sessions, and is especially useful in interactive GUI applications for which testing can be man-hour intensive. For each session or set of merged sessions, TrueCoverage reports the number and percentage of functions and lines of code that were and were not executed.

The TrueCoverage user interface is similar to TrueTime's interface, so familiarity with one tool should make it a straightforward proposition to use the other. Like TrueTime, TrueCoverage's Session window has a left Filter pane that lets you filter the information displayed. There are predefined filters for functions not covered, functions covered to less than a given percentage, and functions belonging to a given package or class. Unlike TrueTime, TrueCoverage lets the user define new filtering criteria to be inserted into the Filter pane.

The TrueCoverage Session Data pane shows the number of times each function was called, as well as the coverage percentage and the number of lines still not-executed for each function. The source tab can display the source code for any selected item (as in TrueTime.) The tool uses colors to show which lines of code were or were not executed, and which lines correspond to unreachable or otherwise non-executable code. TrueCoverage lets users exclude complete packages and directories from analysis by listing them in the Program/Exclude dialog, on which the Java platform runtime libraries, and the standard libraries from Netscape and Symantec, are excluded by default.

As in TrueTime, the "project" concept is lacking in TrueCoverage. Although session merging is supported, there's no way to define and save the configuration information for the different test programs that are likely to comprise a complete test suite.

NuMega's JCheck is a thread viewer and program execution analyzer that provides a graphical view of a program's thread activity, a time-ordered log of program events, and a log of actual and potential problems detected during program execution. JCheck does not try to show program events in real time. Instead, JCheck queues the events to let the application run at a more natural pace. This lets the user continue monitoring detailed program activity, even long after the analyzed program has ended. JCheck is especially useful for debugging problems caused by thread activity, like contention, starvation, deadlocks, and livelocks. Running a Java program from within JCheck is as straightforward as it is from within TrueTime and TrueCoverage.

JCheck's Thread view presents an animation of a program's thread activity. Threads, thread groups, system threads, and synchronization objects are all represented with different symbols. Thread icons are connected to the corresponding thread group icons through thick lines, and arrows are used to show thread joins and pending waits over synchronization objects. A color box jumps from one thread to another, showing which thread is active at any given point (when running over a Java 1.2 VM, JCheck will even show the activity for the garbage collector's "Finalizer" and "Reference Handler" threads).

During the animation, labeled lines are used to show events, such as the start and joining of threads. The Thread view has settings to let the animation run to completion, or to have it pause after each threading event. Buttons for zoom-in, zoom-out, and "view all" make it easy to focus on the part of the animation that's interesting. Passing the mouse over any of the icons produces a pop-up window with details about the state of the item under the cursor. I was impressed with how informative this kind of thread animation can be. It would be impossible to "see" as much by using only ad-hoc means, such as console writes or debugging logs. Despite its usefulness, I wished that I could have replayed the animation from the Thread view without having to run the application again. Being able to control the replay speed, and select which subset of the sequence to play, would have made the animator truly outstanding.

During program execution, JCheck records every application event, from thread starts, stops, suspends and resumes, to class loads and down to every single method call. The Session window provides two views of this event data. The Results view displays information about errors, leaks, exceptional conditions, and advisories about potential problems in the code. Livelock, Deadlock, catching ThreadDeath, thread leaks, and joining of a thread to itself are some of the problems that JCheck automatically detects. Clicking on any of the items in the Results view shows an explanation of the problem, including details about the event and the contents of the call stack at the point of execution. At the same time, the source code pane shows the line in the source code from which the event emanated. Additional information about problems and potential problems can be obtained by selecting "Explain" from the help menu.

The second view provided by the Session window is the Transcript view. This view shows a time-ordered sequence of the program events, organized into a hierarchy that shows their runtime context dependencies. Thread starts and resumes are shown at the top level, and the nested sequence of method calls, errors, and exceptions that occurred while the thread was awake are shown as children. The transcript can be filtered to show only method calls, class loads, exceptions, errors, or application-generated bookmarks—or any combination of them. Clicking on any of the items provides detailed information about the event. The source pane shows the source code line where the event was generated.

NuMega provides a Java class library that provides the means to pause, resume, and bookmark the event stream programmatically from within your application. Running the Jetty Web server under JCheck was very informative, and I encountered no major problems.

Although I ran into some minor usability glitches in the DevPartner tools suite, they were nothing you wouldn't expect from a 1.x version. For example, the tools' lack of smarts about Java package-to-directory mappings generated too many prompts requesting information about the location of source code modules. Also, the list of paths for source and class files must be specified in a plain text box, something which quickly becomes unwieldy with non-trivial projects.

A printed reference manual would be welcome in a future version. At a minimum, it should contain information about the contents and lifetimes of the different files produced by the tools, and it should make explicit the mysterious interactions between TrueTime and TrueCoverage configuration and project files. Things like what exactly needs to be input in the class exclusion dialog also need better documentation.

Finally, I'd like to see better support for the concept of a "project." At least there should be a way to configure and save different VM/main-class combinations for repeated use or abuse. The ability to add annotations to projects, sessions, and session items would also be very welcome.

In virtually all other regards, the DevPartner suite performed beyond expectations, and the tools were very simple to use yet quite powerful. Don't deploy your Java code without analyzing it first with a tool suite like DevPartner.

Juancarlo Añez is the director of IT at Modelistica in Caracas, Venezuela. He can be contacted at [email protected].

Quantity reprints of this article can be purchased by phone: 717.560.2001, ext.39 or by email: [email protected].