| |
| |
|
 |
|
Version Reviewed: 2.0
Current Version: 2.0
Rating: 5 cups
Cup rating system:
5 Outstanding
4 Outstanding
3 Good
2 Acceptable
1 Poor
Vendor Info
Compuware Corp.
31440 Northwestern
Farmington Hills,
MI 48334
248.737.7300
www.compuware.com
|
Beginning with this issue, we are adjusting our product rating system. Both Reader and Evaluator
feedback has lead us to conclude that four cups is just not enough. So, with this issue, we introduce a
5-cup rating system.
The new rating system is as follows:
5 Cups = Outstanding: A "must-have" product that not only does very well with what it is
supposed to do, but in many ways goes above and beyond all expectations.
4 Cups = Very Good: A top-notch product that meets expectations and does very well with what
it is supposed do.
3 Cups = Acceptable: A product that does its job adequately.
2 Cups = Has Potential: This product has the right idea or has its heart in the right place,
but falls short on execution.
1 Cup = Poor: This product fails to do its job appropriately and would not be recommended
for use.
John K. Waters
Product Editor
IF YOU'VE EVER used program analysis and debugging tools, you probably know that many of them
don't live up to their promises. In particular, these tools tend not to scale beyond simple, sample
programs and may become close to unusable when confronted with an actual project of even moderate
size. Simply put, this was not the case with DevPartner for Java.
DevPartner for Java is a suite of program analysis tools aimed at assisting in the testing, performance
tuning, and debugging of Java programs. The suite comprises three tools: TrueTime, a performance
analysis tool; TrueCoverage, a coverage analysis tool; JCheck, a tool for analyzing program execution
and thread activity. The tools provide friendly GUIs, as well as command-line versions of TrueTime and
TrueCoverage. The tools can integrate themselves into the development environments of Microsoft Visual
J++ 6.0 and Symantec Café.3.x (integration was not evaluated). DevPartner works with Sun JVM
version 1.1.7 or later (including Java 2.0/1.2.x), the Microsoft JVM version 4.79.0.2339 or later,
and the VM shipped with Symantec Café 3.x. At press time, the tools were not yet compatible with the
HotSpot performance engine recently released by Sun.
NuMega's DevPartner for Java is a suite of tools for the analysis of the execution of Java programs.
The suite includes TrueTime; a performance analysis tool; TrueCoverage, a coverage analysis tool;
and JCheck, a thread and program event analysis tool.
Price: The suite sells for $689. Printed documentation and support by phone are available separately
for a fee.
System Requirements:
- 32 MB of RAM on Windows 9x, 64 MB on Windows NT.
- 24.5 MB of disk space for the complete suite, and an additional 200 MB free disk space for running
JCheck.
- Sun Microsystems' Java Virtual Machine v 1.1.7A, 1.1.7B, 1.1.8, 1.2, or 1.2.1; or the Microsoft
VM versions 4.79.2339 through 5.00.3167. The VM bundled with Symantec Café 3.0 is also supported.
- Optional: Microsoft SDK for Java 2.01 or later for displaying full symbolic information when
using the Microsoft VM.
- Optional: Microsoft J++ 6.0 or Symantec Café 2.0 for development environment integration.
|
INSTALLATION
Installing the suite was, for the most part, simple and painless. One heads-up: Each tool must be
installed separately so that TrueTime and True Coverage end up in the same directory. Installable
copies of Sun's JDK versions 1.1.7B and 1.2.1, the Microsoft SDK for Java version 3.2, and the
Microsoft VM build 3167 come bundled with the suite.
The printed documentation that comes in the package consists of quick reference booklets for each of
the tools and a "Getting Started" booklet for the Java suite. User manuals are included on the CD
in the form of Adobe Acrobat PDF files. Printed manuals are available from NuMega at $15-$20 apiece.
The documentation is the same as that for the Visual Basic and Visual C++ versions of the tools,
and only small sections are dedicated to Java topics. There are no printed or printable reference manuals.
TEST PARAMETERS
To put the DevPartner suite through some real-world paces, I tested the tools against a project of
some complexity: the Jetty Web server from Mortbay Computing Pty. Ltd.
(http://www.mortbay.com/), a complete Web server with support for
multiple hosts, multiple virtual addresses, Java Servlets, and Java Server Pages. At 30,000 lines of
Java source code, the Jetty Web server is a sizable (medium-to-large) OO program. Another reason I
chose the Jetty Web server was that it was "open source," so I could make changes to the source code
and evaluate their impact with DevPartner.
I conducted the evaluation on a Pentium II 300 MHz, with 128 MB of RAM and several gigabytes of
available IDE disk space. I used the Sun 1.2.1 VM for most of the evaluation; I used the 1.1.7A
and 1.1.8 VMs in a few of the tests. For comparison purposes, I repeated some of the tests on a Pentium
133 MHz laptop, with 48 MB of RAM.
The core of the test consisted of stressing the Jetty Web server with a small stresser program
contributed by Kent Johnson of Transparent Language Inc. The program launches a specified number
of threads, each of which requests a list of documents from a given base URL. I ran the stress
program on the same machine and over a LAN, running between 8 and 16 threads each time, each
requesting 40 or so documents.
The DevPartner tools behaved well on all the tests, even on the lesser of the machines used.
TrueTime shows times and runtime percentages for each function used, and lets you quickly filter
by Java package or module.
TRUETIME
Modern JIT compilers have brought the performance of Java code close to that of traditionally
compiled languages. Yet, Java programs may still present performance problems because of the
complexity beneath the high level of abstraction of the Java class libraries, or the complexity
of the programs we attempt to develop with the platform. Experienced programmers know that
performance bottlenecks often occur in the least expected places, and that to really know where
our code is spending its cycles we need the assistance of a profiling tool.
NuMega describes TrueTime as a "performance analysis tool," a title that fits the software much
better than just "profiler," especially when you compare it to bare-bones tools like JPROF, which
comes bundled with Sun's JDKs.
Among its most useful features is what NuMega calls "Quantum" technology, which lets TrueTime
distinguish between time spent in threads of your application and time spent in threads of other
running applications. TrueTime generates reproducible timing results even under the most diverse
conditions. The timings produced by my tests with the Jetty Web server were all within a few
percentage points of each other, even under a range of CPU load conditions, LAN traffic, and modem use.
TrueTime collects information about the number of calls and about minimum, maximum, average,
and real (elapsed) execution times, not only for every Java method executed by your program,
but also for each entry point to the DLLs and ActiveX controls used by the application. This
functionality proved useful while testing the Jetty Web server. For example, the session data
showed that, even though the time spent in the Java part of calls to
java.lang.Object.wait() was insignificant, the time spent in the
underlying call to the WaitForSingleObject() Windows API
amounted to 24% of the total. This insight suggested that a new way to approach synchronization
in the server's thread pool was required.
I found TrueTime very easy to use. To start a profiling session, you open the .class file that
contains the main entry point for the application or the HTML file that runs the applet and
tell TrueTime to start the program. TrueTime will instrument the program on-the-fly, run it,
and start collecting information. While the application is running, a log window reports events
like the start of system threads and the load of DLLs used by the application.
Once the application ends, TrueTime presents the gathered information in a two-pane "Session Window."
The left pane of the window contains a tree with entries that serve as filters for the information
shown. Four of the options filter the results to contain only the worst 20 performers or the 20
most-called Java and/or system functions. Another item contains the complete tree of Java packages
used by the application. Filtering the data to a set of packages or to a single class is as simple
as clicking on the tree.
The right pane of the Session window contains three tabs. The first tab shows function-by-function
profiling information. The second tab shows the source code for the currently selected item
(when available), along with line-by-line profiling information. The third tab provides a
summary of the session, including information about which Java packages and DLLs were profiled,
and the number of functions present in each. For the first two tabs, the user can choose what information
to view by turning data columns on and off. Changing the sorting criteria is as simple as clicking on
the column headers.
TrueTime's function details view shows how each function spent its time, and what the impact of a
function has been on its callers.
Double-clicking on a function in the session pane pops up a details dialog for that function.
The dialog shows which functions are called by the one on focus, which functions call it, and
the number of calls and percentage of time spent in each case. Navigating to any of the functions
shown is as simple as double-clicking on it. You can always back up and revisit already viewed
information by pressing the "back" button, much how it's done in Web browsers. With the help of
the Session window, I found that the Jetty Web server was spending 35% of its time inside the
very inefficient java.lang.FilterOutputStream.write(byte[]).
Using the Function Details dialog, I was able to quickly find which functions were calling
methods of FilterOutputStream, how many times, and at what cost.
With that information on hand, I was able to change a dozen or so lines in the Jetty source and
improve overall server performance by 23%!
TrueTime lets you associate multiple runs (sessions) with a single "project." Selected session
information can be exported to comma-delimited ASCII files for further processing with other
tools. You can also exclude specific Java packages, directories, and system modules from
instrumentation by listing them in the Program/Exclude dialog.
The tool can be used with several different JVMs, but the required configuration changes must be
done by hand each time, which is an annoying and error-prone chore. One can work around this
limitation by using the command-line version of the profiler along with batch files, but it
would be nice to see a more developed version of the Project concept in future versions of
the product.
TRUE COVERAGE
DevPartner's TrueCoverage coverage analysis tool takes much of the pain out of checking for
test coverage by hand.
For each run of a program, TrueCoverage records which functions were called and which lines
in each function were executed. The program lets you accumulate coverage data from several
executions of your application by merging the session files. This facility allows for simpler
testing sessions, and is especially useful in interactive GUI applications for which testing
can be man-hour intensive. For each session or set of merged sessions, TrueCoverage reports
the number and percentage of functions and lines of code that were and were not executed.
The TrueCoverage user interface is similar to TrueTime's interface, so familiarity with one
tool should make it a straightforward proposition to use the other. Like TrueTime, TrueCoverage's
Session window has a left Filter pane that lets you filter the information displayed. There are
predefined filters for functions not covered, functions covered to less than a given percentage,
and functions belonging to a given package or class. Unlike TrueTime, TrueCoverage lets the user
define new filtering criteria to be inserted into the Filter pane.
The TrueCoverage Session Data pane shows the number of times each function was called, as well
as the coverage percentage and the number of lines still not-executed for each function. The
source tab can display the source code for any selected item (as in TrueTime.) The tool uses
colors to show which lines of code were or were not executed, and which lines correspond to
unreachable or otherwise non-executable code. TrueCoverage lets users exclude complete packages
and directories from analysis by listing them in the Program/Exclude dialog, on which the Java
platform runtime libraries, and the standard libraries from Netscape and Symantec, are
excluded by default.
As in TrueTime, the "project" concept is lacking in TrueCoverage. Although session merging
is supported, there's no way to define and save the configuration information for the
different test programs that are likely to comprise a complete test suite.
JCHECK
NuMega's JCheck is a thread viewer and program execution analyzer that provides a graphical
view of a program's thread activity, a time-ordered log of program events, and a log of actual
and potential problems detected during program execution. JCheck does not try to show program
events in real time. Instead, JCheck queues the events to let the application run at a more
natural pace. This lets the user continue monitoring detailed program activity, even long
after the analyzed program has ended. JCheck is especially useful for debugging problems
caused by thread activity, like contention, starvation, deadlocks, and livelocks. Running
a Java program from within JCheck is as straightforward as it is from within TrueTime and TrueCoverage.
JCheck's Thread view presents an animation of a program's thread activity. Threads, thread
groups, system threads, and synchronization objects are all represented with different symbols.
Thread icons are connected to the corresponding thread group icons through thick lines, and
arrows are used to show thread joins and pending waits over synchronization objects. A color box
jumps from one thread to another, showing which thread is active at any given point (when running
over a Java 1.2 VM, JCheck will even show the activity for the garbage collector's "Finalizer"
and "Reference Handler" threads).
During the animation, labeled lines are used to show events, such as the start and joining of
threads. The Thread view has settings to let the animation run to completion, or to have it
pause after each threading event. Buttons for zoom-in, zoom-out, and "view all" make it easy
to focus on the part of the animation that's interesting. Passing the mouse over any of the
icons produces a pop-up window with details about the state of the item under the cursor. I
was impressed with how informative this kind of thread animation can be. It would be
impossible to "see" as much by using only ad-hoc means, such as console writes or debugging logs.
Despite its usefulness, I wished that I could have replayed the animation from the Thread view
without having to run the application again. Being able to control the replay speed, and select
which subset of the sequence to play, would have made the animator truly outstanding.
During program execution, JCheck records every application event, from thread starts, stops,
suspends and resumes, to class loads and down to every single method call. The Session window
provides two views of this event data. The Results view displays information about errors,
leaks, exceptional conditions, and advisories about potential problems in the code. Livelock,
Deadlock, catching ThreadDeath, thread leaks, and joining of a thread to itself are some of
the problems that JCheck automatically detects. Clicking on any of the items in the Results view
shows an explanation of the problem, including details about the event and the contents of the
call stack at the point of execution. At the same time, the source code pane shows the line in
the source code from which the event emanated. Additional information about problems and
potential problems can be obtained by selecting "Explain" from the help menu.
The second view provided by the Session window is the Transcript view. This view shows a
time-ordered sequence of the program events, organized into a hierarchy that shows their
runtime context dependencies. Thread starts and resumes are shown at the top level, and the
nested sequence of method calls, errors, and exceptions that occurred while the thread was
awake are shown as children. The transcript can be filtered to show only method calls, class
loads, exceptions, errors, or application-generated bookmarks—or any combination of them.
Clicking on any of the items provides detailed information about the event. The source pane
shows the source code line where the event was generated.
NuMega provides a Java class library that provides the means to pause, resume, and bookmark
the event stream programmatically from within your application. Running the Jetty Web server
under JCheck was very informative, and I encountered no major problems.
CONCLUSION
Although I ran into some minor usability glitches in the DevPartner tools suite, they were
nothing you wouldn't expect from a 1.x version. For example, the tools' lack of smarts
about Java package-to-directory mappings generated too many prompts requesting information
about the location of source code modules. Also, the list of paths for source and class
files must be specified in a plain text box, something which quickly becomes unwieldy
with non-trivial projects.
A printed reference manual would be welcome in a future version. At a minimum, it should
contain information about the contents and lifetimes of the different files produced by
the tools, and it should make explicit the mysterious interactions between TrueTime and
TrueCoverage configuration and project files. Things like what exactly needs to be input
in the class exclusion dialog also need better documentation.
Finally, I'd like to see better support for the concept of a "project." At least there
should be a way to configure and save different VM/main-class combinations for repeated
use or abuse. The ability to add annotations to projects, sessions, and session items
would also be very welcome.
In virtually all other regards, the DevPartner suite performed beyond expectations, and the
tools were very simple to use yet quite powerful. Don't deploy your Java code without
analyzing it first with a tool suite like DevPartner.
Juancarlo Añez is the director of IT at Modelistica in Caracas, Venezuela. He can be
contacted at [email protected].
Quantity reprints of this article can be purchased by phone: 717.560.2001, ext.39 or by
email: [email protected].
|