Tracking Software Components -- ADTmag

Tracking Software Components

By Jerry Gao, Eugene Zhu, Simon Shim
October 5, 2001

Component engineering is gaining substantial interest within the software engineering community. A great deal of research has been devoted to analysis and design methods for component-based software. However, only a few articles addressed the testing and maintenance problems of component-based software. This article discusses component traceability and maintenance issues and solutions in supporting software components of component-based software. The authors propose a Java framework, as well as a systematic approach to support tracking and monitoring of software components in component-based programs. Application examples and the supporting system are described. In addition, the concept of traceable components, including requirements, design guidelines, and architecture style, is introduced. The results are useful for adding systematic component tracking features to the current Java and EJB (Enterprise JavaBean) technology to support software components, including third-party components, in software maintenance.

With the rapid increase in the size of software programs, it is important to reduce cost and complexity, while increasing reliability and ability to be modified.¹ With the advances of Internet technology more distributed systems are being built to meet diverse application needs. In addition, component engineering is gaining substantial interest in the software-engineering community. As more third-party software components are available in the commercial market, more software workshops are starting to develop component-based distributed applications.

Although there are many published articles addressing the issues in building component-based programs, very few address the problems and challenges in the testing and maintenance of software components and distributed component-based programs.^2,3,4,5 Today, real-world engineers have encountered a number of issues in the testing and maintenance of component-based software.^3,4,5 Some issues are related to component traceability and program tracking.⁴ They are given as follows:

Hard to understand component behavior in a system. In system testing and maintenance, system testers usually have difficulty understanding and monitoring component behaviors in a system due to the following reasons:

Engineers use ad hoc mechanisms to track the behavior of in-house components, causing system testers to have problems understanding the behavior of a system due to inconsistent trace messages, formats, and diverse tracking methods.
No built-in tracking mechanisms and functions exist in third-party components for monitoring external behaviors.
No configuration functions exist for component clients to control and configure the built-in tracking mechanisms.
No systematic methods and technologies exist to control and monitor the external behaviors of the components.

Difficulty in component error isolation, tracking, and debugging. In a system, components developed by different teams may use different tracking mechanisms and trace formats. Thus, inconsistent tracking mechanisms and trace messages cause difficulty in error detection and isolation of components.
High costs in performance testing and tuning of components. Performance testing for component-based programs is a major challenge in system testing due to the fact that current component vendors usually do not provide users with any performance information. Thus, systems testers and integration engineers must put a lot of effort into identifying performance problems and the components that cause these problems.
Lack of system resource validation for components. Since most components do not provide their system resource information, it is difficult for system testers to locate the system resource problems in system testing and maintenance.

Therefore, there are two major challenges in developing distributed component-based software. The first is to design software components with consistent mechanisms and interfaces to support the tracking and monitoring of component behaviors, functions, performance, and resources. The other challenge is to develop a systematic method and environment to monitor and analyze component behavior and system performance in a distributed environment.

This article addresses component traceability issues in supporting component-based programs. We discuss the component traceability concept, requirements, challenges, and evaluation criteria. Moreover, we examine different program tracking mechanisms. We first introduce the concept of traceable components and then propose a new tracking mechanism called an event-based tracking model. We propose a Java framework and a systematic mechanism for tracking and monitoring various software components in a component-based program. Our application examples and support system show the potential advantages of supporting both in-house and third-party components in component-based programs.

The structure of the article is organized as follows. "Understanding Component Traceability" discusses the perspectives of component traceability in terms of component behavior, interface, performance, events, and status. In addition, it discusses the concept of traceable components, including tracking requirements and evaluation criteria. Moreover, it examines different mechanisms to increase component traceability.

"A Systematic Tracking Solution" presents our tracking solution for use in supporting software components. In this section, a new Java-based framework for tracking is provided and the supporting environment is described. "Discussions and Summary" presents the involved design and implementation issues and our solutions. Application examples are given to demonstrate the basic idea and advantages of our solution. In addition, we discuss some open issues in monitoring and supporting software components in distributed environments by relating to existing work. The concluding remarks are given in the last section.

Understanding Component Traceability
According to IEEE Standard Directory of Electrical and Electronic Terms, "tracking" refers to the process of following a moving object or a variable input quantity, using a servomechanism.⁶ A trace routine refers to a program routine that provides a historical record of specified events in the execution of a program.

Software tracking includes project, product, and program tracking. Project tracking keeps track of important project schedules, activities, and events during the software development process. It is a fundamental activity in software management.^7,8 Product tracking refers to controlling and monitoring a software product in a systematic and manageable way.⁹ It is the essential task in configuration management. Program tracking refers to the activities and effort in tracking various factors of a program, including its input data, output results, and behaviors. It is an effective way to help engineers understand programming, debugging, software testing, and maintenance.

Component Traceability
According to Charles H. Schmauch, "traceability" refers to the ability to show at any time, where an item is, its status, and where it has been.¹ Software component "traceability" refers to the extent of its build-in capability of tracking the status of component attributes and component behavior. Component traceability includes the following two aspects:

Behavior traceability—Relates to the degree in which a component facilitates the tracking of its internal and external behaviors. There are two ways to track component behaviors. One is to track the internal behavior of components. This is useful for white-box testing and debugging. Its goal is to track the internal functions, internal object states, data conditions, events, and performance of components. The other goal is to track external behaviors of components. It has been used for black box testing, integration, and system maintenance. It's major purpose is to track the component's public visible data or object states, visible events, external accessible functions, and interactions with other components.
Trace controllability—Refers to the extent of the control capability in a component to facilitate the customization of its tracking functions. With trace controllability, engineers can control and set up various tracking functions, such as the turn-on and turn-off of any tracking functions and selections of trace formats and trace repositories.

We can classify component traces into five types (See Figure 1):

Figure 1. Different types of component traces.

Operational trace—Records the interactions of component operations, such as function invocations. It can be further classified into two groups—internal operation trace that tracks the internal function calls in a component and external operation trace that records the interactions between components. External operation trace records the activities of a component on its interface, including incoming and outgoing function calls.
Performance trace—Records the performance data and benchmarks for each function of a component in a given platform and environment. Performance trace is very useful for developers and testers to identify the performance bottlenecks and issues in performance tuning and testing. According to performance traces, engineers can generate a performance metric for each function in a component, including its average speed, maximum speed, and minimum speed.
State trace—Tracks the object states or data states in a component. In component black box testing, it is very useful for testers to track the public visible objects (or data) of components.
Event trace—Records the events and sequences occurring in a component. The event trace provides a systematic way for GUI components to track GUI events and sequences. This is very useful for recording and replaying GUI operational scenarios.
Error trace—Records the error messages generated by a component. The error trace supports all error messages, exceptions, and related processing information generated by a component.

Mechanisms Increasing Component Traceability
Since a component-based program is an integration of software components, the program's traceability depends upon the traceability of its components. Further, component observability depends on component traceability.⁴ In real word practice, we have used ad hoc mechanisms or pre-defined facilities to add tracking code into programs to increase program understandability. However, we have encountered difficulty in supporting diverse in-house components and commercial components (COTS). In this section we first discuss three different systematic tracking mechanisms and then examine their pros and cons.

Table 1 lists three basic approaches to add a consistent tracking capability into software components to increase component traceability and enhance the program.

Tracking Perspectives	Framework-Based Code Insertion	Automatic Code Insertion	Automatic Component Wrapping
Source Code	Needed	Needed	Not Needed
Code Separation	No	No	Yes
Overhead	High	Low	Low
Complexity	Low	Very High	High
Flexibility	High	Low	Low
Applicability	All types	OP trace, performance trace	OP trace, performance trace
Applicable Components	In-house components	In-house components	In-house components and COTS

Table 1. Comparisons of different tracking mechanisms.

Method 1: Framework-based tracking—In this approach, a well-defined tracking framework (such as a class library) is provided for component engineers to add program tracking code according to the provided programming reference manual. It usually is implemented based on a trace program library. Component engineers can use this library to add tracking code into components. This approach is simple and flexible to use. It can be used to support all trace types, especially error trace and GUI trace. However, there are several drawbacks. First, it requires a high programming overhead. Second, it relies on an engineer's willingness to add tracking code. Moreover, this approach assumes that component source code is available. Therefore, it is difficult to deal with COTS because usually they do not provide any source code to clients.

Method 2: Automatic code insertion—This approach is an extension of the previous one. Besides a tracking framework, there is an automatic tool that adds the tracking code to component source code. A parser-based tracking insertion tool is a typical example. To add operational tracking code it inserts the operation tracking code into each class function, at the beginning and the end of its functional body, to track the values of its input data and output parameters. Similarly, it can insert the operation tracking code before and/or after each function call. Performance tracking code may be added to a component in the same way to track the performance of each function. Although this approach reduces a lot of programming overhead, it has limitations. First, this approach assumes that component source code is available. This causes the limitation of using it on COTS due to the lack of source code. Next, it is not flexible enough to insert diverse tracking code anywhere in the components due to its automatic nature. Another problem is the complexity of the parser tool. Since a component-based program may consist of components written in different languages, more complicated and costly parser tools are required.

Method 3: Automatic component wrapping—This approach is another extension of the first one. Unlike the second method where tracking code is inserted by parsing component source code, this approach adds tracking code to monitor the external interface and behavior of components by wrapping them as black boxes. The basic idea is to wrap every reusable (or third-party) component with tracking code to form an observable component in the black box view. With the tracking code, engineers can monitor the interactions between a third-party component and its application components. This approach is very useful for constructing component-based software based on third-party software components, for example, EJB. Compared with the other two methods, it has several advantages. One of the advantages is its low programming overhead. In addition, it separates the added tracking code from component source code. Since no source code is required, this method can be used for in-house reusable components and COTS. However, it is not suitable to support error tracking and state tracking because they are highly dependent on the detailed semantic logic and application domain business rules.

It is clear that each approach has its own pros and cons. In real-world practice we need to use them together to support different types of tracking for a program and its components. To design and construct traceable components engineers need more guidelines for component architecture, tracking interface, and supporting facilities. They often encounter the following challenges:

How should one design and implement traceable and observable components in a distributed system?
How can one provide a well-defined tracking framework and effective tracking mechanisms with minimum programming effort and system overhead?
How should one support and monitor the behavior of COTs in component-based software?

A Systematic Tracking Solution
This section provides a systematic solution to solve the above problems in the development of component-based software. Our solution consists of three parts: a well-structured Java tracking package for component developers, an event-based tracking model, and a support environment for component-based software.

Java Event-Based Tracking Model
The event-based tracking model is a systematic mechanism that helps engineers monitor and check the behavior of components and their interactions in a component-based program. Its basic idea is influenced by the Java event model for GUI components. We consider all software components and their elements as tracking event sources.

In a software component, component engineers or an automatic tracking tool can add built-in tracking code to trigger five types of tracking events. They are performance tracking, operational tracking, error tracking, state tracking, and GUI tracking events. These events are packaged as tracking event messages and added into trace message queues according to their types.

To catch different tracking events we use a tracking listener to receive these events, dispatch them to a tracking processor to generate the proper trace, and store them into a specified trace repository. As shown in Figure 2, the event-model tracking mechanism relies on a tracking agent to support a collection of traceable components in a computer. Intuitively, a traceable component is a software component designed to facilitate the observation and monitor its behaviors, data and object states, function performance, and interactions with other components. In our solution, a traceable component contains two extra parts besides its normal functional parts:

Figure 2. Structure of an event-based tracking model.

A tracking interface is used to set-up the connection with the Tracking Agent. Figure 3(a) shows a procedure to dynamically generate a plug-in tracker for a component by issuing a binding request to the Tracking Agent through the tracking interface. ITraceableBean in Listing 1 shows a tracking interface in our Java tracking package.
A dynamic generated tracker is dynamically generated by the Tracking Agent. Each component receives one Tracker after it connects to the Agent. Developers can use the generic interfaces to provide various tracking functions for different trace types. IBeanTracker in Listing 1 shows the details of the generic interfaces for tracking functions.

Figure 3. The communication sequence between a tracker and a tracking agent.

A tracking agent consists of the following three functional parts:

Tracking Listener is a multi-thread program that listens and receives all types of tracking events through trace message queues and dispatches them to a tracking processor. Figure 3(b) shows the interaction sequences among a traceable component, tracking listener, and tracking processor.
Tracking Processor generates program traces according to a given trace event based on its trace type, trace message, and data, and stores them in the proper trace repository.
Tracking Configuration provides a GUI to allow a user to discover and configure various tracking features for each traceable component.

Java Tracking Package
Current component-based technologies (such as JavaBeans, EJB, and CORBA) do not provide developers with systematic mechanisms and facilities to track and monitor component behaviors. Thus, developers only can use ad-hoc methods to construct traceable components. Our Java Tracking Package provides developers with several generic interfaces for constructing traceable Java components. It includes:

An interface for a component tracker (see IBeanTracker in Listing 1). This interface provides the generic tracking function interface between a component tracker and the tracking agent. Various tracking events and requests can be passed to the tracking agent.
Agent interface (see ITRAgent in Listing 1). It is a part of tracking agent. Using this interface, developers can issue a request to the Tracking Agent for a component to create a plug-in tracker and set-up their connection.
Traceable component interface (see ITraceableBean in Listing 1). This interface allows developers to bind a plug-in tracker for a component. Listing 3 shows a sample implementation for a bindBeanTracker. The function getTracePropertie( ) can be used to discover the trace properties of a component.
Tracker adaptor interface (see BeanTrackerAdaptor and GUIListenerAdaptor in Listing 2). This interface is provided to a dummy tracker in case a component tracking environment is not set up.

A Tracking Environment for Component Software
As shown in Figure 4, we developed a support environment for component-based software. It consists of a number of tracking agents and a tracking server. Each machine on the network has a multiple threading tracking agent based on EJB technology. It interacts with the plug-in trackers of components on the same machine using trace message queues (in Java Message Queue).

In a tracking agent, a thread controls, records, and monitors diverse component behaviors in an asynchronous mode. On the other hand, the tracking agent communicates a tracking server to pass trace data in a given trace format. The tracking server plays a role of a central server to allow engineers to control tracking agents to collect diverse trace data and analyze them.

Java's JDK.1.2.2 was used to create our tracking agent and tracking server. The distributed supporting environment for JavaBean components is set up based on an open source implementation of EJB specification 1.0 Java Open Application Server (JonAS) from BullSoft (www.bullsoft.com). In addition, the Java Message Queue (JMQ) is used to perform the asynchronized communications between JavaBeans and a tracking agent. To create the trace data repository we used InstantDB, a 100% Java database server from Relational Database Management System (RDBMS).

Figure 4. Distributed tracking environment.

Trace Format
Using a standardized trace format is essential for generating consistent trace messages for each component in a distributed program. A well-defined trace format facilitates the generation of understandable trace messages. This increases component understanding and helps error isolation and bug fixing. Our distributed tracking environment supports two types of information:

Trace commands—Trace commands support the tracking server in controlling and communicating with tracking agents in the distributed environment. Each command includes command message ID (00), command code, time stamp, and command parameters.
Trace data—Trace data includes message ID (01), trace type, time stamp, component id, trace data, and trace identifier. Trace data indicates where and when the trace message is generated. A trace identifier includes component identifier, thread identifier, and object identifier. Each trace type has its own specific trace data format. Tracking agents generate diverse trace data for each traceable component in a local trace repository.

An Application Example
We have applied our solution to a simple Java Bean-based virtual bank system to demonstrate its feasibility and correctness. Figure 5a shows a login screen for a virtual bank Java Bean. Figure 5b displays a simple GUI that occurs after login from a user (say Eugene Zhu). The user can perform three different operations to request the respective bank transactions. They are check account balance using the Balance button; make a withdrawal using the Withdraw button; and make a deposit using the Deposit button. Figure 5c and 5d show two of these operations.

Figure 5a. The login screen.

Figure 5b. A virtual bank GUI.

Figure 5c. The withdraw action.

Figure 5d. The deposit action.

Figure 6 displays a GUI of a Tracking Agent. This interface allows a user (say a system tester, or an engineer) to turn-on or turn-off tracking functions, select tracking types, and set up the event trace monitor. In the upper right-hand corner, the IDs of tracked bean objects are displayed. In the monitor area, various types of trace messages are displayed according to the receiving order of the Tracker. In Figure 6, each trace message consists of trace type, Bean ID, timestamp, function name, and data. Figure 7 shows GUI tracking results for all GUI events during system-user operations when GUI tracking is selected and set up.

Figure 6. Trace results in a tracking agent.

Figure 7. GUI trace results in a tracking agent.

Discussion and Summary
This section discusses several important design issues involved in our solution. The issues are tracking code insertion, tracking overhead reduction, technology flexibility, third-party component support, and distributed monitoring and support for components.

Tracking Code Insertion
Developers can use our solution to add tracking code to a Java component (such as EJB components, Java components or JavaBeans) in two ways: automatic insertion, and user insertion. Component developers can add tracking code manually using the tracking package. Listing 4 demonstrates how to add tracking code in a Login(..) function for a Java class that is called VbankBean. Three types of trace code are inserted. They are performance trace, error trace, operation trace, and interface trace. Figure 6 and Figure 7 show the trace results in a tracking agent.

Listing 5 demonstrates how to add GUI tracking code to monitor various GUI events in a Java-based login Window frame (loginWin class). Figure 6 shows the GUI trace results in a Tracking Agent. It is clear that the trace codes (except GUI trace and error trace code) can be used to wrap third-party components to monitor their external behaviors, such as operations, interfaces, and functional performance. After we examine the source code of the JonAS, we found that the presented mechanism can be added into the deployment tool of its container in JonAS to implement automatic tracking code insertion using the wrapping approach.

Overhead Reduction: There are three types of overheads involved in the component tracking: programming overhead, system performance overhead, and code overhead. System performance overhead reduction is very important for component tracking. In our experience, we found that the major tracking overheads are the overhead in invoking the tracking functions for components and the overhead in the generation of tracking events. Table 2 shows our experimental results for two JavaBeans on the same machine in the given application example. One has tracking code and the other does not.

Function	Bean with tracking code	Bean without tracking code	% of performance overhead
Login()	2684 ms	1942 ms	38%
Check Balance()	300 ms	210 ms	43%
Withdraw()	501 ms	381 ms	31%
Deposit()	471 ms	340 ms	39%

Table 2. Tracking overhead.

In our design, we use the following three methods to minimize tracking overheads:

An asynchronized method to exchange the trace data between the tracking functions of a Tracker for components and the Tracking Agent. Therefore, the tracking functions do not need to wait for the completion of the tracking data processing.
A configuration data exchange that makes the tracking functions generate the tracking data only when required.
An adapter class to replace the real track engine to reduce the overhead to a minimum.

In addition, more complicated methods may be used to eliminate the tracking performance overhead. Here are some typical examples:

Develop two versions, one with tracking code and the other without. Use the first one to perform the test and release the other one to production. The trade-off is that you can only get the tracking information when performing testing. In most cases, the effect of the overhead on the program logic is minor and only the performance will be affected. Define a special track engine for performance tracking only. The other methods are:
- Add tracking code with a prefix (such as //// in Microsoft Visual C++) and enhance the current Java development environment to support compilation of tracking code to generate two different versions. One version contains the tracking code and the other does not.
- Developers may customize the bindBeanTracker( ) function to bind to a default dummy tracker using BeanTrackerAdapter. Under this setting, no real tracking function codes are actually executed. The only extra overhead is the involved function invocations of these empty tracking functions in the Adapter classes. Figure 5 shows an example of how to bind to the default adapter.
- System supporters and developers can use the tracking configuration user interface offered in the Tracking Agent to turn-on or turn-off the tracking features for a component. When a tracking feature is turned off, the only overhead is involved in the condition checking.

Advantages: The presented solution has several important advantages:

It is flexible to allow component users to control and configure the monitoring and tracking capability of various behaviors in components. (See Figure 5 and Figure 6 for the corresponding GUI.)
The proposed tracking package provides a well-structured tracking interface that separates the implementation of the tracking solution, trace formats, and traceable components. This implies that a software workshop can define and customize the Tracking Agent and trace formats according to its technology and environment requirements. For example, they can implement trace message queues using sockets (or pipes) instead of Java Message Queue.
It can be used to monitor the external behaviors of third-party components, including functional performance, external operations, and external interfaces.
The two-level tracking architecture based on Tracking Agent and Tracking Server provides the scalability. For example, Tracking Agent on a machine is set up as a multithreading program to support many traceable components. Each thread is a lightweight process communicating with one component's tracker. In addition, the Tracking Server provides distribution to monitor components in a distributed environment.

Limitations: The proposed approach has many advantages and it can be very useful in software function testing and system maintenance. However, it has three major limitations. First, it is not very cost-effective to track the system performance due to its performance overhead. Moreover, the tracking result of the performance data has to be analyzed and adjusted to reflect the real system performance.

Second, it does not keep the event sequence in the order they occur due to the usage of the asynchronous communication between a Tracker and a Tracking Agent. This implies that a time stamp is needed for each event trace message to support the future analysis.

Finally, in interface tracking, the current proposed solution only handles the parameters with primitive data types, such as integer or string. To deal with a complicated data object or user defined class object, the current solution only generates a string value of this object by calling the method 'toString(),' and sometimes does not not capture the core information in that object. To extend the solution to deal with any type of data object more work needs to be done to support a complete interface tracking.

Other Issues: There are several other issues relating to the monitoring of component behaviors in a distributed environment: clock synchronization, event ordering, behavior and performance analysis. In the past, there is a number of published research papers addressing the issues in monitoring and debugging of distributed programs and processes.^10,11,12,13 Their focus is on monitoring distributed processes and events at the process level.

Due to the lack of a global clock in a distributed environment, we need to establish temporal relationships between occurrences of system activity. Three primary methods have been used to deal with this issue. The first involves the use of local clocks and timestamps to create a partial ordering among events.^11,12 The second creates a global snapshot by gathering and grouping of state information in the entire system.^13,14 The third uses a hardware module to solve this problem.¹⁰ Although these methods can be applicable to component-based software in a distributed environment, the last two approaches may not be practical and flexible to deal with the complicated object interactions in component software. In our solution, we use the local time and timestamp to find out the partial ordering in the event traces. Unlike other approaches, we focus on Java-based software components and provide engineers a well-defined event-tracking framework to construct traceable components. A flexible supporting environment (including Tracking Agent, Tracking Server and Message Queues) are developed based on the tracking framework.

How to understand and analyze component behaviors, interactions, relationships, and performance in a distributed environment is another open issue. Although, we have seen existing research work and supporting environments addressing the maintenance issues of traditional programs and OO programs.^15,16,17,18 These systems help engineers understand class relationships, object dependence, and program flow. Unlike traditional distributed systems, the performance and behavior analysis of software components in a distributed environment is difficult due to the involvement of third-party components. Engineers are looking for systematic performance models and solutions to support their understanding of components, including their interactions, relationships, dependence, and performance.

Conclusion
In this article, we introduced the concept of traceable components and demonstrated how to construct traceable components so that engineers can easily monitor and check various behaviors, status, performance, and the interactions of components. We presented a Java framework and tracking model to allow engineers to add tracking capability to components in component-based software. Moreover, we discussed the design and implementation of a distributed supporting environment for component-based software. Component developers can use the presented framework and tracking mechanism to develop Java-based components and EJBs-based components. Moreover, the solution is useful to support third-party software components and check various component behavior. The solution has several advantages:

It is simple and easy to use for building traceable components with low programming effort.
It is flexible and configurable enough to allow system supporters to monitor various component behaviors, including GUI behaviors, performance, errors, and interactions.
It has a consistent trace format and lightweight tracking code.
It is scalable and useful for in-house and third party components.
It is changeable and can to fit with different requirements and technologies in organizations.

We are currently investigating new solutions to these open issues in the maintenance of software components in a distributed environment; such as trace analysis models, methods and tools to support fault detection, fault isolation, and performance analysis.

References

Schmauch, Charles H. ISO 9000 For Software Development: Revised Edition, American Society for Quality, 1995.
Weyuker, Elaine J. "Testing Component-Based Software: A Cautionary Tale," IEEE Software, 15 (5): 54–59, Sept./Oct., 1998.
Voas, J. "Maintaining Component-Based Systems," IEEE Software, 15 (4): 22–27, July/Aug. 1999.
Gao, Jerry. "Testing Component-Based Software," Review (STARWEST'99), Nov. 1–5, 1999, San Jose, CA.
Voas, J. M. and Keith W. Miller. "Software Testability: The New Verification," IEEE Software 12 (3):17–28, May 1995.
IEEE Standard for Software Verification and Validation, IEEE, 1986.
McConnell, Steve. Rapid Development: Taming Wide Software Schedules, Microsoft Press, 1996.
Kit, Edward and Suzanna Finzi, Eds. Software Testing in the Real World: Improving the Process, Addison–Wesley, Reading, MA, 1995.
Rada, Roy. Reengineering Software: How To Reuse Programming To Build New, State-of-The-Art Software, 2nd Edition, Intellect Ltd., New York, 1999.
Hofmann, R., et al. "Distributed Performance Monitoring: Methods, Tools, and Applications," IEEE Transactions on Parallel and Distributed Systems, 5(6): 585–598, 1994.
Lamport, L. "Time, Clocks, and the Ordering of Events in a Distributed System," Communications of the ACM, 21(7): 558–565, 1978.
Dieter Haban and Wolfgang Weigel. "Global Events and Global Breakpoints in Distributed Systems," 21st Hawaii International Conference of System Science, 166–175, 1988.
Chandy, K. M. and L. Lamport. "Distributed Snapshots: Determining Global States of Distributed Systems," ACM Trans. on Computer Systems, 3(1):63–75, 1985.
Miller, B. P. and J. D. Choi. "Breakpoints and Halting in Distributed Systems," Proceedings of the 8th DCS, 316–323, 1988.
Rajlich, Vaclav, et al. "VIFOR: A Tool for Software Maintenance," Software Practice and Experience, 20(1): 67–77, Jan. 1990.
Moises Lejter, et al. "Support for Maintaining Object-Oriented Programs," IEEE Trans. On Software Engineering, 18(12): Dec. 1992.
Kung, David, Jerry Gao, Pei Hsia, Yasufumi Toyoshima, Chris Chen, Young-Si Kim, and Young-Kee Song. "Developing an Object-Oriented Software Testing and Maintenance Environment," Communications of the ACM, 38(10):75–87, Oct. 1995.
Wide, N. and R. Huitt. "Maintenance Support for Object-Oriented Programs," IEEE Trans. on Software Engineering, 18(12):1038–1044, Dec. 1992.