In-Depth

When objects collide, you must rethink your test strategies

• By Linda Hayes
• July 19, 2001

There is an upheaval taking place in software development. Instead of hand-crafted, proprietary systems residing in a protected environment, applications today are virtual galaxies of software objects strewn across platforms. This shift is creating aftershocks in software quality, especially in the area of integration testing.

At the beginning of the software chain are objects, which may be developed or purchased. Next are middleware applications, which provide components to vendors of end-user products and which are themselves composed of distributed objects and subsystems. Further downstream are applications sold to an end user, which are likely compiled using multiple development languages and third-party components from different vendors and executing across multiple tiers. The final level of software development -- at least for now -- is the enterprise, where applications are integrated into complete operating environments.

Each of these levels poses unique challenges and is spawning new approaches and tools. A look at the most vivid trend today shows that integration testing is expanding not only in scope and complexity, but also across departments and company lines. Integrated testing joins all the participants up and down the software chain to leverage -- rather than struggle against -- diversity. Integration testing may save the day because it embodies the age-old concept that earlier is better when it comes to testing, and it is the responsibility of each contributor to not only produce a quality product, but also provide support to everyone down the line.

Key elements of integration test software methods include more sophisticated use of test hooks and monitors, better simulation of interfaces, and creative partnerships between development and test team members and between vendors and customers.

On ramp

Consider the object. At a technical level, it is a code unit that responds to a defined set of calls; the form of the calls may be determined by the standard to which it adheres, such as Corba or DCOM. In functional terms, it may provide any of a wide range of services: communications, device support, data services, even spell checking. For integration test purposes, the object's response to these calls from other objects or systems must be verified.

It is becoming easier to develop objects, but unfortunately, managing the environment becomes more complex. The application boundaries are blurred, both internally and externally. Creating and managing applications comprised of objects has become the focus of a new class of development tools. ObjecTime Limited of Kanata, Ontario, Canada, is one of the companies that is delivering capabilities for facilitating the development of large scale, complex systems. ObjecTime is a spin-off from Brampton, Ontario, Canada-based Nortel (Northern Telecom), a telecommunications firm. Their industry is a prime example of complexity, encompassing business applications as well as embedded data communications, switching and other services acquired from an assortment of hardware and software vendors.

"With ObjecTime Developer, we provide a visual modeling language that allows components to be developed and assembled into applications," described Steve Saunders, director of Product Marketing Management at ObjecTime. "This model is used to generate C or C++ code." Even externally developed objects can be included, using wrappers that convert them into "active objects" or "actors" that ObjecTime can incorporate. This model makes it easier to see the interrelationships among objects and the overall structure of the application.

"Testing support is built right in," said Saunders. "The generated application code can be run at any time during development, [and] instrumented to observe messaging among objects as well as monitor their internal states." ObjecTime also offers unique security aspects for components developed with it. "Black box packaging allows you to lock down a component and control access to it," explained Saunders. "This allows downstream vendors to inherit your objects and refine them without corrupting the core or revealing the implementation." This capability protects the intellectual property of object vendors who want to make their functionality available without revealing how they accomplished it.

The incorporation of test hooks, monitors and security are positive indicators that quality is being designed into objects early for downstream developers.

Middle of the road

The object developer will probably never know how object software will be used downstream, or what environment it will be executed in. In fact, the object vendor's customer is often itself a provider of components for yet another customer's product. An emerging class of higher level components is positioned as middleware, software that is incorporated with other technology into end-user applications.

Tandem Telecom is a case in point. This Plano, Texas-based division of Tandem Computers in Cupertino, Calif. provides a software platform to telecommunications providers, who in turn create custom services for their end users such as home location registry and local number portability. Tandem Telecom's software platform is an engine with which its customers can write applications. This engine is itself configurable, and the customers are able to add options based on their needs.

"Our products provide capabilities in the form of API calls," explained Edna Clemens, Manager of Development Services at Tandem Telecom. As part of the integration test process, an application has been written to exercise each call, verify the parameters and return flags. But, as Clemens pointed out, "although we know what each call should do, we have no way of knowing what the customer will do with it." No two customers implement the same set of functionality the same way, and of course, the customer environment is itself a moving target.

As part of an ongoing process improvement program, integration testing is becoming more formalized at Tandem Telecom. The development group has devoted resources to establishing an integration environment and process and documenting a formal test plan. This phase focuses on testing all the different flavors available through varying configuration options and is the final stage before system test.

Delivering a solid build has alleviated some of the burden previously on system test, which has been reinvested in further refining the system test function. Now, instead of being totally focused on test execution, system test has been divided into test development and test engineering. "Test development addresses new features and functions by extending our coverage," said Janet Brown, the test manager at Tandem Telecom. "Test engineering is responsible for actually executing tests and maintaining the test environment." This division of responsibility helps to assure that the schedule pressure of test execution does not detract from keeping pace with product enhancements.

Testing middleware requires the exercise of both internal interfaces as well as external ones. The middleware itself comprises several subsystems, and the messaging between them must be verified. The API sets that are delivered to customers are also comprised of building blocks. "We provide an engine that executes SIBs (service instruction blocks). Each SIB provides a certain set of functionality, such as support for a particular switch or protocol," explained Clemens. Some of these SIBs are internally developed, some are outsourced and some are licensed.

External interfaces, such as the call traffic, must be simulated or tested in conjunction with the customer's own facilities. The variety and complexity of the customer's environment -- different platforms, switches, protocols -- and the fact that each implementation of the API engine is unique, makes comprehensive test coverage by Tandem Telecom a virtual impossibility. Yet, in the telecommunications market, quality is essential. Clearly, cooperation is the best strategy.

"We have recently established a Partner Lab with one of our key customers," said Clemens. "This enables us to test our system with the customer's application and environment." The shared lab has become increasingly popular as a means for accomplishing the transfer and acceptance of the software between the two companies, and is indicative of the cooperative trend that is taking shape. Essentially this approach recognizes that development spans companies, and so should integration testing.

Software freeway

At the next stop downstream are end-user application vendors. Consider Oacis Healthcare Systems, a Greenbrae, Calif. company that markets application software to the healthcare industry. As with so many applications houses today, complexity is a given. Their products -- eight front-end applications that communicate with three back-end processes -- integrate capabilities from a literal dozen vendors, including two development languages (VB and Visual C++), two development libraries, a database, middleware and server services, all supporting four different operating systems and executing in a multitier configuration ... not to mention the hardware variations.

Oacis has a separate, formal integration test group. Herb Davis, the manager of the Integration Team, has implemented both formal and informal processes supported by a training program for team members. "In our environment, integration testing is the gatekeeper between development and the product line," said Davis. "The purpose of the Integration Team is to identify and isolate interactions among the components in the build, confirm compatibility and test for cross-application impact."

Developers must submit a formal Oacis Integration/Change form, detailing changes or additions being introduced into the product. These changes are then introduced in an orderly manner; the new build is first verified, then staged for acceptance and regression testing. "This approach has greatly increased the quality of our products," noted Herb Isenberg, quality assurance architect at Oacis. "It allows us to test earlier, and it has also forged a cooperative relationship between testing and development."

This cooperation reflects an increased understanding that everyone is on the same side, trying to produce a quality system, and that it is easier if everyone understands their role. Although formality is necessary to define the process, "it's also important to stay flexible," said Isenberg. "With this many components, we may have to react quickly on an exception basis." This mindset is critical to positioning integration test as an accelerator, not a roadblock.

But creating and managing an integration test environment is far easier decided than done. "Aside from maintaining communication and compatibility among all the development groups and coordinating schedules, the biggest issue involves environments," Isenberg pointed out. "We have to maintain three for each product -- the old version, the current one and the new one -- and keep multiple sets as we migrate to new environments." This requires a major commitment of hardware and software and a corresponding dedication of resources to manage and maintain it.

Not only is there an impressive array of components and subsystems, but they are further multiplied by the list of Oacis supported customer configurations: workstations, servers, operating systems, third-party components such as databases and report writers ... the list goes on. Each component is versioned and the minimum and maximum configurations documented.

As already noted, the expansion of the integration test function does not happen in a vacuum. Process definition is critical to assuring that everyone understands their link in the chain. Tina Brooks, director of process improvement at Oacis, is defining the complete development and test cycle. Integration Test has planned to expand its scope up to Alpha Acceptance, which will include high level system test now performed by QA. This will in turn free up QA to invest in expanding their coverage.

In an environment as complex as Oacis', uncovering -- let alone diagnosing -- issues that are introduced during integration is itself an adventure. Typical debugging and diagnostic aids are language-centric, provided with the compiler or development kit. How can you effectively test components written in different languages and objects supplied by other companies?

It is for this type of testing that products like BoundsChecker from NuMega Technologies are made. "Under the hood, all objects are just forms of dynamically linked libraries (DLLs), except with different extensions," said Doug Carrier, director of product marketing at NuMega of Nashua, N.H. "Our products are able to intercept and monitor events between objects at the system level. Using a runtime injection technique, we can see all API calls that the code makes outside itself." Runtime injection intercepts and logs calls made between objects.

These calls form the conversation among components. Although there are standards for component calls, such as Corba and COM/DCOM, each component has a unique set of parameters and flags, all with different constraints. Simply testing all valid and invalid ranges and types of calls to a single component may be fairly extensive. For this, NuMega has developed a utility called API Gen, which is part of the BoundsChecker offering.

"API Gen can generate an API test framework for component calls, creating a custom API checking capability," said Carrier. "Complete log files capture all calls and provide diagnostic information for debugging." This type of "on-board" test facility reinforces the concept that testing is an activity that spans all levels.

In the future, NuMega promises enhanced support for tracing issues all the way into the source code of the component. Current support for C and C++ is being extended to Visual Basic as part of the Fail Safe product line. "Most development language debuggers miss system level calls," Carrier said. "Our products execute at the protected system level and can monitor events at any desired level of granularity."

Crossroads

It is important to re-evaluate integration testing: who does it, as well as when and how it is done. Companies that direct their focus to this phase, and thus deliver a higher quality system to system test, enjoy increased quality and faster time to market. Issues are either prevented or discovered sooner, requiring less rework and leaving more time to invest in wider test coverage.

At the most, new development and test technologies promise a new age of "cooperative quality." Development and test groups are integrating their processes, leveraging their respective skills in a combined effort instead of the old, throw-it-over-the-wall approach. Vendors and customers are joining forces. Vendors are building test capabilities into their products, providing support for their customers to test their applications, and customers are teaming up with vendors to test their unique configurations.

Perhaps the most important advice is to pay attention. The rules of development are changing, and quickly. Test processes that worked before are simply no longer valid, and organizations that do not admit and deal with these changes will find themselves mired in complexity, strangled by it, instead of empowered.

--Linda Hayes