In-Depth

Testing key to component quality

• By Johanna Ambrosio
• September 30, 2002

Testing components is similar to testing ''regular code.'' But there are many differences as well, including how much more important testing becomes when developing your own components or using someone else's components in your application.

There are also more stages of the life cycle during which major testing is needed, and more uncertainties or potential weak links to plan around and test for. Further, developers need to become more involved with testing than ever before, and are required to stay involved with testing for longer periods, experts say. Add Web services components to the mix, and the ante is upped even higher. (See the related story ''Testing Web services: Even more complex.'' )

The biggest difference, most observers agree, is that components require a higher degree of quality. ''A traditional application is 'good enough' 85% of the time,'' said Theresa Lanowitz, research director at Gartner Inc. in Stamford, Conn. ''But if you're using multiple components, the system is only as good as its weakest link. They have to be tested to a flawless state,'' she explained. ''The notion of 'good enough' software ceases to exist.''

Enzo Micali, chief technology officer at 1-800-flowers.com in Westbury, N.Y., agrees. ''The more shared the code is, the more dependent the rest of the environment is on it. So the performance of that particular component is critical.'' His shop is creating Java components that will serve as the middle tier of the company's computing architecture and hold the business logic used throughout all corporate applications. The company is doing unit, functional and stress/load testing of its components using multiple tools.

Testing is key because components can be reused in ways not always foreseen by developers when they initially sit down to code. In addition, the more pervasive the component -- the more various analysts or applications touch it -- the higher the quality required.

Given this interaction with other components and pieces of a distributed system, testing must become more fundamental to the component development process. It is also something to plan actively during the component's concept phase. Some believe this requires an agile development approach, whether it is Extreme Programming or something else, that calls for a ''test first, develop second'' life cycle.

''Buggy'' software costs the U.S. economy almost $60 billion each year, according to a recent study by the National Institute of Standards and Technology. Improvements in testing could reduce this cost by a third, the study said.

Getting to the highest quality components will require a shift in thinking on the part of developers, IT executives and their peers, most observers agree. ''Each organization will have to define a process that works for them -- I don't think we can pinpoint a specific methodology,'' said Gartner's Lanowitz. ''But it will require more cross-functional communication and meaningful reports.''

Sam Guggenheimer, senior director of technology at Rational Software Corp. in Cupertino, Calif., said most of the heavy-duty work in component testing has been happening in the embedded systems world, at places like the Federal Aviation Administration (FAA). ''There's a fairly evolved set of practices there,'' said Guggenheimer, ''the most rigorous of which is in a document the FAA uses.''

In general, a traditional application is ''fairly monolithic,'' Guggenheimer said, and ''you have a fairly small degree of variation in the kinds of things you expect to see and have happen.'' But in the component world, there are ''many possible points of failure,'' including the application server, back end and user interface, among others. In a traditional application, there are ''fewer moving parts'' than in the component world, he added.

Thus, the kind of testing needed in the component world is much more thorough than the testing done in the traditional mainframe or client/server worlds, where one group controlled all the code and there was a high degree of certainty about what was in the code, how it worked, where its weaknesses were and whether the company could ''live'' with those weaknesses until the next software version came along.

What testing is needed?
Rudolf Hauber, principal consultant at Koelsch & Altmann, an IT consultancy in Munich, Germany, has worked on numerous projects in which components were used for clients in the military, insurance, financial services and other fields. He said the kinds of testing components require differ from project to project. For instance, a real-time system might need testing for timing constraints to make sure a component has the sub-second performance it needs, while another project might require more functional testing.

That said, Hauber did offer some advice for all kinds of component projects. ''It's very important to have a testing strategy -- to know up-front how and what you want to test,'' he noted. Next, test at as low a level as possible -- class, component, subsystem -- whatever you can. ''There's less effort needed to test on a lower level than on a higher one,'' he explained. Finally, use a tool to automate testing as much as possible; this way you do not have to hand-code the tests themselves, he said.

Hauber's firm automated its process with a tool they developed in-house. Called TestExpert, the tool is based on Rational Rose and was ''developed over a year, during the course of many projects,'' he said. Essentially, it builds a model of the system and then generates code to test each class, each operation in the interface and then the entire component.

In addition, components require the usual ''black-box,'' ''white-box'' or functional testing done in traditional applications. In black-box testing, components are sent a message to see how they respond and what functions they perform. It is ''black-box'' in the sense that users only know the results of the test, not why tests are done. Each function and feature is tested to make sure it works and does what it is expected to do. This is behavioral testing.

In white-box testing, one uses internal knowledge of the software to help define the tests that will be used. This is more structural or architectural testing, also known as glass-box or clear-box testing.

With components, some observers maintain, there is more black-box testing required to make sure all the components work along multiple paths and with each other. There is also a need to combine different types of testing in new ways.

Rational's Guggenheimer, for example, suggested combining runtime analysis with black-box testing. This will show, he said, ''which lines of code have been covered, what functions have been tested, what the profile of the response is and whether there are specific bottlenecks that might be resolved.'' Guggenheimer also talked about ''gray-box'' testing, a process that combines white-box and black-box testing in new ways.

However, most of today's tools ''don't think about that combined usage,'' said Guggenheimer. ''Most are focused on either the black-box or white-box side.'' Another key set of features for testing includes the ability to automatically generate test harnesses for code that has never been exercised, and to rank the components according to risk, so ''I can know where I need to focus,'' he added.

Class I.Q., a testing vendor in Wilmington, Del., defines a four-tier process for components used to help integrate corporate applications: process testing, to test workflow processes independently of the apps; rule or middleware testing, to test the applications being integrated; component protocol or message testing, to make sure the component works through the interface (commonly known as ''black-box'' testing); and integrated component simulation, to simulate a component's response to a specific message.

The general idea is to find any problems or bugs as early as possible, before the component is integrated with others and it becomes more difficult - if not outright impossible -- to track down the source of a bug. Then again, some bugs only occur in combination with other components -- meaning each potential path the component might take needs to be tested. Tests also need to be repeatable across multiple components and paths, and the results must be tracked in reports that make sense.

Today's tools
There are already a bevy of testing tools on the market (see the related story ''Representative testing tools'' ), and more on the way. Compuware Corp.'s Remote Agent, for example, looks at different levels of an application to find out where problems are. ''Where the bug shows up may have nothing to do with where the error actually occurred,'' said Peter Varhol, product manager for the Farmington Hills, Mich., vendor's DevPartner Studio family. ''We test components on two levels: as standalone pieces and then as part of an integrated system.''

The software looks at what happens in each and every component of an application, and then correlates events across components, Varhol claims. ''We follow a transaction from one component to another, from beginning to end. If an error shows up, we can go into our distributed analysis feature and see exactly what event generated that error and in what component,'' he said.

At Mercury Interactive Corp. in Sunnyvale, Calif., the idea is to make ''QA testing more accessible to earlier phases in the development life cycle,'' said Simon Berman, director of product marketing. ''We wanted to customize testing tools for server-side EJBs and make it available within the native IDE.'' Borland's JBuilder and BEA Systems' WebGain platforms now include those add-ins. And whatever tests are created by developers in those platforms can also be used by the QA staff via Mercury's LoadRunner, Berman explained.

The goal is to create a connection for component testing between the development and QA sides of the house, so that each group can use their preferred tools, but also share tests and test results. ''We've found a way of providing better cooperation, so testing can be done earlier and the assets can be shared,'' Berman said. The IDE add-ins are available free of charge, so developers can ''create all the unit and functional tests they want,'' he said.

The developer's role
Still, ''testing frameworks are no substitute for a good testing strategy,'' said Adam Wallace, vice president of research and development at Flashline Inc., a Cleveland-based vendor that sells component management software. ''You have to make sure you're testing for the things you want to,'' he explained.

Developers and testers need to work together to define what needs to be tested and when during the process that needs to happen. Developers also need to do more testing than perhaps they once did. ''The responsibilities of both the developer and tester need to change,'' said Rational's Guggenheimer. ''The developer's responsibility is to guarantee a level of reliability for this component. The tester then needs to creatively push the test, not just to look for clearly identified defects, but [for] general fitness for use.''

Neither side can simply ''test through straight-line, happy-day scenarios,'' said Guggenheimer; they have to ''pick representative cases that are sufficiently complex to exercise the code and illustrate the possible risks.''

Similarly, ''we find developers involved right until the stages of integration testing, where you bring together all the components and make sure they work reliably,'' said Compuware Corp.'s Varhol. ''The application developer's job isn't done until all the components come together.''

Related
Testing key to component quality -ADT, Oct. 2002
Testing Web services: Even more complex -ADT, Oct. 2002
Representative testing tools -ADT, Oct. 2002
Create your own test for Java/EJB code -ADT, Oct. 2002

Individuals contacted for this story were presenters at the Sept. 2002 "Software Test Automation Conference & Expo" To learn more, go to the conference page at http://www.sqe.com/testautomation/.