In-Depth

Testing Smooths Path to Integration

• By Lana Gates
• September 27, 2001

"Application integration is certainly something we're seeing a lot more of," noted Theresa Lanowitz, research director in the Applications Development Group at GartnerGroup, Stamford, Conn. According to Lanowitz, there is an increase in the number of organizations integrating multiple types of packages, and the first question they are asking is whether or not they need to test. "Yes, they do need to test unless they want to run the risk of the integration they're doing not working," she said.

"The demand for quality has moved up dramatically," added Dick Heiman, research director for application development and deployment at research firm IDC, Framingham, Mass. "Saying, 'Let's build something, throw it out there and fix it when it doesn't work' just won't fly."

So how does an organization make testing a bigger part of its development process? The best way, according to analysts, vendors and end users, is to start early and test throughout the entire life cycle.

Keep in mind that there is a difference between testing and quality. Testing, according to Heiman, is a verification that things are working effectively. Quality, on the other hand, is the entire process. The two work hand in hand. "Quality assurance [QA] includes doing the right testing at each point of the life cycle," explained Phil Hollows, vice president of technology at RadView Software Inc., Burlington, Mass.

RadView's strategy to application integration testing is to take the technologies that are limited to QA functions and to spread them around the life cycle. The software vendor believes developers should test functionality and scalability before tools ever leave their desktops. Part of this process involves testing components individually before integrating them.

Bowstreet, a business-to-business e-commerce company in Portsmouth, N.H., is using RadView's WebLoad product to do load, performance, stress and reliability testing of the HTTP server for its Business Web Factory product.

RadView's WebLoad Console delivers centralized management of all WebLoad functions, including test setup, test management, real-time monitoring and post-test session analysis.

Bowstreet needed a tool that would provide HTTP testing, performance testing, support for Windows NT and Solaris, reliability, and reporting and statistics abilities, as well as work in a distributed fashion, said Lance Hartford, the company's senior software quality engineer. Bowstreet also considered LoadRunner from Mercury Interactive Corp., Sunnyvale, Calif., and SilkPerformer from Segue Software, Lexington, Mass., before choosing WebLoad.

Cost was a major factor when the tools were evaluated, Hartford said; WebLoad cost one-quarter of the price of competitive tools. Bowstreet also liked the fact that WebLoad was very much geared toward testing Web applications.

The company is very pleased with WebLoad and RadView's iterative approach to testing. "Testing needs to happen very early on," Hartford said. "The people who are developing need to be testing as they go to make sure there are no surprises in the end."

Four-tiered approach
Mercury Interactive takes a four-tiered approach to implementing testing into an integration project. The first step involves applications requirements gathering in which the development team decides what platforms the system needs to be able to run on, how many users it needs to be able to handle, what actions it needs to be able to perform, and so on.

With the requirements knowledge in place, test planning can begin. This is where a company sets up tests to cover its requirements. Mercury's TestDirector product can generate a skeleton, based on the requirements knowledge, that plans and builds all tests, both manual and automated.

The third stage is execution—running the tests and getting results, while the fourth stage is keeping score of the results and identifying defects. TestDirector identifies and fixes these defects, and also ensures that they are fixed before the product goes into production.

Total Systems, a Columbus, Ga., credit-card processing firm, is using Mercury's TestDirector and WinRunner products to test its integration processes. The company is using these tools to track its test plans and automate its business processes, as well as to maintain its input and output results, according to Carey Thornhill, automation architect at Total Systems.

Total Systems considered a few other tools, but none did what the company was looking for. Each of Mercury's products is integrated with its other products, said Thornhill, and they are all integrated with Microsoft products. This provides Total Systems with the functionality that competitive products did not have at the time of evaluation.

The credit-card processor uses three different environments: one for quality assurance testing, one for user-based testing and one for a user production lab. "For each environment, we have the same business process," Thornhill noted. "TestDirector allows us to use a workgroup product for the entire process. WinRunner allows us to verify input and output for each of those different environments as well."

Mercury's TestDirector TestLab Manager schedules test runs on grouped machines to optimize testing resources. It runs tests based on predefined rules that are based on time or on the pass, fail and completion of other tests.

Prior to implementing Mercury's tools three or four years ago, Total Systems did everything manually. Now "we're able to focus on more complex problems because a lot of the trivial day-to-day tasks have been eliminated," Thornhill said. And integration is what keeps the company going.

The best way to implement testing into an integration project, according to Thornhill, is to analyze the project, implement some type of test organization tool, and then work to the automation. "It makes the transition a lot smoother," he said, "because you define the business processes ahead of time."

Process, products and people
Compuware Corp.'s approach to application integration testing is to concentrate on the combination of process, products and people. "You can't survive without automation," noted Doug Turner, vice president of product management at the Farmington Hills, Mich., software vendor. Test automation, he added, is a combination of the best product, the best process and the best people.

"Functional testing," Turner continued, "validates correct user interaction with the application and helps confirm the correct integration with back-end systems. With Web-based applications in particular, functional testing triggers integration with other processes and applications." On a Web site, for instance, when a customer makes a purchase, the purchase is often approved by an online credit authorization system, then handed over to a billing system and finally forwarded to the inventory database.

It is imperative that this integration work smoothly and properly. To ensure that it does, the integration of these applications and components needs to be tested thoroughly. "Too many people take the approach from a testing point of view of trying to test quality of performance into an application," Turner said. That is a fatal flaw, he added. "You can't test the quality back into an application. Testing needs to be built into the process from the beginning, especially as it relates to the development of a Web-based application."

Guild.com, an online art retailer based in Madison, Wis., is using Compuware's QARun, QADirector and QALoad to test its integration processes. QARun is a test creation and execution tool that automates test-script generation. It records user actions and system responses into reusable scripts that test specific app functions. QADirector provides a framework for managing the entire testing process, while QALoad simulates the load generated by hundreds or thousands of users on an application to test for scalability.

Guild.com considered comparably priced products from other vendors but was concerned about the learning curve associated with them, said Nathan Harper, chief information officer. The company already had some experience with Compuware, so it seemed to make sense to stick with them.

Harper is pleased with the decision. "Compuware's tools let us do code-level testing, unit-level testing, integration testing and testing of our Web site to see how it handled stress," he said. "Before, we could handle about 25 simultaneous users with a 25% error rate. Now we can handle 1,000 users with a 1% error rate."

To ensure smooth integration, Harper has a QA person on each development team. These people are part of the team from the beginning and help set the schedules. "If they're not [included from the beginning], we can't be sure the schedules are going to be accurate," Harper said.

Berkeley, Calif.-based Nolo.com is also using Compuware products. In addition to QARun and QADirector, the firm is using Compuware's TrackRecord, File-AID and NuMega DevPartner Studio Enterprise Edition to test its integration processes. TrackRecord is a defect-tracking product, while File-AID lets users find, create, extract, transfer, convert, load, edit, age and maintain data. NuMega Dev- Partner Studio (formerly DevCenter) is a suite of integrated debugging, testing and management tools for development teams using Microsoft Visual C++, Microsoft Visual Basic, Java, ASP or HTML.

Nolo.com provides access to legal information in plain English through books, software and a Web site. The firm, said Chad Henry, manager of quality assurance, needed code coverage, a way to verify data integrity up front, and software that could work on both shrink-wrapped programming languages and Web sites. Nolo.com also wanted other departments to be able to use the same tools.

Nolo.com considered products from Mercury and Segue, but found them to be focused more toward "just the QA stuff," said Henry. "They didn't have the breadth of functionality we needed." Compuware was able to handle C++ and Visual Basic plug-ins right out of the box. "We would have had to have extra components for Mercury," explained Henry.

File-AID lets Nolo.com tie in QARun with it to generate test data. And NuMega DevPartner Studio works with all of the different platforms the company uses. It allows the legal information provider to check for memory leaks and do code coverage analysis. "It gives us a lot of information we didn't have before," Henry noted.

Eliminate risks up front
Rational Software Corp., Cupertino, Calif., believes the key to application integration testing is an iterative approach. In an iterative approach, according to Sam Guckenheimer, senior director of technology for automated testing, iterations are prioritized based on risk. Integration, he said, is one of the highest risks in any project.

Rational does everything based on its Rational Unified Process (RUP), which mandates doing the risky things first. The idea, Guckenheimer said, is to "do the integration early, and test that and prove it as part of an early iteration. The key emphasis in all our testing products is to be able to do integration tests early to drive out the risk."

Milwaukee-based Rockwell Software Inc. is using Rational's Robot test recording tool, the RequisitePro requirements management tool and LoadTest for the majority of its testing tasks, said Andrew Krebs, test manager in the Logic Business Unit.

The industrial software provider develops software requirements with RequisitePro, extracts the test cases and then automates them. It then builds the tools to enable first-level automation, said Krebs. "This allows us to concentrate on the higher level functions. The Robot and LoadTest combination allows us to create Web reports that give dynamic status of the current state of any one product under test."

Rockwell had used Mercury's WinRunner in the past, but the flexibility of Rational's tools and the quick, reliable scripts sold the company on Rational's products. "Due to the flexibility of Robot and RequisitePro, and the fact that we are not so tied down to the GUI, we can build and implement faster with more reliability," Krebs explained.

Integration process testing is crucial to Rockwell due to its rapid development cycle. "Without Rational's toolset we would spend about six months in a cycle. With Rational's toolset, it takes us two months to complete the integration cycle," Krebs noted.

According to Krebs, defining your function feature set is the key to integration testing. "The simple function feature set approach will always build into an integration test set. You define what your function feature set is, automate what you can iteratively, and build with the system/integration in mind," he noted.

Test the middle tier
Segue Software has a bit of a different approach to app integration testing. It is focused on the e-business space and is embracing the concept of a multitiered architecture for e-business. The tiers include a Web front end, business logic, and databases and data repositories. The middle tier of middleware and business logic is often overlooked in a testing process.

Segue's SilkPilot 2.0 displays script execution status while testing transactions completed across two integrated applications.

Part of Segue's strategy is to test early and often. "Start testing as early as you can because [you will have a] smaller component to test. Then build on that," said Todd Scallan, vice president of the California development lab. "When you go to test the whole application, it won't be such a daunting task."

Manhattan, N.Y.-based brokerage firm BNY/ESI is doing exactly that using Segue's SilkPilot product. SilkPilot offers functional and regression testing of middle-tier servers. "Developers are using testing tools as part of standard component development to alpha test all methods on a component based on regression test set data," explained Bruce Ostrover, vice president of Global Trade Management Systems. "Since there are very few tools on the market that allow us to invoke components on our app server easily, this tool provides us with a mechanism for doing that."

BNY/ESI needed the ability to invoke middleware components easily, the ability to script out component calls to be read back, as well as readability of data. The brokerage firm considered tools from other vendors, but none met its requirements. Segue's SilkPilot allows the firm to test each method within a component as well as keep a repository.

"Testing those components in their own domain allows us higher integration testing when we're completed," Ostrover said. "Knowing that a component is well-tested allows us to forecast the component's reliability." The key to integration testing, according to Ostrover, is to start testing when development starts. "Testing has to start with the developer and move up to QA. Developers have to be responsible for testing the components they're developing," he said.

The verdict is clear: An iterative approach to testing is the best way to ensure smooth application integration. But application integration testing needs to be an ongoing process. And an iterative approach to testing is becoming more of a necessity as more e-businesses abound.

Testing tool vendors seem well positioned for an iterative approach. Now it is up to development teams to jump on the bandwagon. While iterative testing may seem like more work, it leads to a proliferation of better quality tools. As Nolo.com's Henry said, "The work doesn't become shorter quicker, but the work gets much better and much more thorough."