In-Depth

New ways to test packages

They are widely known as “off-the-shelf” applications, but ERP and CRM packages are not off-the-shelf applications at all, which can mean trouble for testers. That is because there is a big helping of customization in every working ERP and CRM system, which in turn makes testing the bellwether applications -- SAP, PeopleSoft, Siebel and Oracle -- a fairly large task.

At the same time, integration is growing as a stubborn test issue in the packaged app world just as surely as in the proprietary sphere. With CRM and ERP apps increasingly linked to homebrewed and Internet-based applications, the job of testing grows even more difficult.

Yet effective testing can cut the cost of ERP system configuration. Enterprises can reduce the total cost of an ERP implementation or upgrade by 10% to 20%, according to the Boston-based technology research and consulting firm Yankee Group, through a business technology optimization strategy that includes system testing, tuning and performance management focused on user experiences within key business processes.

“Nobody takes a piece of off-the-shelf software and implements it into their organization,” quipped David Kapelanski, ASQ director at Compuware Corp., Farmington Hills, Mich. Instead, users take the functionality of a packaged application and customize it for their own use.

“That puts packaged application vendors and customers in a tough spot in terms of predicting how applications will perform,” added Dan Koloski, director of Web testing solutions at Empirix, Waltham, Mass. This customization is driving the need for testing.

The late 1990s saw numerous customers buying flagship ERP and CRM offerings and putting them into production without testing, said Theresa Lanowitz, research director in Gartner Inc.’s Santa Cruz, Calif., offices. That time period also saw a lot of failure in these apps, she noted. Now they are starting to come back into IT shops and need good testing, she acknowledged.

“Packaged application vendors, because of some failures and lack of testing, realized they couldn’t provide automated testing tools,” Lanowitz explained. “Nobody thought about integration -- with existing applications, with systems, with all the servers and so on -- before.

“These packaged apps tended to have a life of their own. Now things are brought back into the greater IT world,” she said.

As IT departments move toward integration in many areas, and as technology matures, it only makes sense that testing tools and ERP/CRM applications should be integrated as well.


Driving forces

One of the driving forces in the rise of integration is a fundamental shift in architecture from client/server apps to Web-based apps. Many packaged applications vendors are moving to a .NET or Java architecture, which affects new ERP/CRM apps and testing tools. “The world of browser and Web applications is a different set of issues than client/server,” explained Empirix’s Koloski.

“The technology advancements that ERP and CRM vendors are putting into their applications are throwing tool vendors for a loop,” said Jeff Schuster, director of product management, automated testing at IBM Rational in Lexington, Mass. “Unless you know about the details of an application, you’re not going to do a very good job testing that application.”

John Williams, CARS launch director at Compuware, said that testing tool vendors have always been interested in this kind of integration, but “there has been some resistance from packaged application vendors in the past.”

Compuware’s Kapelanski agrees. “Most ERP vendors out there create technical environments unique to their own product to differentiate themselves,” he said. “There’s nothing generic off-the-shelf I can buy to test that application. Technology is not something I can go out and get shareware for.”

And IBM Rational’s Schuster admits that “ERP vendors would like to have a set of packaged tests they can push out with their ERP packages.”

Packaged application vendor SAP AG, headquartered in Walldorf, Germany, seems to be backing that up. The company claims to have built the majority of its testing tools itself.

“We want to be able to ship all of the necessary testing tools to our customers so that as they use the system, they can use the same testing method we apply,” explained Rainer Zinow, VP of the MySAP CRM interaction center. “We believe that every customer needs to have these testing tools in-house. We don’t want customers to spend another amount of money on testing tools;

then they would rather not buy [the package].” But test tool vendors would beg to differ and, in fact, ERP vendors such as SAP have begun to forge new links to these third-party test offerings.

The problem with using the incumbent ERP test tool, according to IBM Rational’s Schuster, is that it only makes sense for packaged app vendors to push their own testing tools if an end user is not making any customization. But users that fall into that category today are rare.

How fast and how well packaged apps work is the issue. The real driving force behind today’s vendor collaborations are the high stakes in these implementations. Quality is of the utmost importance.

Customers of ERP and CRM apps “are saying, ‘We don’t need more functionality; we need better quality, better performance, better reliability,’” noted Jonathan Rende, VP product marketing at Mercury Interactive Corp., Sunnyvale, Calif. “If you focus on quality, you don’t have to sacrifice any one of those.”


Hidden integration?

A slew of recent strategic partnerships and integration announcements show the future of enterprise application testing. “One of the things we’re starting to see is collaboration coming to a point where ERP vendors are integrating products from a tool vendor and building connections into their own ERP systems,” pointed out Compuware’s Kapelanski.

According to Kapelanski, SAP went to Compuware to take Compuware’s TestPartner and integrate its capabilities into the MySAP Extended Computer Aided Test Tool (eCATT) environment. SAP has an interface called BC-eCATT that allows third-party vendors to integrate their test tools with eCATT. This integration then allows SAP users to start third-party test tools, and to maintain and execute test scripts from the well-known SAP GUI. “From within their own testing environment, [SAP developers] can execute [eCATT] using Compuware’s tool,” Kapelanski explained.

In addition to SAP, Compuware has collaboration agreements with Oracle and PeopleSoft to develop baseline scripting technology to use against those particular applications to ensure that they will work, he said.

Other testing tool vendors have also risen to the challenge and forged agreements with packaged application vendors. Segue Software, Lexington, Mass., joined San Mateo, Calif.-based Siebel Systems’ alliance program and announced Siebel validation. This means that Segue’s SilkTest 6.0, SilkPerformer 5.1 and SilkVision 2.0 products are integrated with Siebel 7 and can test and monitor Siebel eBusiness applications.

Mercury has agreements with SAP, Oracle, PeopleSoft and Siebel with its Optane suite of products. Mercury, after tremendous success in the development test sector, is increasingly positioning itself as a business technology optimization firm. It has partnerships with all four vendors in three areas: technical alliances, standardization and channel use of its solutions. Mercury also has a fourth-level agreement with SAP as a channel reseller.

IBM Rational, for its part, claims to have agreements across the board, said the firm’s Schuster. IBM Rational’s specialty is testing tools, but the firm has been establishing relationships with ERP/CRM vendors during the past three to five years, Schuster said. And now that it falls under the IBM umbrella, Rational is able to leverage some of IBM’s relationships.

Empirix has also established a relationship with Siebel. The testing tool vendor originally started working with Siebel because one of its areas of expertise is in testing voice applications. As Siebel is a CRM application that is often used in call centers, it seemed like a good fit with Empirix, said the firm’s Koloski.

Empirix is a Siebel software partner, and has certified and validated its products with Siebel 7. “We provide expertise, but they provide knowledge of their application,” Koloski noted. “It’s essential that you have both to test an application.”

RadView Software Ltd., Burlington, Mass., has an agreement with SAP, specifically for the MySAP application. RadView’s WebFT functional testing product and WebLoad load-testing tool give SAP users the ability to verify the quality of an application before deployment.

RadView’s offerings now include Web services support, said Laura Naylor, senior VP marketing. In addition, the products utilize the same test script throughout the entire development life cycle.


Testing tools mature

IBM Rational has been focusing on deeper-level support for Web infrastructure and Web client testing, according to the company’s Schuster. It came out with a new product, XDE Tester, which is written in Java and targeted specifically at testing HTML and Java-based applications. IBM Rational is also focusing on deeper-level support for end-user customization of solutions, Schuster added.

After doing an executive study, Compuware discovered that there were significant challenges in delivery quality apps that did not fail, said the firm’s Williams. “A large percentage of companies had invested in tools or processes, but those did not lead to less failure or higher quality.”

With that knowledge, the company created the Compuware Application Reliability Solution (CARS), which combines processes, technology and QA expertise. According to Compuware’s Kapelanski, one problem has been that firms have tried to use one of two groups when implementing ERP/CRM systems: a group inside their company that has no knowledge of ERP/CRM systems; or a large outsourcing firm that comes in, does the work and walks out the door.

“Compuware is saying, ‘We’re going to come in with our people and show you this process, but [we’re also going to] cross-train your people so when we walk out the door, the knowledge transfer stays with your organization,’” Kapelanski said. In addition to CARS, Compuware has reengineered its TestPartner product for the new technology market.

Empirix’s tools have evolved along with technology, noted the company’s Koloski, adding that Empirix has always been focused on next-generation applications. The company has now set its sights on Web services, J2EE apps and .NET.


Shelf-life evolution

Close work between test vendors and application vendors will probably gain in importance. That is because these “off-the-shelf” applications are constantly evolving.

“What we are finding is that a key area is object identification in off-the-shelf apps such as ERP apps,” said Mike Sowers at vendor Software Development Technologies (SDT). “These apps, as they have been evolving, have been creating more customized objects,” he said. The problem is that the custom objects could break automation scripts.

Proprietary objects might be giving way to newer, more standard objects, but that does not mean the test job is simplified. Testers must “keep up with the latest technology implementations as application vendors move to Java and .NET component architectures,” said Sowers.

“An object to test may have previously been written in VBA, but now it’s in .NET,” he said. The issue is exacerbated for testers because the degree of integration of the various components may be increasing. “You need a set of tooling that understands these apps. The vendors are good at it to the extent that they can keep pace with the underlying technology innovations,” said Sowers.

“The world was moving and embracing VBA, but now the world is beginning to embrace VB .NET. As a test tool vendor, I have to modify my tooling to address those issues,” he said.


Early to test?

As enterprise applications become part of a more complex “integration ecosystem,” it does not become easier to follow the age-old testing admonition to “test early.” Until the various systems (including Web servers, file servers, application servers, data stores and more) are actually integrated, it is hard to characterize performance. Meanwhile, load test tool vendors are ever improving their offerings to better characterize complex loads.

“One of the key issues we’ve seen is in the whole area of designing tests, whether off-the-shelf or not, and maintaining the right core competencies related to testing,” said Sowers, taking a broader view. And behind all these advances, and despite a move to more automation, test practices are evolving in such a way that humans still have an important role, he noted.

“Where we see this headed is toward an evolution that parallels the evolution of software engineering. Once [upon a time], a programmer was a programmer, and we didn’t distinguish roles. But now we have designers, business analysts and architects, as well as programmers. That’s exactly where we see the testing profession evolving,” he said.

“We are beginning to have the test architect, test designer, test planner, test manager and test executor. In this way you can bring core competencies to bear to make testing more effective,” he said.

That means a special role for individuals with “domain knowledge.”

“If, for example, I know the SAP or CRM world, I then know the domain,” noted Sowers, “so I am very good at designing tests for the SAP environment.” When your testers can assert this degree of expertise, then you can get better at finding defects early, he said.


Packed fair and square

The certification efforts of test automation vendors are important steps toward taming the deceptively simple packages, said Compuware’s Kapelanski. “For certification you go against their apps to prove you can handle not only the apps, but the objects created for developers to use to create customized applications,” he said. “And it can be a pretty rigorous routine to handle those objects.”

Still, like others, Kapelanski sees objects as less of an issue as ERP and CRM fat clients give way to thin clients using HTML. Behind the scenes, changes in object architecture are occurring.

“What’s important is that people test their apps against the delivered product from ERP vendors,” he said. “Application development groups have typically manipulated the delivered applications they received from the SAPs, PeopleSofts and Siebels.” When subsequent updates to application suites arrive, these changes may not make the transition on first pass.

“People buying ERP systems today want to prove that it works in their environment before the release. What happened the first time around with these applications is that they all got stung,” said Kapelanski.

Both load testing and function testing are at work here, said Kapelanski, but he suggests the load testing of ERP apps to gauge, or reduce, hardware requirements is still in the early stages of adoption.

“The changes developers made on delivered systems might have bogged down the back-end execution. Because of the changes customers made, it didn’t execute like the vendor said it would [execute],” said Kapelanski. “We are talking about a combination of functional and load testing. We are starting to see an increase in load testing of these apps.”

Load-testing automation comes to the fore in these settings as thin, Web client architectures are more widely used. With the so-called “fat clients” of yore, the system architect had a more definite understanding of the number of users the system would serve.

The response to “throw hardware at the problem” is less heard these days. “People are looking at load along with server traffic, and consolidating servers and making code changes to make the system run optimally on the client,” said Kapelanski. “They are watching what they are doing hardware-wise.” Network, server and load tools are now required.


Are we there yet?

While the testing tools market is pretty mature at this point, tools for testing the integration of packaged and other apps are just beginning to arrive. IBM Rational’s Schuster believes this particular market is in its infancy. Gartner’s Lanowitz believes maturity in this market is about a year and a half out.

As often happens with technology, the rules of the testing game may be changing just as some of the newest rules have been learned. Count Solstice Software as at least one vendor among several that believes application integration testing is becoming more of an issue than mere off-the-shelf application testing.

“The integration testing space is fundamentally different,” said Lori Gipp, vice president, marketing at Solstice Software (formerly Class IQ). She suggests that Solstice’s tools are most useful “if you got stuck with the integration process and you have to coordinate different teams and technologies, and have control over very little.” Solstice, she said, seeks to automate end-to-end testing of the message-based systems found in the modern enterprise. An extensive protocol library and simulation capability help to make this possible, Gipp said. Protocols covered include XML over HTTP, SOAP, JMS, plus MQ, webMethods and Tibco message types.

“We focus on the integration points; that means testing messages, not just GUIs,” Gipp said. She added that the company’s simulation capabilities allow testing early, when working systems are not yet available. “It is important to provide software spanning many technologies,” she said, as well as to orient planning to an “assembly-centric” view.


Please see the following related stories:

“Dow Chemical uses Optane to fine-tune SAP” by Lana Gates

“When in the cycle does formal testing begin?” by Jack Vaughan

“A sampling of testing tools” by Lana Gates and Maggie Madden

“Toolmakers await ‘wireless revolution’” by Mike Bucken


Or click here for a Web-only Special Report on Testing.

Most   Popular