In-Depth

Are You Ready?

• By Deborah Melewski
• March 1, 2000

The concept of online shopping holds great appeal, especially during busy holiday seasons. One can shop online 24 hours a day, seven days a week. Minimum requirements include a PC with a browser and modem, and a credit card.

Pretty much anything you would want to purchase is available online - books, groceries, music, vitamins and lingerie - with more items joining the list daily. According to the so-called e-business plan, orders go through hassle-free and goods are delivered to a shopper's front door within a few days.

That formula, of course, is accurate only when everything goes according to plan. Unfortunately, things can and do go very wrong, and it takes just a few failures before an online shop can face a catastrophic result. Some observers call the Internet "The Great Equalizer," because all businesses start out on pretty much equal footing. But therein lies its ultimate challenge: There are multitudes of online cybershops struggling for a profitable share of this booming online market, and the competition is but a click away.

Yes, out of the box, your Windows machine can participate with transactions that may span local and remote databases, as well as send and read messages within atomic units of work.

Forrester Research Inc., Cambridge, Mass., has estimated that by year-end 1999 more than 17 million households had shopped online, and that online retail revenue for the year would total about $20.2 billion. In such a rapidly growing marketplace, online retailers must dramatically shorten application development cycles in order to be competitive. Those that cannot handle surprise heavy loads face negative stories in both traditional and Web-based media outlets.

For example, online pioneer eBay faced widespread criticism last year when its site crashed following the posting of standard changes - not an unusual occurrence on the Web, which is all about the ability to post changes regularly. There have also been many problems of sites where credit card clearances take an inordinate amount of time, orders are taken for unavailable products, and sites are inaccessible for hours.

In February, the need for testing and securing e-commerce sites became even more obvious when hackers crippled major online companies such as Yahoo!, Buy.com, Amazon.com, eBay.com, eTrade. com and CNN.com. The so-called Denial of Service was caused by hacker attacks that flooded computers at these sites with an overwhelming number of data packets.

Likened to bolting shut the doors of a major retailer, online outages such as these have cost cyber-companies access to the dollars and trust of customers. Most sites recognize the need for testing links, data accuracy, browsers and load capacity. But what can be done to test for such unpredictable cases of cybercrime?

Experts believe that security issues may be at the root of the problem. As a result, most major online forces are scrambling to beef-up security. At the very least, companies must ensure that their computers are free from attack and free from hackers that would use them to assist in other attacks.

While load testing probably will not prevent large-scale server pummeling due to cybercrime, there are ways to test the effectiveness of a site's defenses. Some tools can emulate an attack, allowing testers to determine site infrastructure vulnerabilities and to evaluate response time fluctuations as defenses are activated.

Today, the big question for many managers is, 'What can IT do to avoid becoming tomorrow's bad news?'

Test, test and test

For many organizations the answer is simple - implement a tenacious testing process that leaves nothing to chance. Testing was important in client/server development, and it is vital for Web development.

Vic Salemme, manager of QA at MotherNature.com, Concord, Mass., is charged with keeping that online company's site up and running glitch-free 24x7. From a QA perspective, Salemme stresses that testing is absolutely required to ensure that masses of people can enter the site, remain on the site, search a catalog of some 25,000 natural health products and make a purchase. "If the customer can't do that, they'll go elsewhere," Salemme said.

MotherNature.com employs a multi-server environment comprising seven to 10 servers in a round-robin queued server environment. It is imperative that the servers are available at all times, and that the database can be accessed and searched from anywhere in the world. The Mother-Nature.com database houses some 13,000 health-product categories, including vitamins, minerals, supplements, sports nutrition, teas and herbs.

Engineers at MotherNature.com began testing the system early in the pre-deployment phase, and have continued in post-deployment. MotherNature. com hastened the testing process by using the packaged e-Suite toolset from RSW Software based in Waltham, Mass. "It was the quickest to deploy, and is specifically manufactured for Web-based organizations," said Salemme.

MotherNature.com uses the e-Monitor component of e-Suite to perform server response testing. A record-and-playback method alerts the firm to server delays and allows for immediate corrective action.

Because MotherNature.com does a lot of advertising, there is constant concern about increased loads after any strong marketing push. Salemme said the load-testing process has been the key to avoiding perilous overloads. Facing crashes after marketing campaigns "essentially implies there wasn't a lot of work on [the firm's] end in terms of load testing prior to deployment. We do that testing, and then we continue testing in live server environments prior to any advertising campaign," he said.

The test environment

at MotherNature.com is test case and intuitive-based. The company knows that most people go in and search the site for products; in fact, the process calls for covering at least 80% of a typical user's actions in a record-playback scenario. "The scripts have been right on," said Salemme. "They have kept us from site outages, eliminated downtimes and they project when we'll see new loads."

Keep it proactive

Send.com, a high-end online gift-giving business in Waltham, Mass., faces many of the same online challenges. The company just concluded its third Christmas season online, its busiest time of year. Send.com uses a network of merchants around the country to gift wrap and ship personalized items such as fine foods, cigars, crystal, flowers, beverages or a day at a spa.

The company re-evaluates every aspect of its infrastructure on an ongoing basis, and grew significantly during 1999 in preparation for the holiday season. Kenneth Surdan, vice president, technology and systems development, noted that as a characteristic of performance the firm "manages the heck out of the network.

"People like to think of the Internet as a homogeneous kind of concept, a ubiquitous network that runs the same everywhere," said Surdan. "Folks who've spent any kind of time trying to run a business on it have learned very quickly that it's very heterogeneous. It's got a million interconnected pieces, and tons and tons of vendors, and it gets difficult to try to figure out point-to-point what's going on.

"If you don't know about the problem, you can't fix it," added Surdan. "You read quite often about different site outages and problems. Sometimes those are blamed on a single point, and sometimes the company takes a hit because nobody cares that it was the network provider."

Send.com must also guarantee that the application will perform at levels required to keep to its business forecasts. This brings about another set of business challenges, such as how to simulate 20,000 or 50,000 simultaneous users, and how to ensure that a design will scale. "We need to know where it can break, why it can break, and how can we break it before the business does," noted Surdan.

To have a safety margin on every aspect of the app - network, hardware, database, firewall, Web servers and the like - all of the parts must be joined together as one entity. Send.com resolved that issue by combining individual test cases and test plans with automated tools. They also brought in the LoadRunner testing tool and the Astra and Topaz performance monitoring and diagnostics tools from Mercury Interactive Corp., Sunnyvale, Calif.

"The tools help us get through testing more quickly and with a smaller team," said Surdan. Different test scripts find different bottlenecks, he said. The company tests for Web server capacity and can also get a read on the middle tier or database transactions. The results can show, for example, whether a database server is big enough for the app, whether transactions slow significantly with heavy loads, and which components are involved in specific tasks, he said.

Testing of the site is constant. "You have to be more precise on the Internet, and you have to be sure you can scale quickly," Surdan said. "Unlike the internal organization where you know how many simultaneous users you will have, you can have any number of people show up on any given day on the Web."

Testing in Internet time

Perhaps one of the most dramatic differences between testing on the Web and testing in traditional or client/server environments is the rate and pace of innovation and change. When IT projects ran 18 months or more, project schedules sometimes allowed as much as six weeks for test planning. Longer testing cycles and an ability to retest were built in to the process.

Not so on the Internet. "In the Web environment, especially in the dot.com space, the pace is blistering," said Steve Caplow, director of marketing and business development at RSW Software. "Some of these development projects are literally about four to six weeks long from concept to deployment."

Indeed, many dot.coms may have only a week prior to deployment for a complete testing operation. And a week after deployment, it is not unusual for a new version to be required, which means having to modify or rebuild all existing test scripts.

RSW built its tool suite specifically around Web technology for testing browser-based applications. Mercury Interactive and Newton, Mass.-based Segue Software, both long-time players in the traditional testing environment, recently added product lines devoted to the Web architecture. Once the appropriate tools are chosen for a specific project, a company is often faced with the daunting task of knowing what, when and where to test.

Professional services

In addition to testing tools, many an IT manager looks for outside help in quickly developing and implementing a testing process. And there are more and more service operations available to help dot.coms.

Inforonics, a professional services provider and Internet hosting firm based in Littleton, Mass., has uncovered multiple testing challenges in e-commerce, said CEO Bruce Buckland. "One of the biggest challenges is the rapid rate of change required by e-business," said Buckland. He explained that dot.coms have to put new features and capabilities into their sites regularly in order to remain competitive. Such requirements are unprecedented compared to traditional software development processes, and thus warrant a different approach to testing and deployment.

For starters, Buckland points out that Internet applications must be managed differently. A key notion is to implement changes that can be reversed easily if a problem arises. IT organizations became very aware of this requirement when the eBay site crashed. eBay lost full use of its database for several hours, causing customers to abandon the site in droves.

"We have a methodology for doing this," said Buckland. "We engineer changes to applications that are running in such a way that you can back an application change out without backing the database change out, for example."

Buckland also emphasizes the need for both testing and monitoring tools. "A broad and diverse set of tools can help in different situations," he said.

Guild.com Inc. is one dot.com company that learned this lesson. The Madison, Wis., firm offers a large collection of original art items for sale online. The company was formed as a venue for more than 1,200 artists to present their work to a larger audience, and went online in April 1999.

Guild.com had originally outsourced the design, development and hosting of its Web site to a local Internet Service Provider (ISP) that the company would not identify. The plan also called for a local ad agency to design the site's look and feel, and for the individual artists to be responsible for fulfillment.

Nathan Harper, CIO at Guild.com, was recruited in 1999 to take the Web site forward because of problems with the original plan. "We discovered that design and development of the original site was business for the ISP, but not critical to them; so a delay on their part didn't strike them as very important," said Harper. "To us, delays meant lost business and opportunity."

To make matters worse, the original site utilized a peculiar database - the coding was done in the Perl language and not very well documented - and the hosting standard was not at the level required. At one point, Guild.com asked about 50 of its employees to get on the site simultaneously for a stress test. "We discovered a failure rate of about 75%," Harper said. "This meant that our customers were experiencing the same thing; that only one out of four people could complete a transaction."

That was the final straw. "We fired the ISP, declared independence, brought in our own engineers and took off down the road," said Harper. Since then, Guild.com has ported the database to Oracle, rewritten the Perl code to Java, and moved the site to the Exodus Data Center in Illinois. The company also purchased a new IBM RS6000 NonStop system to ensure high availability.

Given the nature of its business, Guild.com decided it did not make sense to bring in an entire QA department, so they opted to contract with Farmington Hills, Mich.-based Compuware Corp. for services. The company also licensed Compuware's QA line of testing tools.

Compuware's QA Load tool was used to stress test the site with 2,000 simultaneous users. Because the site is used only for e-commerce, the most important testing was the ability of a customer to purchase items from the site.

The project also called for redesigning the site's interface. For example, would the graphics look the same to an AOL user, and how does the site look on a Mac, on a PC or with Netscape? It was also important that the site's search routine be tested because Guild.com carries a selection of some 7,000 items.

The new site went online on November 17, 1999. "We anticipated very well, so [we] had no negative surprises through the holidays," said Harper. "There was no downtime, no slowdown in traffic; the site remained robust."

Developers used the QACenter tools for automated testing and performance, while the Compuware service unit created a methodology to help with what assumptions should be made during testing. Often, concerns included how many users to test for, what types of transactions to monitor, and what should be tested and how.

Guild.com continues to use the tools on an ongoing basis. "When we're ready to bring up our next major release we can probably bring the Compuware team back," said Harper. "Having your own engi- neers testing their own work just doesn't cut it for the Internet."

Site operation is key

The SmarterKids.com online educational toy store has been operating since November 1998. There are approximately 4,000 items for sale on the site, including software, books, games, puzzles and toys.

"We have all the usual e-commerce challenges, including customer acquisition, inventory, fulfillment and customer service," noted Rich Secor, vice president and CIO at the Needham, Mass.-based concern. " If the site isn't available, all those other things don't really matter," he added.

The site uses the Exodus Internet Data Center for hosting. Exodus provides the real estate, high-speed Internet connection and security, Secor said. SmarterKids.com's staff of Internet engineers designed and built the site and is responsible for administering its own server.

Testing tools were brought in very early in the game, but it was clear in 1998 that the online company was not ready for the testing tools and that the tools were not ready for the site. SmarterKids.com went through its first holiday season using internally written testing tools combined with some Microsoft testing technologies. In early 1999, the firm evaluated and finally chose the e-Test suite from RSW Software.

To gear up for the 1999 holiday season, SmarterKids.com used RSW tools for functional testing of its site, to verify scalability, and to monitor the site and ensure performance. SmarterKids.com saw an enormous increase in its loads over the holiday, and the architecture supported the demand. Officials credit regular testing and a formula to calculate the sizing of the Web site based on projected sales.

eCampus.com is an online college textbook store in Lexington, Ky., that went from concept to deployment in six months. Ted Willis, QA director, and Tom Wright, lead tester at eCampus.com, turned to the Rational Suite of tools from Rational Software, Cupertino, Calif. The Rational product bundles configuration management, defect tracking and testing tools.

The site's busiest times are in the fall and in January between semesters. Load testing provided a gauge of how many users the site could handle, and allowed eCampus.com to spot and resolve problems before its last rush. According to Willis, problems typically can stem from a high number of users executing time-intensive processes such as searching the database. Another difficult area for e-commerce firms, Willis said, is figuring out how many people will access a site in a specific time period.

Testing was done prior to going online, Willis said. Now load testing is done with all major code changes to the application. In production, the company is writing scripts that go out and check the site every 15 minutes. If a problem is detected, operations people can fix it. "It's important to find a problem before it happens; you want to go more in-depth than just checking the site to see if it's alive," Willis said.

Changes can also be a force of contention for QA - something this industry has never seen before - because the Internet is dynamic. "If we versioned our code every time a change went up, we'd probably be in triple digits by now," said Willis.

Jewelry.com, El Segundo, Calif., went online in November 1999. The company uses CandleNet from Candle Corp., Los Angeles, to test application performance and availability. CandleNet monitors customer response times by pinging the site at regular intervals and providing reports that alert Jewelry.com to any potential downtime and the need for more hardware or memory. The company did not use testing tools, which proved to be a problem in the early implementations.

"We thought we were going to have one of two problems with our site," said Paul Rajewski, chairman and CTO at Jewelry.com. "One, that we wouldn't have enough traffic, or two, that we wouldn't have planned for enough traffic."

The latter turned out to be the case. Despite completing complex internal calculations, traffic was triple what the site was built for as the company began advertising campaigns. The accompanying problems left Jewelry.com scrambling for new gear that could support very high numbers of users.

The online jewelry store built its system to be fault tolerant, using dual systems along the way to prevent hardware outages. The front end was redesigned using Sun's NetraSystems to handle increased loads expected for the Valentine's Day rush. "Our objective there is that even if one or two servers go down, it won't affect things because traffic will be automatically redirected," said Rajewski.Rajewski said the firm did standard testing, such as trace routes and pings, but conceded that testing tools were not used. He now expresses interest in these types of tools. The site has benefited from CandleNet, which confirmed suspicions that the site was too slow.

The presence of online sites has grown significantly. Many dot.com firms, hoping for increased traffic, even risked 50% of their projected revenue this year to advertise during the Super Bowl. With an eye to the future, most agree that testing and planning for site growth are required on an ongoing basis.

Inforonics' Buckland stresses the importance of keeping an online company's operations and systems people involved in the decision to go out and spend millions on advertising. Be warned: If your system is not tested and ready, then your e-commerce Web site can die a quick death.