Are You Ready?
- By Deborah Melewski
- March 1, 2000
concept of online shopping holds great appeal, especially during busy
holiday seasons. One can shop online 24 hours a day, seven days a week.
Minimum requirements include a PC with a browser and modem, and a credit
Pretty much anything you would want to purchase is available online
- books, groceries, music, vitamins and lingerie - with more items joining
the list daily. According to the so-called e-business plan, orders go
through hassle-free and goods are delivered to a shopper's front door
within a few days.
That formula, of course, is accurate only when everything goes according
to plan. Unfortunately, things can and do go very wrong, and it takes
just a few failures before an online shop can face a catastrophic result.
Some observers call the Internet "The Great Equalizer," because all businesses
start out on pretty much equal footing. But therein lies its ultimate
challenge: There are multitudes of online cybershops struggling for a
profitable share of this booming online market, and the competition is
but a click away.
Yes, out of the box, your Windows machine can participate with transactions
that may span local and remote databases, as well as send and read messages
within atomic units of work.
Forrester Research Inc., Cambridge, Mass., has estimated that by year-end
1999 more than 17 million households had shopped online, and that online
retail revenue for the year would total about $20.2 billion. In such a
rapidly growing marketplace, online retailers must dramatically shorten
application development cycles in order to be competitive. Those that
cannot handle surprise heavy loads face negative stories in both traditional
and Web-based media outlets.
For example, online pioneer eBay faced widespread criticism last year
when its site crashed following the posting of standard changes - not
an unusual occurrence on the Web, which is all about the ability to post
changes regularly. There have also been many problems of sites where credit
card clearances take an inordinate amount of time, orders are taken for
unavailable products, and sites are inaccessible for hours.
In February, the need for testing and securing e-commerce sites became
even more obvious when hackers crippled major online companies such as
Yahoo!, Buy.com, Amazon.com, eBay.com, eTrade. com and CNN.com. The so-called
Denial of Service was caused by hacker attacks that flooded computers
at these sites with an overwhelming number of data packets.
Likened to bolting shut the doors of a major retailer, online outages
such as these have cost cyber-companies access to the dollars and trust
of customers. Most sites recognize the need for testing links, data accuracy,
browsers and load capacity. But what can be done to test for such unpredictable
cases of cybercrime?
Experts believe that security issues may be at the root of the problem.
As a result, most major online forces are scrambling to beef-up security.
At the very least, companies must ensure that their computers are free
from attack and free from hackers that would use them to assist in other
While load testing probably will not prevent large-scale server pummeling
due to cybercrime, there are ways to test the effectiveness of a site's
defenses. Some tools can emulate an attack, allowing testers to determine
site infrastructure vulnerabilities and to evaluate response time fluctuations
as defenses are activated.
Today, the big question for many managers is, 'What can IT do to avoid
becoming tomorrow's bad news?'
Test, test and test
For many organizations the answer is simple - implement a tenacious
testing process that leaves nothing to chance. Testing was important in
client/server development, and it is vital for Web development.
Vic Salemme, manager of QA at MotherNature.com, Concord, Mass., is charged
with keeping that online company's site up and running glitch-free 24x7.
From a QA perspective, Salemme stresses that testing is absolutely required
to ensure that masses of people can enter the site, remain on the site,
search a catalog of some 25,000 natural health products and make a purchase.
"If the customer can't do that, they'll go elsewhere," Salemme said.
MotherNature.com employs a multi-server environment comprising seven
to 10 servers in a round-robin queued server environment. It is imperative
that the servers are available at all times, and that the database can
be accessed and searched from anywhere in the world. The Mother-Nature.com
database houses some 13,000 health-product categories, including vitamins,
minerals, supplements, sports nutrition, teas and herbs.
Engineers at MotherNature.com began testing the system early in the
pre-deployment phase, and have continued in post-deployment. MotherNature.
com hastened the testing process by using the packaged e-Suite toolset
from RSW Software based in Waltham, Mass. "It was the quickest to deploy,
and is specifically manufactured for Web-based organizations," said Salemme.
MotherNature.com uses the e-Monitor component of e-Suite to perform
server response testing. A record-and-playback method alerts the firm
to server delays and allows for immediate corrective action.
Because MotherNature.com does a lot of advertising, there is constant
concern about increased loads after any strong marketing push. Salemme
said the load-testing process has been the key to avoiding perilous overloads.
Facing crashes after marketing campaigns "essentially implies there wasn't
a lot of work on [the firm's] end in terms of load testing prior to deployment.
We do that testing, and then we continue testing in live server environments
prior to any advertising campaign," he said.
The test environment
at MotherNature.com is test case and intuitive-based. The company knows
that most people go in and search the site for products; in fact, the
process calls for covering at least 80% of a typical user's actions in
a record-playback scenario. "The scripts have been right on," said Salemme.
"They have kept us from site outages, eliminated downtimes and they project
when we'll see new loads."
Keep it proactive
Send.com, a high-end online gift-giving business in Waltham, Mass.,
faces many of the same online challenges. The company just concluded its
third Christmas season online, its busiest time of year. Send.com uses
a network of merchants around the country to gift wrap and ship personalized
items such as fine foods, cigars, crystal, flowers, beverages or a day
at a spa.
The company re-evaluates every aspect of its infrastructure on an ongoing
basis, and grew significantly during 1999 in preparation for the holiday
season. Kenneth Surdan, vice president, technology and systems development,
noted that as a characteristic of performance the firm "manages the heck
out of the network.
"People like to think of the Internet as a homogeneous kind of concept,
a ubiquitous network that runs the same everywhere," said Surdan. "Folks
who've spent any kind of time trying to run a business on it have learned
very quickly that it's very heterogeneous. It's got a million interconnected
pieces, and tons and tons of vendors, and it gets difficult to try to
figure out point-to-point what's going on.
"If you don't know about the problem, you can't fix it," added Surdan.
"You read quite often about different site outages and problems. Sometimes
those are blamed on a single point, and sometimes the company takes a
hit because nobody cares that it was the network provider."
Send.com must also guarantee that the application will perform at levels
required to keep to its business forecasts. This brings about another
set of business challenges, such as how to simulate 20,000 or 50,000 simultaneous
users, and how to ensure that a design will scale. "We need to know where
it can break, why it can break, and how can we break it before the business
does," noted Surdan.
To have a safety margin on every aspect of the app - network, hardware,
database, firewall, Web servers and the like - all of the parts must be
joined together as one entity. Send.com resolved that issue by combining
individual test cases and test plans with automated tools. They also brought
in the LoadRunner testing tool and the Astra and Topaz performance monitoring
and diagnostics tools from Mercury Interactive Corp., Sunnyvale, Calif.
"The tools help us get through testing more quickly and with a smaller
team," said Surdan. Different test scripts find different bottlenecks,
he said. The company tests for Web server capacity and can also get a
read on the middle tier or database transactions. The results can show,
for example, whether a database server is big enough for the app, whether
transactions slow significantly with heavy loads, and which components
are involved in specific tasks, he said.
Testing of the site is constant. "You have to be more precise on the
Internet, and you have to be sure you can scale quickly," Surdan said.
"Unlike the internal organization where you know how many simultaneous
users you will have, you can have any number of people show up on any
given day on the Web."
Testing in Internet time
Perhaps one of the most dramatic differences between testing on the
Web and testing in traditional or client/server environments is the rate
and pace of innovation and change. When IT projects ran 18 months or more,
project schedules sometimes allowed as much as six weeks for test planning.
Longer testing cycles and an ability to retest were built in to the process.
Not so on the Internet. "In the Web environment, especially in the dot.com
space, the pace is blistering," said Steve Caplow, director of marketing
and business development at RSW Software. "Some of these development projects
are literally about four to six weeks long from concept to deployment."
Indeed, many dot.coms may have only a week prior to deployment for a
complete testing operation. And a week after deployment, it is not unusual
for a new version to be required, which means having to modify or rebuild
all existing test scripts.
RSW built its tool suite specifically around Web technology for testing
browser-based applications. Mercury Interactive and Newton, Mass.-based
Segue Software, both long-time players in the traditional testing environment,
recently added product lines devoted to the Web architecture. Once the
appropriate tools are chosen for a specific project, a company is often
faced with the daunting task of knowing what, when and where to test.
In addition to testing tools, many an IT manager looks for outside help
in quickly developing and implementing a testing process. And there are
more and more service operations available to help dot.coms.
Inforonics, a professional services provider and Internet hosting firm
based in Littleton, Mass., has uncovered multiple testing challenges in
e-commerce, said CEO Bruce Buckland. "One of the biggest challenges is
the rapid rate of change required by e-business," said Buckland. He explained
that dot.coms have to put new features and capabilities into their sites
regularly in order to remain competitive. Such requirements are unprecedented
compared to traditional software development processes, and thus warrant
a different approach to testing and deployment.
For starters, Buckland points out that Internet applications must be
managed differently. A key notion is to implement changes that can be
reversed easily if a problem arises. IT organizations became very aware
of this requirement when the eBay site crashed. eBay lost full use of
its database for several hours, causing customers to abandon the site
"We have a methodology for doing this," said Buckland. "We engineer
changes to applications that are running in such a way that you can back
an application change out without backing the database change out, for
Buckland also emphasizes the need for both testing and monitoring tools.
"A broad and diverse set of tools can help in different situations," he
Guild.com Inc. is one dot.com company that learned this lesson. The
Madison, Wis., firm offers a large collection of original art items for
sale online. The company was formed as a venue for more than 1,200 artists
to present their work to a larger audience, and went online in April 1999.
Guild.com had originally outsourced the design, development and hosting
of its Web site to a local Internet Service Provider (ISP) that the company
would not identify. The plan also called for a local ad agency to design
the site's look and feel, and for the individual artists to be responsible
Nathan Harper, CIO at Guild.com, was recruited in 1999 to take the Web
site forward because of problems with the original plan. "We discovered
that design and development of the original site was business for the
ISP, but not critical to them; so a delay on their part didn't strike
them as very important," said Harper. "To us, delays meant lost business
To make matters worse, the original site utilized a peculiar database
- the coding was done in the Perl language and not very well documented
- and the hosting standard was not at the level required. At one point,
Guild.com asked about 50 of its employees to get on the site simultaneously
for a stress test. "We discovered a failure rate of about 75%," Harper
said. "This meant that our customers were experiencing the same thing;
that only one out of four people could complete a transaction."
That was the final straw. "We fired the ISP, declared independence,
brought in our own engineers and took off down the road," said Harper.
Since then, Guild.com has ported the database to Oracle, rewritten the
Perl code to Java, and moved the site to the Exodus Data Center in Illinois.
The company also purchased a new IBM RS6000 NonStop system to ensure high
Given the nature of its business, Guild.com decided it did not make
sense to bring in an entire QA department, so they opted to contract with
Farmington Hills, Mich.-based Compuware Corp. for services. The company
also licensed Compuware's QA line of testing tools.
Compuware's QA Load tool was used to stress test the site with 2,000
simultaneous users. Because the site is used only for e-commerce, the
most important testing was the ability of a customer to purchase items
from the site.
The project also called for redesigning the site's interface. For example,
would the graphics look the same to an AOL user, and how does the site
look on a Mac, on a PC or with Netscape? It was also important that the
site's search routine be tested because Guild.com carries a selection
of some 7,000 items.
The new site went online on November 17, 1999. "We anticipated very
well, so [we] had no negative surprises through the holidays," said Harper.
"There was no downtime, no slowdown in traffic; the site remained robust."
Developers used the QACenter tools for automated testing and performance,
while the Compuware service unit created a methodology to help with what
assumptions should be made during testing. Often, concerns included how
many users to test for, what types of transactions to monitor, and what
should be tested and how.
Guild.com continues to use the tools on an ongoing basis. "When we're
ready to bring up our next major release we can probably bring the Compuware
team back," said Harper. "Having your own engi- neers testing their own
work just doesn't cut it for the Internet."
Site operation is key
The SmarterKids.com online educational toy store has been operating
since November 1998. There are approximately 4,000 items for sale on the
site, including software, books, games, puzzles and toys.
"We have all the usual e-commerce challenges, including customer acquisition,
inventory, fulfillment and customer service," noted Rich Secor, vice president
and CIO at the Needham, Mass.-based concern. " If the site isn't available,
all those other things don't really matter," he added.
The site uses the Exodus Internet Data Center for hosting. Exodus provides
the real estate, high-speed Internet connection and security, Secor said.
SmarterKids.com's staff of Internet engineers designed and built the site
and is responsible for administering its own server.
Testing tools were brought in very early in the game, but it was clear
in 1998 that the online company was not ready for the testing tools and
that the tools were not ready for the site. SmarterKids.com went through
its first holiday season using internally written testing tools combined
with some Microsoft testing technologies. In early 1999, the firm evaluated
and finally chose the e-Test suite from RSW Software.
To gear up for the 1999 holiday season, SmarterKids.com used RSW tools
for functional testing of its site, to verify scalability, and to monitor
the site and ensure performance. SmarterKids.com saw an enormous increase
in its loads over the holiday, and the architecture supported the demand.
Officials credit regular testing and a formula to calculate the sizing
of the Web site based on projected sales.
eCampus.com is an online college textbook store in Lexington, Ky., that
went from concept to deployment in six months. Ted Willis, QA director,
and Tom Wright, lead tester at eCampus.com, turned to the Rational Suite
of tools from Rational Software, Cupertino, Calif. The Rational product
bundles configuration management, defect tracking and testing tools.
The site's busiest times are in the fall and in January between semesters.
Load testing provided a gauge of how many users the site could handle,
and allowed eCampus.com to spot and resolve problems before its last rush.
According to Willis, problems typically can stem from a high number of
users executing time-intensive processes such as searching the database.
Another difficult area for e-commerce firms, Willis said, is figuring
out how many people will access a site in a specific time period.
Testing was done prior to going online, Willis said. Now load testing
is done with all major code changes to the application. In production,
the company is writing scripts that go out and check the site every 15
minutes. If a problem is detected, operations people can fix it. "It's
important to find a problem before it happens; you want to go more in-depth
than just checking the site to see if it's alive," Willis said.
Changes can also be a force of contention for QA - something this industry
has never seen before - because the Internet is dynamic. "If we versioned
our code every time a change went up, we'd probably be in triple digits
by now," said Willis.
Jewelry.com, El Segundo, Calif., went online in November 1999. The company
uses CandleNet from Candle Corp., Los Angeles, to test application performance
and availability. CandleNet monitors customer response times by pinging
the site at regular intervals and providing reports that alert Jewelry.com
to any potential downtime and the need for more hardware or memory. The
company did not use testing tools, which proved to be a problem in the
"We thought we were going to have one of two problems with our site,"
said Paul Rajewski, chairman and CTO at Jewelry.com. "One, that we wouldn't
have enough traffic, or two, that we wouldn't have planned for enough
The latter turned out to be the case. Despite completing complex internal
calculations, traffic was triple what the site was built for as the company
began advertising campaigns. The accompanying problems left Jewelry.com
scrambling for new gear that could support very high numbers of users.
The online jewelry store built its system to be fault tolerant, using
dual systems along the way to prevent hardware outages. The front end
was redesigned using Sun's NetraSystems to handle increased loads expected
for the Valentine's Day rush. "Our objective there is that even if one
or two servers go down, it won't affect things because traffic will be
automatically redirected," said Rajewski.Rajewski said the firm did standard
testing, such as trace routes and pings, but conceded that testing tools
were not used. He now expresses interest in these types of tools. The
site has benefited from CandleNet, which confirmed suspicions that the
site was too slow.
The presence of online sites has grown significantly. Many dot.com firms,
hoping for increased traffic, even risked 50% of their projected revenue
this year to advertise during the Super Bowl. With an eye to the future,
most agree that testing and planning for site growth are required on an
Inforonics' Buckland stresses the importance of keeping an online company's
operations and systems people involved in the decision to go out and spend
millions on advertising. Be warned: If your system is not tested and ready,
then your e-commerce Web site can die a quick death.