Illusory connectivity: Web sites to databases -- ADTmag

Illusory connectivity: Web sites to databases

By Alan R. Earls
November 1, 2004

Ellis Admire, director of emerging technologies at Direct Holdings, parent company of catalog fashion retailer Lillian Vernon, found himself in a tough spot about four years ago when it became clear the company needed to jump into Web retailing in a big way. His options were to create a freestanding site with its own processes or try to unify a Web site with “50-plus years of business processes,” Admire says.

Although the former choice would have been easier, Admire says he realized it would be a nightmare in the long term. “It was critical to us to be able to sustain a consistent shopping experience between the Web and our call center.”

When it comes to tying Web sites to databases for Web services, there are many options, and that’s the problem. Do you roll your own or use an off-the-shelf solution? If it’s a DIY job, what tools (and there are many) should you use?

“It’s an enormous area in that you have a bazillion different ways of going about it,” says Jonathan Eunice, president, principal analyst and IT adviser at Illuminata Inc., Nashua, N.H. For example, says Eunice, you can start with template engines like ASP or JSP directly connected and explicitly programmed to a back-end DBMS. Alternatively, you can drive XML out of a DBMS (or other data source), and then use XSLT or XPathScript to transform it dynamically into HTML. Or you can write Web server code that communicates not with a pure DBMS, but rather with a Web service that can be a DBMS, but could also be an application (such as an ERP or CRM system) that owns enterprise data sets.

“There are even remote user interface techniques like Canoo or Macromedia’s Shockwave, and Flash cousins that can create rich [GUI] apps through the Web,” he says. And that still does not cover all the available options, he warns.

Time for templates
From a practical perspective, template languages “are probably the easiest way to go because there is not a vast amount of programming required,” Eunice says. Candidates include Active Server Pages, JavaServer Pages (JSP) and the Typed Assembly Language in the content-management world.

“To some degree you might consider XML stylesheets, too,” he says. The advantage is that templates provide a place for data and a place for code. “There are about 25 of these template languages and which one you choose depends on whether you are a Java or .NET shop or perhaps using PERL,” Eunice says. But even if you use a template language, it still isn’t a done deal because choices vary in simplicity and performance. “Some, like JSP are higher performance, can handle high volume, and are the most sophisticated in terms of the complexity of interaction they can handle,” he notes.

However, he warns, “when you really want to do a complex B2B scenario or anything that has multiple back-end applications, you start to get into the sphere of programming regardless of whether it is accomplished with Extensible Stylesheet Language Transformations, Java or C#,” Eunice says.

“It isn’t like starting from scratch because there are tools in Java to parse XML or to crack open Web services without having to decode the envelope,” adds Eunice. “The people who advocate for Web services try hard to minimize the need for programming, but that’s just putting lipstick on a pig.”

Adapters and other third-party products can help, Eunice says. “The latest database management systems, whether they are from SAP or Oracle, tend to have some kind of Web services adapters at least as an option, or a native ability to speak either Web services or XML,” he says. That built-in capability means you might not need to invest in a third-party adapter. However, he adds, a lot of application providers have put in their own wrinkles.

“XML is supposed to be a common language but it isn’t that simple,” he says. For instance, says Eunice, the way Oracle represents data is different from the way other vendors represent data. “They have different models of how to represent data in their HR package and they think about inventory differently,” he notes.

Admire at Direct Holdings says the ultimate success of his company’s project was in large part attributable to the adoption of two third-party products: Jacada and DataDirect. Admire says the Web site itself “has almost no business logic” and instead interfaces to the back office with the help of Jacada, which translates what happens on the Web site into something that emulates a call-center session. Jacada can support 80 connections to the AS/400, and each of those connections can spawn up to 10 “children” yielding a maximum of 800 Web sessions at a time.

“Last Christmas, we ran 30,000 orders and 300,000 page views a day without any problem,” he says. Indeed, the Web site now accounts for 40% of sales.

Jacada was also helped by DataDirect, which, Admire says, addresses one of the major misunderstandings about ODBC. “Many people think that just because your database is ODBC-compliant you can sit anywhere in the world and make a call.” In fact, he says, you need to set up a connectivity path and expedite the connections -- tasks that can consume programming time to build and maintain. “By using DataDirect, we eliminated the need to do all that coding,” says Admire.

Get the business processes right
St. Louis-based Upbeat Inc. positions itself as a leader in marketing unique indoor and outdoor products for business and government properties of all types. But the company needed to dramatically improve its Web functionality to keep up with customer expectations, according to Carla Russo, VP of materials management and MIS. Russo says her organization successfully met the challenge of linking legacy information from a package called Open Systems, written in BBX (Business Basic), to a new customer-facing Web site.

Completing the picture, Upbeat added Microsoft Commerce Server (which had a built-in BizTalk adapter). Although that’s the bare bones of the story, Russo says what has developed since is a greater understanding of what it means to be Web-centric. “It’s one thing to say we should attach our Web site to the back end, but it is another to actually live with the consequence of having orders flow through unattended,” she says.

In point of fact, Russo says she and her team scrutinized what they had done and realized they had to rethink the process. For example, she explains, the older manual system could only accommodate customer purchase order numbers of eight characters. When this limit was exceeded, the staff used manual workarounds. Once things were automated, “we had to figure out where to put that information so it would get transmitted to all the right people,” she says. Similarly, says Russo, it was crucial to develop a reasonably accurate, automated means of customer matching -- comparing data from sales initiated over the Web to existing customer records. “That was another piece we found we had to build,” she says.

Looking back on the process, Russo says, “there were almost two distinct pieces going on -- we had the pure logistics of moving the data and getting the disparate systems to talk to each other, and the other piece is the whole business process issue.” The project also ultimately required the creation of a program for master customer record management.

“We have a flat file structure in the old database and we ultimately want to build a data warehouse,” explains Russo. “This is a side benefit; we can now use the same adapter to help populate the warehouse and then we can move the enterprise toward more fact-based decision-making,” she adds.

For those planning to go down the same path, Russo recommends “really thinking about your business process to make sure you understand it clearly before you write a single line of code. Sketch it on a whiteboard and design the exact flow you would like and how it will populate.”

Buy a winning ticket
San Francisco-based StubHub is a Web site where users can buy or sell tickets to sporting events, concerts and the theater. All transactions on StubHub.com are 100% guaranteed, which necessitates a database that works seamlessly and rapidly through the Web site.

Shawn Kernes, vice president of technology, says the operation first went live in September 2000, running Oracle 8i on Solaris. Then in February of this year, Kernes migrated to Oracle 9i on Linux. “We have been very happy with that,” he adds. “In general, we aren’t doing anything that out of the ordinary, we are just doing it well,” he says.

Currently, the hardware infrastructure is all Intel-based IBM hardware. “Our front end is built on an IBM blade chassis with a number of blades running Red Hat Linux,” he says. The Web code is written mostly in ColdFusion with some Java and C, he adds. All of that is then linked to “a bunch of shared storage” as well as the Oracle database.

“We have redundancy at every level of the infrastructure with a cluster of servers and a cluster of database servers; our goal is to go for massive scalability,” he adds. For instance, he says, StubHub’s page view traffic has tripled in the past year and that trend is expected to continue. “Choosing the blade servers means we can scale horizontally,” he explains. To achieve high availability, StubHub connects to its ODBC-compliant data through a load balancer. “It provides us with rock-solid high availability,” he says. The load balancer is BIG-IP from F5 Networks. That lets StubHub decouple the database and the app. Another key to the real-time movement of data is the way StubHub pre-processes data to XML and then parses the XML structures, he says.

Counter the culture
Direct Holdings’ Admire says building robust back-end/Web site connections taught him an important lesson: two groups are needed for such projects.

“It’s two distinct cultures. Our AS/400 people are very orderly and careful, but the guys who build the Web site and have the spiked hair seem to believe they should just build it and let the customers tell them if it doesn’t work,” he says. The “traditional” AS/400 staff would never take that approach, says Admire. The groups also communicate in distinct languages: the traditionalists, mostly in their 40s or older, “speak” in Cobol, while the “new age” crew, mostly 20-somethings, “speak” in .NET or Java. Both groups ultimately need to find common ground to make a project successful, he says.

Jim Duggan, VP and research area director, application development at Gartner, agrees that many Web integration projects must span two different programming communities. “You have the people who know Flash and HTML, and who are good at the interactive part of the site -- for them, database access is something they have to figure out and their level of success is uneven,” he says.

On the other side of the equation are corporate developers who live in the world of C++, C# and Java. “They don’t do as much with the user interface, but they build data connections all the time,” he explains. “From being the first guy in the process, they have now shifted to the job of janitor who has to come in at the end and make it work.”

Drilling deeper, Duggan says he sees database programmers “struggling to adapt from a world with waterfall-based development where they were the first responders, to a world using object-oriented techniques where you start with the object and end with the persistent data.”

Please see the following related story: “Strike up the band ” by Alan Earls