In-Depth

App Integration Reflects a New Ideal

• By Alan Radding
• July 1, 2005

Talking Points
NEW WAYS TO INTEGRATE

• Although experienced IT managers have seen it all before, this time around,
  it looks like the EAI industry is getting closer to the app integration ideal.
• Traditional EAI vendors are scrambling to re-architect for service-oriented
  architectures.
• Even conventional middleware, EAI and message queuing vendors are adopting
  the latest technology advances and incorporating them into their toolsets

App integration once meant painstaking point-to-point system integration with code specially written to enable one app to tap the processing of another and to share information between the two.

Today, an organization faced with the need to automate manual processes and integrate systems has a host of options to choose from. XML, Web services, SOAand ESB have emerged as the new buzzwords of EAI, along with integration technologies such as business process management and enterprise content management. “These are just the latest in a long string of technologies. Hopefully, they will make the different platforms transparent,” says a 20-year IT veteran. Experienced IT managers have seen it all before. This time around, however, it looks like the EAI industry is getting closer to the integration ideal.

XML, for example, simplifies the integration of data from different systems. Web services streamline the process by which one system can request and receive processing and data from another system. Messaging service buses expedite the flow of information through the organization and make it simple for systems to grab the information they need while guaranteeing delivery. A variety of middleware transforms and translates system calls and data as needed. BPM tools and ECMsystems also facilitate integration, as do Web portals, which can simplify integration and make it transparent to systems and users.

Even the conventional middleware, EAI and message queuing vendors, such as Tibco, SeeBeyond, BEA Systems and webMethods are adopting the latest technology advances and incorporating them into their toolsets. “The traditional EAI vendors are scrambling to change their tools,” says Judith Hurwitz, president of Hurwitz & Associates. “They are re-architecting for service-oriented architectures.”

As a result, demand for application integration software continues to grow. According to market researcher Gartner, worldwide app integration, middleware and portal new license revenues was up 5.8 percent to $6.7 billion in 2004. The leaders are IBM, with its popular MQ Series and WebSphere integration products, and BEA. Notably, Gartner says, was Microsoft’s first-time appearance on the leader board right behind Fujitsu and Oracle, rounding out the top five.

Standardize and modularize
Effective app integration, according to Hurwitz, comes down to standardize and modularize. Standardize means to take advantage of industry agreement on interfaces, protocols, formats and techniques. Modularize means breaking up monolithic applications and data into smaller, manageable discrete chunks. “This is about encapsulating old code as modules,” she explains. Modules can be integrated on the fly through standardized interfaces to form composite apps.

The advances in integration technologies fit right into Hurwitz’s standardize and modularize approach. XML, for example, provides a standardized, machine readable syntax for identifying a piece of data. Web services are small, discrete chunks of code that perform a recognizable task. Service-oriented architectures combine standardized interfaces and discrete services to support the deployment of integrated composite applications, which are assembled from existing services.

The goal, Hurwitz continues, is to avoid having to programmatically integrate apps. “If you have to write code to integrate the pieces, then every time you need to make a change, you will have to write that code again,” she explains. By modularizing applications into sets of reusable services, “when you need to make a change to the business, you can just click the appropriate pieces together,” Hurwitz says. The services take advantage of well-defined, standardized interfaces and a standardized message bus to move the service requests and their responses throughout the enterprise. (See related article, “What’s the point?”)

Back to the future
In 1998, HVB Group, the New York banking and financial services unit of a large European banking group, was squeezed between growing volumes of business and a mandate against adding personnel. The bank’s ITmanagers adopted a strategy to automate what had been primarily manual business processes and integrate its systems. “It was our way to increase the volume of business we could handle without increasing the number of bodies here,” says Chris Wren, COO.

Just as HVB was deciding on the need for app and system integration, a number of middleware tools that streamline the process hit the market. The company checked out a variety of these new products, including message queuing from IBM (MQ Series), middleware from Tibco and Neon Software, and various sorts of brokers. One newcomer, SeeBeyond (previously called Software Technologies) caught HVB’s interest. “Although it was written in a proprietary language, even back then—in 1998—it had a GUI programming front end,” Wren recalls. Enticed by the potential inherent in the GUI programming, HVB decided to try it.

The initial effort was aimed at automating a simple transaction end-to-end. This called for integrating with five or six apps, each with its own interfaces and ways of handling data. At that early stage in its development, SeeBeyond did not offered pre-built connectors, adapters or standardized APIs. “It was crude,” Wren says. “There were no Web services. There was no XML. With one application we even had to resort to screen scraping.” However, they made it work, and the company has grown its integration and automation with SeeBeyond ever since.

HVB did not make its task easy either. “We had an internal limitation,” Wren says. “We could not change any core application, any legacy application.” To overcome this challenge, the company developed an integration architecture that would turn out to be a forerunner of what is now called an enterprise service bus.

The architecture was built on the SeeBeyond integration engine, which managed the electronic connectivity (messaging) and transformed data. In addition, HVB built business logic and rules into the engine that automated and constantly validated everything that passed through the integration system. The only manual intervention involved looking at transactions that were kicked out as exceptions.

Today, HVB is deep into the third generation of SeeBeyond’s Integrated Composite Application Network Suite. The company has progressed along with SeeBeyond from a proprietary programming language to Java and is building a set of reusable services. About the only thing HVB is not doing is using XML, because the company already standardized on the banking industry’s Society for Worldwide Interbank Financial Telecommunication message formats, syntax and structure.

Messaging-based integration
Abebooks, a privately held company, has established itself as a leading used books broker. The company collects information about available books from 13,000 booksellers, which it brokers as a virtual inventory of 75 million books (Abebooks never takes physical possession of the actual books.). Anyone buying a used book online through Amazon.com, Barnes & Noble or any other bookseller probably crosses paths with Abebooks without realizing it. The company handles about 6 million transactions each day.

Abebooks’ success hinges on a massive and powerful app integration system. Each day, it translates and transforms 3,000 different file formats as it collects inventory data and generates sales transactions. Originally, the company tried to do it all in a database, but that proved to be a mistake. “We tried to use the database, but what we were really doing was message queuing,” says Jayson Minard, CIO. “It really abused the database. It couldn’t keep up in near real time.”

Abebooks never keeps a transaction. It only passes information between resellers such as Amazon.com, which sells the used book to the customer, and the bookseller, which has the book in inventory and handles the fulfillment. Between them, Abebooks, passes information about the available books and their prices to the resellers and passes back the details for fulfillment when a transaction is completed, taking its cut along the way. What it needed was not a database, but a high-performance, message-based integration system.

The company embarked on an initiative to build a messaging system using JMS and open-source tools. The result worked on the messaging level, but Abebooks needed more than straight message passing and queuing. It needed message integration, transformation and orchestration. So it turned to Sonic Software’s ESB. “We went to ESB so we could use Web services to orchestrate all this activity,” Minard says.

The company developed a system that uses publish-and-subscribe messaging on the front and back ends. In the middle, it uses a Sonic collaboration server to push through the order processing. It also created a set of Web services in the middle to handle format translation, while ESB orchestrated the various Web services. To manage the whole process, Abebooks built a business activity mon itoring console that watches the flow and generates alerts when problems arise. Best of all, the company was able, for the most part, to avoid touching its own core systems or the core systems at each end.

Sonic turned out to be the right choice for Abebooks, Minard says. “We looked at Tibco but didn’t think they handled Web services smoothly. webMethods was a little too expensive. Sonic was right in the middle from a cost standpoint, and they were very responsive to what we were trying to do.” Sonic also had the collaboration and orchestration pieces the others lacked.

Driven to integration
A few years ago, the governor of South Dakota remarked that when he asked a question, such as how many people were on welfare, he often got back different answers. It all depended on which system the answer from. The state’s CIO at the time took that as a challenge: to find a way to get a consistent answer across all the state’s systems. “That drove us to EAI,” explains Denise Luckhurst, director of development for the South Dakota Bureau of Information and Telecommunications.

The EAI effort began early in 2001. “We had a product that helped us get at data in our legacy systems. We used it to try to cobble things together, but we ended up with technical spaghetti,” Luckhurst says. The spaghetti arose from the department’s efforts to integrate systems on a point-to-point basis. “If we changed something in one system, we had to change all the systems,” she continues.

The EAI issue became serious when the state had to comply with a growing number of mandates, such as the Health Insurance Portability and Accountability Act and later, Homeland Security directives. Beyond those mandates, every system needed information from other systems. For example, the system that manages custodial support uses information culled from, say, the driver’s license and fishing license systems to track down deadbeat parents.

The bureau turned to SeeBeyond. “SeeBeyond gave us a guaranteed message delivery bus between multiple different systems, including old legacy applications and newer applications,” Luckhurst says. It began using SeeBeyond ICAN with HIPAAin 2003. It added integration with a child welfare system a year later. Today, it has EAI efforts under way on 15 systems. These efforts make available information such as birth certificates and driver’s licenses to a variety of other systems.

The bureau is building Web services with SOAP and XML, in conjunction with ICAN, to simplify the process of accessing information front to back. XMLprovides a standardized data syntax that helps the systems identify the information they need. SeeBeyond provides the guaranteed message delivery.

The EAI effort had been limited, focusing on access to legacy data. To that end, the bureau built a series of standard reusable interfaces to different types of data structures. It has hired an enterprise integration architect. With the architect and the EAI tools in place, Luckhurst hopes to speed up the state’s app integration efforts.

Non-traditional integration
Not every app integration effort has to revolve around conventional middleware and messaging tools. Depending on the objectives, organizations can use nontraditional tools, such as business process management and enterprise content management tools. These tools often include messaging, transformation and workflow engines beneath the covers that handle the bulk of the integration chores.

“One of the side benefits of BPM is that it can also encompass EAI-like integration,” says Dave Kelly, president, Upside Research. However, the integration tends to be process-oriented rather than transaction-oriented, he says. “If companies have only basic integration needs, a BPM tool could very well do it.”

When federal, state and local law enforcement agencies in Washington’s Puget Sound area decided to share data, they faced a serious data integration challenge. “Law enforcement agencies have well-defined processes that are incompatible,” says Ted Sisk, director of criminal justices solutions at Northrup Grumman, the systems integrator spearheading the agency-wide integration effort. “The way they define data fields, the way they arrest, book and charge someone is all different.”

Although none of the agencies was willing to change its internal systems, each was willing to make it possible for data to be retrieved from its systems. “This was really a records management problem and a business process problem,” Sisk says. The solution turned out to be a data warehouse and a Web portal built around Intalio’s Intalio|n3 BPM product.

Using Intalio, the developers built a data staging system, which Sisk refers to as a front porch. After agreeing on the required data, agencies leave their data on the front porch; the BPM tool collects it and links it to the correct process. Using its internal analysis and linking capabilities, Intalio collects, sorts, organizes, analyzes and integrates the data, even cutting through multiple aliases some crooks employ using a technique called disambiguation. Police officers, investigators and analysts access the data through what appears to them as a Web page, but is actually a set of Web services.

The app consists of a series of modules based on pre-defined and agreed-upon processes and data. The BPM engine tracks the data and the processes and performs any necessary transformations, processing, integration and organization. “BPM allowed us to exchange, document and integrate all of this different data,” Sisk says.

App servers’ performance plays key role in SOA
By Kathleen Ohlson

Web services: Standards breed like crazy...
By Alan Joch

The enterprise gets on the bus
By George Lawton

ILLUSTRATION BY DAVID RIDLEY