App Integration Reflects a New Ideal
- By Alan Radding
- July 1, 2005
NEW WAYS TO INTEGRATE
- Although experienced IT managers have seen it all before, this time around,
it looks like the EAI industry is getting closer to the app integration ideal.
- Traditional EAI vendors are scrambling to re-architect for service-oriented
- Even conventional middleware, EAI and message queuing vendors are adopting
the latest technology advances and incorporating them into their toolsets
App integration once meant painstaking point-to-point system integration with
code specially written to enable one app to tap the processing of another and
to share information between the two.
Today, an organization faced with the need to automate manual processes and
integrate systems has a host of options to choose from. XML, Web services, SOAand
ESB have emerged as the new buzzwords of EAI, along with integration technologies
such as business process management and enterprise content management. “These
are just the latest in a long string of technologies. Hopefully, they will make
the different platforms transparent,” says a 20-year IT veteran. Experienced
IT managers have seen it all before. This time around, however, it looks like
the EAI industry is getting closer to the integration ideal.
XML, for example, simplifies the integration of data from different systems.
Web services streamline the process by which one system can request and receive
processing and data from another system. Messaging service buses expedite the
flow of information through the organization and make it simple for systems
to grab the information they need while guaranteeing delivery. A variety of
middleware transforms and translates system calls and data as needed. BPM tools
and ECMsystems also facilitate integration, as do Web portals, which can simplify
integration and make it transparent to systems and users.
Even the conventional middleware, EAI and message queuing vendors, such as
Tibco, SeeBeyond, BEA Systems and webMethods are adopting the latest technology
advances and incorporating them into their toolsets. “The traditional
EAI vendors are scrambling to change their tools,” says Judith Hurwitz,
president of Hurwitz & Associates. “They are re-architecting for service-oriented
As a result, demand for application integration software continues to grow.
According to market researcher Gartner, worldwide app integration, middleware
and portal new license revenues was up 5.8 percent to $6.7 billion in 2004.
The leaders are IBM, with its popular MQ Series and WebSphere integration products,
and BEA. Notably, Gartner says, was Microsoft’s first-time appearance
on the leader board right behind Fujitsu and Oracle, rounding out the top five.
Standardize and modularize
Effective app integration, according to Hurwitz, comes down to standardize and
modularize. Standardize means to take advantage of industry agreement on interfaces,
protocols, formats and techniques. Modularize means breaking
up monolithic applications and data into smaller, manageable discrete chunks.
“This is about encapsulating old code as modules,” she explains.
Modules can be integrated on the fly through standardized interfaces to form
The advances in integration technologies fit right into Hurwitz’s standardize
and modularize approach. XML, for example, provides a standardized, machine
readable syntax for identifying a piece of data. Web services are small, discrete
chunks of code that perform a recognizable task. Service-oriented architectures
combine standardized interfaces and discrete services to support the deployment
of integrated composite applications, which are assembled from existing services.
The goal, Hurwitz continues, is to avoid having to programmatically integrate
apps. “If you have to write code to integrate the pieces, then every time you
need to make a change, you will have to write that code again,” she explains.
By modularizing applications into sets of reusable services, “when you need
to make a change to the business, you can just click the appropriate pieces
together,” Hurwitz says. The services take advantage of well-defined, standardized
interfaces and a standardized message bus to move the service requests and their
responses throughout the enterprise. (See related article, “What’s
Back to the future
In 1998, HVB Group, the New York banking and financial services unit of a large
European banking group, was squeezed between growing volumes of business and
a mandate against adding personnel. The bank’s ITmanagers adopted a strategy
to automate what had been primarily manual business processes and integrate
its systems. “It was our way to increase the volume of business we could
handle without increasing the number of bodies here,” says Chris Wren,
Just as HVB was deciding on the need for app and system integration, a number
of middleware tools that streamline the process hit the market. The company
checked out a variety of these new products, including message queuing from
IBM (MQ Series), middleware from Tibco and Neon Software, and various sorts
of brokers. One newcomer, SeeBeyond (previously called Software Technologies)
caught HVB’s interest. “Although it was written in a proprietary
language, even back then—in 1998—it had a GUI programming front
end,” Wren recalls. Enticed by the potential inherent in the GUI programming,
HVB decided to try it.
The initial effort was aimed at automating a simple transaction end-to-end.
This called for integrating with five or six apps, each with its own interfaces
and ways of handling data. At that early stage in its development, SeeBeyond
did not offered pre-built connectors, adapters or standardized APIs. “It
was crude,” Wren says. “There were no Web services. There was no
XML. With one application we even had to resort to screen scraping.” However,
they made it work, and the company has grown its integration and automation
with SeeBeyond ever since.
HVB did not make its task easy either. “We had an internal limitation,”
Wren says. “We could not change any core application, any legacy application.”
To overcome this challenge, the company developed an integration architecture
that would turn out to be a forerunner of what is now called an enterprise service
The architecture was built on the SeeBeyond integration engine, which managed
the electronic connectivity (messaging) and transformed data. In addition, HVB
built business logic and rules into the engine that automated and constantly
validated everything that passed through the integration system. The only manual
intervention involved looking at transactions that were kicked out as exceptions.
Today, HVB is deep into the third generation of SeeBeyond’s Integrated
Composite Application Network Suite. The company has progressed along with SeeBeyond
from a proprietary programming language to Java and is building a set of reusable
services. About the only thing HVB is not doing is using XML, because the company
already standardized on the banking industry’s Society for Worldwide Interbank
Financial Telecommunication message formats, syntax and structure.
Abebooks, a privately held company, has established itself as a leading used
books broker. The company collects information about available books from 13,000
booksellers, which it brokers as a virtual inventory of 75 million books (Abebooks
never takes physical possession of the actual books.). Anyone buying a used
book online through Amazon.com, Barnes & Noble or any other bookseller probably
crosses paths with Abebooks without realizing it. The company handles about
6 million transactions each day.
Abebooks’ success hinges on a massive and powerful app integration system.
Each day, it translates and transforms 3,000 different file formats as it collects
inventory data and generates sales transactions. Originally, the company tried
to do it all in a database, but that proved to be a mistake. “We tried
to use the database, but what we were really doing was message queuing,”
says Jayson Minard, CIO. “It really abused the database. It couldn’t
keep up in near real time.”
Abebooks never keeps a transaction. It only passes information between resellers
such as Amazon.com, which sells the used book to the customer, and the bookseller,
which has the book in inventory and handles the fulfillment. Between them, Abebooks,
passes information about the available books and their prices to the resellers
and passes back the details for fulfillment when a transaction is completed,
taking its cut along the way. What it needed was not a database, but a high-performance,
message-based integration system.
The company embarked on an initiative to build a messaging system using JMS
and open-source tools. The result worked on the messaging level, but Abebooks
needed more than straight message passing and queuing. It needed message integration,
transformation and orchestration. So it turned to Sonic Software’s ESB.
“We went to ESB so we could use Web services to orchestrate all this activity,”
The company developed a system that uses publish-and-subscribe messaging on
the front and back ends. In the middle, it uses a Sonic collaboration server
to push through the order processing. It also created a set of Web services
in the middle to handle format translation, while ESB orchestrated the various
Web services. To manage the whole process, Abebooks built a business activity
mon itoring console that watches the flow and generates alerts when problems
arise. Best of all, the company was able, for the most part, to avoid touching
its own core systems or the core systems at each end.
Sonic turned out to be the right choice for Abebooks, Minard says. “We
looked at Tibco but didn’t think they handled Web services smoothly. webMethods
was a little too expensive. Sonic was right in the middle from a cost standpoint,
and they were very responsive to what we were trying to do.” Sonic also
had the collaboration and orchestration pieces the others lacked.
Driven to integration
A few years ago, the governor of South Dakota remarked that when he asked a
question, such as how many people were on welfare, he often got back different
answers. It all depended on which system the answer from. The state’s
CIO at the time took that as a challenge: to find a way to get a consistent
answer across all the state’s systems. “That drove us to EAI,”
explains Denise Luckhurst, director of development for the South Dakota Bureau
of Information and Telecommunications.
The EAI effort began early in 2001. “We had a product that helped us
get at data in our legacy systems. We used it to try to cobble things together,
but we ended up with technical spaghetti,” Luckhurst says. The spaghetti
arose from the department’s efforts to integrate systems on a point-to-point
basis. “If we changed something in one system, we had to change all the
systems,” she continues.
The EAI issue became serious when the state had to comply with a growing number
of mandates, such as the Health Insurance Portability and Accountability Act
and later, Homeland Security directives. Beyond those mandates, every system
needed information from other systems. For example, the system that manages
custodial support uses information culled from, say, the driver’s license
and fishing license systems to track down deadbeat parents.
The bureau turned to SeeBeyond. “SeeBeyond gave us a guaranteed message
delivery bus between multiple different systems, including old legacy applications
and newer applications,” Luckhurst says. It began using SeeBeyond ICAN
with HIPAAin 2003. It added integration with a child welfare system a year later.
Today, it has EAI efforts under way on 15 systems. These efforts make available
information such as birth certificates and driver’s licenses to a variety
of other systems.
The bureau is building Web services with SOAP and XML, in conjunction with
ICAN, to simplify the process of accessing information front to back. XMLprovides
a standardized data syntax that helps the systems identify the information they
need. SeeBeyond provides the guaranteed message delivery.
The EAI effort had been limited, focusing on access to legacy data. To that
end, the bureau built a series of standard reusable interfaces to different
types of data structures. It has hired an enterprise integration architect.
With the architect and the EAI tools in place, Luckhurst hopes to speed up the
state’s app integration efforts.
Not every app integration effort has to revolve around conventional middleware
and messaging tools. Depending on the objectives, organizations can use nontraditional
tools, such as business process management and enterprise content management
tools. These tools often include messaging, transformation and workflow engines
beneath the covers that handle the bulk of the integration chores.
“One of the side benefits of BPM is that it can also encompass EAI-like
integration,” says Dave Kelly, president, Upside Research. However, the
integration tends to be process-oriented rather than transaction-oriented, he
says. “If companies have only basic integration needs, a BPM tool could
very well do it.”
When federal, state and local law enforcement agencies in Washington’s
Puget Sound area decided to share data, they faced a serious data integration
challenge. “Law enforcement agencies have well-defined processes that
are incompatible,” says Ted Sisk, director of criminal justices solutions
at Northrup Grumman, the systems integrator spearheading the agency-wide integration
effort. “The way they define data fields, the way they arrest, book and
charge someone is all different.”
Although none of the agencies was willing to change its internal systems, each
was willing to make it possible for data to be retrieved from its systems. “This
was really a records management problem and a business process problem,”
Sisk says. The solution turned out to be a data warehouse and a Web portal built
around Intalio’s Intalio|n3 BPM product.
Using Intalio, the developers built a data staging system, which Sisk refers
to as a front porch. After agreeing on the required data, agencies leave their
data on the front porch; the BPM tool collects it and links it to the correct
process. Using its internal analysis and linking capabilities, Intalio collects,
sorts, organizes, analyzes and integrates the data, even cutting through multiple
aliases some crooks employ using a technique called disambiguation. Police officers,
investigators and analysts access the data through what appears to them as a
Web page, but is actually a set of Web services.
The app consists of a series of modules based on pre-defined and agreed-upon
processes and data. The BPM engine tracks the data and the processes and performs
any necessary transformations, processing, integration and organization. “BPM
allowed us to exchange, document and integrate all of this different data,”
More on ADTmag.com
App servers’ performance
plays key role in SOA
By Kathleen Ohlson
Web services: Standards
breed like crazy...
By Alan Joch
The enterprise gets on
By George Lawton
ILLUSTRATION BY DAVID RIDLEY