Columns

Integration: This decade's theme

• By John D. Williams
• May 1, 2001

The integration issue is at the heart of the way companies choose to do business, and it is a defining characteristic of e-business. This is why the deployment of technologies such as Enterprise Application Integration (EAI) is really a business issue and not a technology issue. The forces driving business integration are rapidly changing markets, new business opportunities and customer expectations. These forces of change are working their way into the IT organization, driving budgets and projects. An aspect of these forces at work in the growth of EAI spending. In 1999, IT organizations spent $500 million on EAI tools. In 2000, they spent $900 million. Analysts predict that by 2005, companies will spend $7.3 billion on EAI.

These forces are also at work in our approaches to development and the development challenges of e-business projects. In March, ComponentSource and WebGain announced a partnership, in which WebGain is going to provide access to ComponentSource's component marketplace through its development environment. The partnership's purpose is to make it easier for developers to find the components they need to develop systems quickly. So, not only do companies see the need for integration, but vendors recognize the need for integration among themselves to deliver the power and capability that development organizations demand. We also see technologies integrating, such as the work to integrate XML and SQL that Merant and others are doing.

The types of projects that IT organizations are asked to tackle also reflect the need for integration. One area where this is easy to see is Customer Relationship Management (CRM). Companies are finding that managing customer relationships is more complicated than sticking a bit of personalization software on their Web site.

Within any organization, there is a lot of different information about each customer stored in many different systems. The development challenge is to break down the barriers between the silos that contain these individual systems. This is a daunting task and is not as simple as collecting Web information. Some have taken the data warehouse approach and tried to centralize all their customer data, but this has often been costly and difficult to implement. Others leave the data where it is and look for new ways to access the information.

This challenge of overcoming barriers, integrating information and delivering it to customers, vendors and other interested parties is driving the need for EAI. But what is EAI? Some think it's synonymous with messaging. Others claim it's remote access to data or RPC mechanisms. If you think of EAI only in these terms, you will unnecessarily restrict your view of potential solutions.

Ovum defined EAI this way: "EAI solutions are software products that completely or partially automate the process of enabling custom-built and/or packaged business applications to exchange business-level information in formats and contexts that each understands." Cherry Tree & Co. added, "It is essential to keep in mind that it is the chaining together of discrete transactions, in the form of a business process, from one application to the next that constitutes EAI." These definitions give us a starting point to think about EAI, but I think it is helpful to use a framework to understand different needs in EAI and how different tools meet those needs.

Imagine a four-layer framework describing EAI capabilities. The lowest layer is the Transportation layer, which has five communication models describing how one system communicates with another. These models, as GartnerGroup defines them, are Conversational; Request/Reply; Message Passing; Message Queuing; and Publish and Subscribe. The next layer up is Data Transformation. This layer describes the mechanisms for taking information from one database and transforming it before putting it into another database. The third layer is Business Rules. The Business Rules layer describes the method of taking select information from one system and transforming it into use by others. This may be a one-to-one transformation or a one-to-many transformation. The top layer is the Business Process layer, which coordinates the flow of information throughout a complete business process. It describes the workflow and transformation of information across multiple systems. In our framework, we also see that meta data lies across all these layers. As we move from the lower levels of our framework to the top, we typically see the business value of integration increase.

Most middleware tools, such as those for messaging and data warehousing, provide capabilities for the lowest two layers: Transportation and Data Transformation. Most EAI-specific tools are focused on the Business Rules layer, while some venture into the Business Process layer. EAI tools also often integrate with or provide support for tools working in the lower layers.

I've mentioned that meta data plays across all layers, but what do I mean by meta data? At its simplest, meta data is information about data. Typically, this is thought of as data types or table structure information. However, in the world of EAI, it is evolving into much more. Today, meta data includes such things as data transformation conditions and rules, or source and target mapping. Meta data exists across all types of systems and documents. It is often embedded in applications or structures and not captured explicitly. Organizations are only now beginning to realize its value.

One key to understanding meta data in integration is to use models to capture information about what is exchanged between systems. Models document the 'what' and 'why.' They provide the semantics that allow systems to work together. Meta data can also help you shape how to exchange information and provide a common mechanism for all systems. This exchange is not just data transfer, but an understanding about the data being transferred as well. This approach of using meta data for exchange supports dynamic integration. Meta data can also shape the documentation of the model.

XML has sparked much interest in this area of meta data exchange. It provides a mechanism for the dynamic interchange of meaningful information. In particular, there are components of XML that have tremendous value in the support of system integration. The most useful components are DTDs, XML Schema (and related variations), XSLT and XMI. DTDs and XML Schema define the structure of a document or information interchange. DTDs are the standard today. They do not use XML syntax and have some important limitations. For example, DTDs do not support the automatic validation of values. On the other hand, XML Schema does support the automatic validation of values. It also has the ability to define recurring blocks of elements or attributes once. Unfortunately, it is not yet a standard, though that should change soon. There are other non-standard alternatives available. XSLT allows you to transform one document type into another. XMI is the XML Meta data Interchange format from the Object Management Group (OMG). XMI is currently a proposal for the open interchange of application components and assets.

But do we really have all the tools we need to develop robust e-business systems? I don't think so. I've mentioned that EAI breaks down the stovepipes when you integrate systems. This can have serious implications. Higher levels of integration can lead to higher levels of unintended interaction. For example, the Chernobyl meltdown occurred during a test of enhanced safety procedures. Workplace automation has led to an epidemic rise in Carpal Tunnel Syndrome. These were not intended consequences. We need to learn to think about systems as systems. This systems approach is going to be a required thinking tool if we are to succeed in building integrated enterprises. As Stafford Beer, renowned author, professor, researcher and management sciences consultant, said, "Acceptable ideas are competent no more and competent ideas are not yet acceptable." We have much to learn in this area.

So what should we keep in mind in this decade of integration? Use a framework to understand the business value of integration projects and proposed solutions. Adopt a robust and flexible architecture. Remember that silos are bad things in the e-business world. Be sure to pick the right tools for the right level of integration. One tool is not the answer to every need. Use wisdom to create the best value for your business. Integration of systems is a necessity in today's world. And last but not least, do it in a way that adds value to your company.