Rethinking data delivery -- ADTmag

Rethinking data delivery

By Sally J Cusack
July 23, 2001

Just when information systems (I/S) architects and application designers think they may be successfully conquering the world of distributed data and client/server concepts, another computing distribution paradigm enters the world. In the case of push/pull technology, and its more sophisticated cousin publish/subscribe, the low-cost communications of the Internet are driving companies to more carefully rethink their software strategies to truly leverage technology for the business. However, research shows that users are still confused between these two approaches to software distribution and information delivery.

There is little wonder for this. If there is one thing that has changed the I/S world significantly in the past two years, it is the availability of low-cost, widespread connectivity. So which is the right choice? Should I/S push and pull, or publish and subscribe?

Pushing the information envelope

Actually a form of broadcasting, push, or pull (depending on whether you are looking at it from the client or the server side of the equation) technology has probably received the lion's share of media hype and attention.

As the Internet and World-Wide Web usage has exploded in recent years, so has the public's demand for information delivered to the desktop in a timely and transparent fashion. Users are excited to know that by purchasing a certain software package or service, information would automatically arrive at their desktops either at specified intervals or in a continuous stream. News updates and services, weather reports, stock market fluctuations and other events can flow to almost any desktop machine anywhere in the world via the Internet and push technology. Start-ups like Marimba Inc., Palo Alto, Calif., and PointCast Inc., Cupertino, Calif., are able to market and sell the concept of push software with very little trouble, and as such, add-ons and variations to this technology are rapidly forthcoming.

Marimba, for example, has defined the market in the public's mind for push technology; the company views it as a natural extension to the client/server environment. Founded in 1996 by members of Mountain View, Calif.-based Sun Microsystems Inc.'s original Java development team, the company has captured industry imagination in the past 12 months as both its products and its executives have taken the spotlight at industry trade shows and events focusing on ways to maximize Internet and Intranet channeled software. Marimba currently offers Castanet, a technology designed to distribute software applications and create channels across the Internet/Intranet, and Bongo, a visual interface builder designed specifically for Java applications.

However, push technology is just beginning to scratch the surface of its full potential to I/S installations. Companies such as Prolifics, a division of Jyaac Inc., both based in New York City, are taking the push concept into the next logical realm of software development -- actual push transactions optimized for the three-tier, enterprise, Java/Web environment. According to Dan Kara, director of Analyst Services at Software Productivity Group, Natick, Mass., this type of technology is going to be central to the success of many software developers and vendors in the near future. The ability to add fault tolerant capabilities and maintain state across Internet-enabled applications will help to redefine the market for application development tools and their uses. To date, push technology on the Internet has been confined primarily to synchronous, static delivery methods. For technology users to truly leverage Internet-based applications, issues such as consistent data views across multiple Web clients and the constant state of the database must be addressed.

"What is really required is a multithreaded client, and asynchronous peer-to-peer communication must exist between the Web server and browser," said Kara. "Unless you have this type of functionality, you really can't do Internet transaction processing. You need a common view of data across all clients interested in a given piece of data. Depending on the implementation, if the view is inconsistent, the information could be worthless."

Optimizing push

Prolifics is currently a leader in this area. With the release of Prolifics 3, the company has expanded its three-tier application development environment to provide an infrastructure upon which Java applications can reliably perform transactional business-to-business systems processing. In fact, Sun's JDK1.1 is the basis for Prolifics 3. Using the Prolifics 3 push development tools, applications can be enabled with full push transactional capabilities. This includes runtime load balancing, reliable queuing, two-phase commit, data dependent request routing and server replication such as that found in full-scale enterprise middleware like Tuxedo from BEA Systems Inc., Sunnyvale, Calif. Perhaps most importantly, the Prolifics 3 push technology can offer users transactional Java. Java applications typically communicate with databases through Java database connectivity (JDBC). However, it must be remembered that JDBC, like its predecessor open database connectivity (ODBC), relies on the lowest common denominator approach to accessing databases. This results in JDBC sacrificing the performance derived from native database features, things like stored procedures and triggers.

As an alternative to the JDBC method, Prolifics's transactional Java routes database interactions through the transactional object model (TOM), which contains native database drivers for multiple databases. Based on the architecture, TOM can seamlessly route database requests through transactional middleware, thereby removing the limits imposed by one-to-one database connection to each user. This provides the scalability necessary for enterprise transactional applications. Further, modification of these transactional Java applications is simplified by encapsulating database access code into a single class hierarchy. This allows database developers to change SQL and other transaction logic in a single location and introduce changes globally via multilevel inheritance. This capability reduces redundant code and encourages code reuse, a major benefit of any object-oriented (OO) development effort.

Expanding push

Another player in the push transactional arena is Passport Corp., Paramus, N.J. Passport introduced its IntRprise software in October 1996. The OO4GL, visual programming product is geared toward building multitier Internet/Intranet applications with Java clients. This uses Java as the windowing system to display a particular system across the Internet. This reduces bandwidth requirements, since there is no delay when entire HTML scripts are downloaded to accommodate single changes. Only the changed information is downloaded. Like Prolifics, Passport has expanded the simple push paradigm to allow for persistence in multitier transaction situations. In this case, IntRprise is designed to provide persistent session connection via Passport application servers, which in turn are directly and persistently connected to one or more database servers.

The product is designed for highly interactive, event-driven applications where messages must be passed among software modules that are asynchronous, synchronous, or both. The IntRprise development environment offers concurrent and direct access to all major and ODBC-compliant databases. Additionally, Passport also supports a degree of fault tolerance through the use of object persistence. Passport defines this as the ability to deliver high availability transaction processing (HATP) to enable checkpoint/restart of an application in the event of a system failure. This is an especially attractive feature for customers in the banking and finance industries, but it is also beginning to come into play in other segments of the market. This is reflected in Passport's customer list, which includes American Airlines, Equifax, PaineWebber, Corning and the New York Power Authority.

Publish/subscribe spells action

A little further up the computing foodchain from a technical standpoint, but less well-known, is the somewhat older and more traditional publish/subscribe data delivery method. Once confined almost entirely to Wall Street brokerage firms, the Internet is driving this concept into other industries and broadening the applicability of this technology to a variety of applications. This technology is more sophisticated than the push approach. In publish/subscribe, users subscribe to specific events, or subjects, which are published to a specific set of subscribers based on more detailed and predefined information requests. In fact, push has been characterized in some circles as simply an abstraction of publish/subscribe. In addition, while push is making inroads as an enterprise infrastructure for certain types of applications, publish/subscribe technology is still better suited to play in the mission-critical systems arena where fault tolerance, load balancing and scalability are more likely to be application design issues.

"Companies must now become active enterprises, acting on information and events," said Mark Bowles, chief technology officer at TIBCO Inc., Palo Alto, Calif., one of the pioneers of publish/subscribe technology. There is hardly a firm in the brokerage industry that does not have the TIB message bus product in house to drive their fast-paced publish/subscribe mechanisms. Bowles, an architect of the TIB product, sees opportunities for the publish/subscribe paradigm beyond Wall Street. "This is really about the distribution of events. Events are news, but not just in the Wall Street or NBC sense of the word," he said.

Enterprisewide usage

News is the notification that things have changed, and many of today's companies want to electronically respond to and act on data, not just receive it. An example of this can be seen at San Francisco-based Chevron International Oil Co. (CIOC), which currently uses the TIB platform as the basis for its Chevron Advanced Trading System (CATS). CATS is a suite of integrated applications designed to capture and display market data, and enter trade information into the system, which handles credit, contracts and accounting functions. Additionally, CATS is responsible for managing back office financial systems and interfacing with the corporate general ledger system.

The TIB platform allows distributed services and applications to be integrated and managed across an enterprise. The software allows applications to dynamically publish and subscribe to information while providing both location and application transparency. Applications can simply be placed on the bus for automatic availability to other applications on the network. In order to shield developers from network complexities, TIB has been architected to provide "subject-based addressing." This allows developers to generate and label objects for "publishing" via the TIB platform. Users can request this data by simply subscribing to the subject of their choice. As is the case with several high-end, enterprise-level, publish/subscribe-based systems, TIB also provides reliable message delivery and incorporates a reliable broadcast to deliver data and queuing mechanisms for guaranteed delivery.

According to Greg Lee, manager of Trading Information Systems at CIOC, before implementing the TIB publish/subscribe technology, the company used to spend a great deal of time with multiple people putting information into multiple systems and then reconciling the differences. Via the publish/subscribe mechanism, realtime information can be immediately accessed and shared as necessary on a global basis. By using Intelligent Multicasting and other patented communications protocols, the TIB platform can distribute information to thousands of networked workstations and can be implemented across different computer architectures. The system ties together CIOC's San Francisco headquarters with its main offices in Houston, London and Singapore.

Lee points out that crude oil and petroleum product trading takes place around the clock, crossing all time zones. Therefore, the ability to exchange and update information is crucial. By utilizing the TIB platform as a "highway" to facilitate information flow, Lee said that CIOC can better manage exposure in a number of areas, eliminate costs and subsequently increase productivity.

Processing, not broadcasting

Another example of enterprise-caliber publish/subscribe activity can be seen in BEA's Tuxedo system. BEA Systems plays in what it likes to term the "business event broker" arena. Edward Braginsky, vice president Corporate Architecture and Strategic Initiatives, partitions the publish/subscribe and push market into three separate spheres. First, there are the traditional players in the trading floor arena. This includes players such as TIBCO and Reuters Ltd., London. (Reuters, incidentally, acquired TIBCO and then Teknekron Software Systems Inc., in 1994.) These companies rely on things like IP multicast technology when moving information. The second, and currently more widely identifiable category according to Braginsky, includes the Internet-centric, push-oriented companies such as PointCast and Marimba. Then there is the emerging third category, the business event broker software model, which Tuxedo falls into. (Both the aforementioned Prolifics and Passport products utilize Tuxedo when applications call for full atomicity, consistency, isolation and durability [ACID]-level online transaction processing [OLTP].)

The Tuxedo EventBroker enables the processing of business events, as opposed to broadcasting bits of information (such as realtime stockbroker data), around a local-area network, said Braginsky. A subscriber can choose a method in which the data associated with the event is delivered. For example, the data can be handed to the subscribing application in realtime, or the data can be placed on a durable queue. It can even be passed onto a third-party service, if desired. Within EventBroker, a publisher may post a transactional event; EventBroker is responsible for managing the transaction coordination across the subscribers, thereby ensuring all-or-nothing processing of the data associated with the event. Importantly, Tuxedo events are available to Internet-based clients. A Java applet running inside a browser from Netscape Communications Corp., Mountain View, Calif., for instance, can subscribe to events posted by Tuxedo servers.

EventBroker is designed to provide a robust service for delivering event notifications, and the data associated with those events, to subscribers. The company describes the product as part of a "comprehensive enterprise middleware" platform. In reality, this translates into a "full-scale, distributed transaction processing environment." As such, this is something systems architects must consider seriously before deciding upon the Tuxedo system. From a publish/subscribe perspective, the product has some distinct advantages in a mission-critical systems setting. Since EventBroker is built on top of the Tuxedo fundamentals, users are guaranteed the industrial-strength operational characteristics common in the OLTP environment. Additionally, BEA is working to integrate its recently acquired BEAmessageQ system and BEA ObjectBroker within Tuxedo, which will give users in the enterprise development area a richer set of asynchronous messaging capabilities, as well as access to industry-leading object request broker technology.

The middleware angle

It should be pointed out that publish/subscribe communications, unlike simple push communications, are usually a component of a larger middleware cloud -- sometimes a bus topology or set of middleware services used to flow and/or integrate data between systems, applications and even different companies. So, while there are many variations on the publish/subscribe paradigm, there are also several middleware offerings that fall somewhere in between the Java-Internet-OO4GL environments such as Passport's IntRprise and the full-scale transactional systems such as BEA's Tuxedo. This middle area would include products like San Francisco-based Open Horizon Inc.'s Ambrosia and EventWay, and Mountain View, Calif.-based Talarian Corp.'s SmartSockets.

"If there is one thing that has changed the I/S world significantly in the past four years, it is connectivity," said Nicholas Zaldastani, president of Open Horizon. "Look at the opportunities created by the Internet -- we see them in oil derricks, laundromats, even Pepsi machines. These all become potential sources of information of business events in realtime, as they are happening. Collecting and assembling realtime usage data, inventory numbers and so on, via publish/subscribe methodologies can give companies a marketing edge in many situations."

The Open Horizon approach to this type of event-driven data collection is the Ambrosia and EventWay software products. The Ambrosia client consists of a handful of Java class libraries that can be embedded into an application. Using the Ambrosia publish/subscribe technology and guaranteed message delivery system as a basis, the EventWay product is an event management system (written entirely in Java) that can provide a connection to existing middleware architectures such as TIBCO's TIB, as well as provide links from Ambrosia to legacy systems.

SmartSockets, in comparison, is a middleware product that adds both fault tolerant and publish/subscribe messaging capabilities to I/S environments where reliability is crucial. Though not as heavyweight a software offering as that of TIB or Tuxedo, nonetheless it has proved to play well in traditional areas where publish/subscribe technology is found. An example of this can be seen at the Philadelphia Stock Exchange. According to Douglas M. Schafer, project manager for the Options Trading System at the Philadelphia Exchange, the organization is close to releasing its first application using this technology. The Options Trading System group has undertaken a restructuring initiative around its Sun/Unix systems and basic client/server design.

"We generate all software for the Options Trading Floor," said Schafer. "Orders are delivered from firms that are doing business with the Philly Stock Exchange, which are routed to traders at workstations on the floor. There are three types of transactions: quotes, trades and orders." What this really amounts to, according to Schafer, is a big order routing and message delivery system. This is one of the primary reasons the SmartSockets product was chosen. The product offers guaranteed message capability, which means that a message does not get removed from the system until confirmation is sent to the original sender. The other major selling points were the product's scalability and the consistency of the API.

Although Schafer and his staff looked at several products before choosing SmartSockets, he said, "We saw that we could develop rapid application development (RAD) applications quickly by utilizing the [SmartSockets] hierarchical namespace, subscribing to it, and then writing applications to massage the data." The SmartSockets software allows for load balancing and failover as well.

Everybody wins

One thing is certain, push technology does not have to compete with publish/subscribe technology. Depending on an organization's technical criteria and business needs, variations on both paradigms could be appropriate. Particularly in large organizations, there is no single solution. As demonstrated above, products and capabilities run the gamut from the simple to the sophisticated. It becomes a matter of determining your technical and business needs, and then selecting the correct (and in many cases, this means a combination of) software delivery platform solution.