News

Java Toolmaker Gives Maturing Big Data Market a Middleware App Tier

• By John K. Waters
• November 5, 2013

What's the next essential step in the evolution of Big Data? If you ask Tony Baer, principal analyst at Ovum, it's the emergence of an applications market. But just as Java-based apps needed app servers to get things rolling on the Internet, Big Data apps will need an operational application-oriented middleware platform for Hadoop, he says -- something like the latest release of Java toolmaker Continuuity's flagship Reactor app server.

"So far, Hadoop has developed in the data warehousing mold, where either homegrown programs or [business intelligence] tools access it," Baer told ADTmag in an e-mail. "But the natural evolution for Hadoop -- and Big Data sources in general -- is that they will also play a part with enterprise transactional applications as the underlying platform matures. It is inevitable that applications will be developed that run against Big Data, and as that occurs, it will be necessary to have an application layer that allows developers with Java and other languages to develop apps that run against it. Continuuity is pioneering development of a middleware application tier that will soon draw competition from incumbents."

Released last week, Reactor 2.0 is billed as the first scale-out application server for Apache Hadoop. The latest version is designed for production enterprise apps, with a list of features that includes MapReduce scheduling, Resource Isolation, High Availability, and full support for REST APIs. Reactor 2.0 has been re-architected, the company says, to enable regular old Java developers with no Big Data specialization to start building applications on Apache Hadoop and HBase.

"If you ask developers in a lot of the companies that have started using Hadoop, they'll tell you that the experience is just awful," said Continuuity CEO's Jonathan Gray. "Think about it this way: The Linux kernel wasn't intended to be written against by developers. That's why higher level languages, application servers, and services like Heroku exist, to enable the application developer to focus on the app, and not have to worry about the low-level infrastructure. The reality of Hadoop is that it's a kernel. It's very low level, it has very few primitives, and the API is inherently tied to the architecture of the system. That's part of Hadoop's power, of course, but it puts a lot of the onus on the developer to, essentially, write their own middleware stack."

According to the company, Reactor 2.0 is designed to make it possible for developers to run an end-to-end application on a single platform with an integrated Web server. A new set of APIs in this release takes advantage of YARN, the next-gen Hadoop data processing framework (sometimes called MapReduce 2.0) to allow devs to run data-rich apps with support for batch processing and real-time streaming capabilities in a single Hadoop cluster. The new app server also uses the Kerberos authentication protocol.

Not surprisingly, Gray, who co-founded his Palo Alto-based company with Todd Papaioannou and Nitin Motgi in 2011, agrees with Baer about the evolution of Big Data.

"We believe that the future of Big Data is applications," he said. "And we always set out to improve the platform with the developer in mind."

Continuuity may be the first to hit what is likely to become a very sweet spot for developers, according to Mike Dauber, principal at Battery Ventures. He cites IDC's estimates that the Big Data market will be worth $23.8 billion by 2016, but enterprises will be scrambling to find the talent they need to exploit that market.

"Even large enterprises with great resources might only have five people who can run MapReduce, while that same organization may have 20,000 very skilled Java developers," he said in a statement. "Continuuity is filling a real need by enabling those skilled Java developers to step-in and fill the Big Data talent gap."