News

Pivotal Adds Hadoop Distribution to In-Memory Big Data Parade

• By David Ramel
• March 17, 2014

Pivotal Software Inc. today announced the release of a new version of its enterprise Hadoop distribution based on the recently upgraded Apache Hadoop 2.2

Along with the new Pivotal HD 2.0, the company announced an integrated in-memory database, Pivotal GemFire XD, following a general industry trend.

"Pivotal GemFire XD bridges GemFire's proven in-memory intelligence and integrates it with Pivotal HD 2.0 and HAWQ," the company said. "Pivotal GemFire technology enables businesses to make prescriptive decisions in real-time, such as stock trading, fraud detection, intelligence for energy companies [and] routing for the telecom industries."

HAWQ is the company's parallel SQL query engine that natively reads and writes data on the Hadoop Distributed File System (HDFS) through a standards-compliant SQL interface. Together, Pivotal HD 2.0, HAWQ and GemFire XD form the basis of the company's Business Data Lake architecture.

Announced in beta last October, GemFire XD provides a Java interface that "will allow applications using Hadoop data to access hundreds of gigs of data in-memory in a single process without incurring the penalties usually associated with garbage collection issues in a [Java Virtual Machine]," Pivotal said.

By directly writing to the HDFS Big Data store, GemFire XD improves back-end analytics, Pivotal said, while providing a complex event framework in which to make real-time business decisions while capitalizing on the strengths of SQL, without "the penalties" of relational databases.

The company said GemFire XD allowed its customers to reduce risk analysis time from six hours to 20 minutes and improve end-user response time from three seconds to 50 ms.

Another new feature of Pivotal HD 2.0 is integration with GraphLab, described by the company as "an advanced set of algorithms for graph analytics that enables data scientists and analysts to leverage popular algorithms for insight, i.e. page rank, collaborative filtering and computer vision." In addition to GraphLab, Pivotal HD 2.0 provides integration and support of MADlib, which provides Big Data machine learning with SQL, and programming languages such as R, Python and Parquet.

Pivotal is an EMC Corp. spinoff founded about a year ago with VMware Inc. to focus on projects such as open Big Data applications in the cloud.