News

Big Data Gets Big Change with Hadoop 2 Release

• By David Ramel
• October 17, 2013

The Apache Software Foundation yesterday announced the release of Apache Hadoop 2, a major upgrade of the open source cloud computing platform nearly synonymous with Big Data.

After more than four years of development, Hadoop 2 officially introduces an alternative to MapReduce: Apache YARN, also known as Yet Another Resource Negotiator or MapReduce 2. The MapReduce job batch processing framework is a core component of Hadoop and has been the subject of much criticism. The long-awaited YARN technology has been available in several independent vendor Hadoop distributions.

With YARN, much of the functionality of MapReduce has been separated into different processes so jobs and applications can run at the same time, with more user interaction in a more accessible model.

YARN "sits on top of HDFS [Hadoop Distributed File System] and serves as a large-scale, distributed operating system for big data applications, enabling multiple applications to run simultaneously for more efficient support of data throughout its entire lifecycle," Apache said in a blog announcing the new release. MapReduce isn't going away, however, and can still be used in Hadoop 2, which has binary compatibility with Hadoop 1.x applications using MapReduce.

Hadoop 2 also introduces support for Microsoft Windows, straying far from its Java-based, open-source roots as a batch processor. Microsoft had previously jumped on the Big Data bandwagon by offering HDInsight, its Apache-compatible Hadoop distribution for running on Windows.

The new release also features the addition of high availability, data snapshots and federation for HDFS. Federation increases scale by supporting mutliple namespaces, or Namenodes, which can run independently without coordination.

"Today, with the announcement of Hadoop 2 and YARN, we've taken another step," said Aaron Myers, member of the Apache Hadoop Project Management Committee and engineer at Cloudera. "Beyond the basic multitenancy customers have enjoyed for the past year, enabling them to mix batch, interactive and real-time workloads, they now have the ability to do so from within a stable foundational part of the Hadoop ecosystem. It's a testament to the community's work that now every distribution of Apache Hadoop will enjoy these benefits, ensuring that customers can deliver the applications they need, on a single Hadoop platform."

Users can now download the new release here.