News

Big Data Rivals Team Up On Hadoop Management

• By David Ramel
• July 31, 2014

Hortonworks Inc. and Pivotal Software Inc. this week announced they were teaming up to further development work on the open source Ambari project for managing Hadoop clusters.

Ambari provides a Web UI and RESTful APIs that developers use to provision, manage and monitor Hadoop clusters. Both Ambari and Hadoop are open source projects under the auspices of the Apache Software Foundation (ASF).

Hortonworks, formed in 2011 by engineers from the original Yahoo Hadoop team, is a major contributor to open source projects in the Hadoop ecosystem and offers a packaged enterprise product, the Hortonworks Data Platform. Often described as one of the "big three" Hadoop distributors along with Cloudera Inc. and MapR Technologies Inc., Hortonworks has contributed the vast bulk of Ambari development work.

Pivotal, spun out last year by parent companies EMC Corp. and VMware Inc., provides a Big Data Suite along with cloud computing and Agile development products. It has also been a contributor to Apache open source projects and has developed its own complementary modules, including the Pivotal Command Center, a Web-based administration console that provides similar functionality to Ambari, targeting the company's Big Data-related products.

Despite the similar functionality, Pivotal said it will expand its open source efforts by assigning more engineers to the Ambari project, working with Hortonworks and other contributors. "At the same time, we will continue to deliver on our commitments to existing customers and work closely with them to benefit from this collaboration," company exec Jamie Buckley said in a blog post Monday, the same day Hortonworks announced the latest version of Ambari. Buckley said the company's increased investment in the open source project will benefit "our Pivotal HD and Big Data Suite customers who favor Ambari."

Hortonworks exec Shaun Connolly, meanwhile, welcomed Pivotal to the Ambari development effort, which he said will strengthen what he called one of the five basic pillars of enterprise Hadoop: operations. The others are data access, data management, security and governance.

"I've stated many times that you don't make a platform like enterprise Hadoop easy to use and enterprise-grade by going it alone," Connolly said. "You do it by working with the broader ecosystem."

Developer Hari Sekhon, who works on Nagios open source monitoring tools, welcomed the collaboration. "Excellent news," Sekhon said in a comment to Connolloy's post. "Standardization on ASF technology is the future. I've released API monitoring Nagios plug-ins for every major Hadoop vendor’s management console except Pivotal Command Center (which didn't have a suitable REST API). But now the shift in modern times to prefer Ambari as the priority standard open-source management tool for Hadoop is great news because it means less duplication of effort for commoditized functionality. Momentum seems strongly against non-ASF technologies these days."