News

Pivotal Goes Open Source on Big Data

• By David Ramel
• February 17, 2015

After weeks of industry speculation, Pivotal Software Inc. today unveiled what all the hoopla was about: The company is open sourcing its Big Data platform, helping to found an Open Data Platform to promote technologies based on open source software, and is joining with Hortonworks Inc. to further Apache Hadoop capabilities.

Pivotal, known for its Cloud Foundry Platform-as-a-Service (PaaS) and related Big Data technologies such as the SQL-based front end for Hadoop called Pivotal HD HAWQ, is going to open source the underlying technology of its Big Data Suite.

"Core components of the Pivotal Big Data Suite -- including world-leading, analytical MPP data warehouse Pivotal Greenplum Database, the world's most advanced enterprise SQL on Hadoop analytic engine, Pivotal HAWQ, and the leading premium NoSQL in-memory database, Pivotal GemFire -- will be, for the first time, based on an open source core," the company said in an announcement. "The contributions will be an open, fully functioning core that will provide mission-critical resiliency, advanced client support, performance optimizations for demanding enterprise workloads and advanced operational tools."

The company said the suite will support bare-metal commodity hardware deployments, appliance-based delivery, and virtualized instances, along with now supporting public, private, and hybrid clouds. The Pivotal Cloud Foundry Operations Manager will provide Pivotal's Big Data capabilities as services running with and within Pivotal Cloud Foundry applications.

Company executives appeared on a live streaming event (available for replay) to talk more about all the big news, and Pivotal exec Sundeep Madra extolled the virtues of the open source movement, the company's new direction. He emphasized the importance of developers in creating appropriate software solutions, and with developers, he said, the action today is open source.

"The last thing, when I think about open source software, is that if I were to start a project tomorrow and try to solve a particular problem, my first place to look to solve a problem is open source now," Madra said. He added that anyone evaluating the correct open source solution will look at how well the software addresses a problem, the activity on its mailing list and other signs of a vibrant community surrounding the project.

Establishing and fostering such an active community is key, he said. "And what we've learned from Cloud Foundry is how to do that." The company maintains an open source Cloud Foundry project and develops a commercial edition based on that technology.

Madra said Pivotal will continue to maintain a commercial Big Data Suite product along with the newly open sourced project, offering advanced capabilities, enterprise support, customer indemnification and other add-on products and services. "And this is going to be a big differentiator for us," Madra said.

Pivotal also joined a host of other Big Data heavyweights in announcing the new Open Data Platform (ODP). "The ODP will promote Big Data technologies based on open source software from the Apache Hadoop ecosystem and optimize testing among and across the ecosystem's vendors," the company said in a news release. "These efforts will accelerate the ability of enterprises to build or implement data-driven applications."

At the live streaming event, Madra said the ODP Big Data consortium is being formed to "reduce fragmentation in the ecosystem." He said Pivotal and its partners will develop a common core of technologies so developers can advance the Hadoop ecosystem, much as the Linux project developed a common core to make the Unix OS more usable. While Unix was finicky, coming in different, perhaps incompatible distributions, anyone running applications on a specific Linux distribution would know that they would also work on other competing Linux distributions, he said. The ODP has a similar goal.

Pivotal CEO Paul Maritz took to the stage at the event (accompanying Twitter activity can be seen here) to explain more about the importance of the "data-driven future."

"From my perspective, what's really interesting is this deep transformation that's going on in business, not necessarily in technology, because we're seeing many businesses throughout the world have to confront the fact that the future is a platform -- an algorithmic and data-driven future. And the challenge is how do you take data and algorithms and use a platform to transform them to compelling end-user experiences. A lot of the interesting and innovative companies that are emerging are, first and foremost, platform, algorithmic and data-driven businesses."

Maritz said that as systems begin to leverage new data sources such as those coming from the exploding Internet of Things (IoT), and absorb new data from things along with data from people, "we see more and more businesses coming to grips with the fact that they have to chart a digital future for themselves. And a digital future means understanding how you can use platforms, data and algorithms and turn them into compelling, end-user experiences in order to drive your business. And as people are looking at doing that, they're understanding that a lot of the innovation in how one does that is coming out of the open source world. Just about every interesting innovation that's going on -- in data today, in machine learning and other areas -- has its roots in an open source ecosystem."

In the third major piece of news emanating from Pivotal today, the company and partner Hortonworks announced "a unified approach to meeting enterprise data management and analytics needs through a strategic and commercial alliance. Pivotal and Hortonworks will focus their efforts around a consistent core of Apache Hadoop-based capabilities, including product integrations, joint engineering and production support."

"Specifically, we will work with Pivotal on joint engineering that accelerates the enterprise capabilities of Apache Hadoop and YARN with Pivotal technologies like HAWQ and Pivotal GemFire," said guest Hortonworks bloggist Shaun Connolly on the Pivotal site. "We will also work with Pivotal to certify the components of Pivotal's Big Data Suite on the Hortonworks Data Platform (HDP) in support of customers deploying Pivotal technologies on centralized, YARN-based HDP environments."