News

Reports Indicate Big Data Poised to Pay Off in 2017

• By David Ramel
• January 9, 2017

One is from a Big Data vendor, and one is from a data-driven management consultancy, but two new reports both indicate large-scale analytics is finally poised to pay off on its promise in the new year.

"Data and analytics are already shaking up multiple industries, and the effects will only become more pronounced as adoption reaches critical mass," says the McKinsey Global Institute (MGI) in its executive overview of last month's report: "The Age of Analytics: Competing in a Data-Driven World."

Meanwhile, BI-on-Hadoop specialist AtScale Inc. published its second "Big Data Maturity Survey," taking issue with what "Big Data is dead" naysayers have been proclaiming for years.

"Big Data is far from dead, and instead technologies and solutions that make up the Big Data space are maturing at an ever increasing rate," the AtScale report said.

Not that there isn't much work to be done for Big Data to pay off on its much-hyped potential. Both reports indicate organizations are at different stages of success in that department, and both offer insights to help organizations finally realize the Big Data dream.

To highlight that work to be done, MGI -- the research arm of management consultancy McKinsey & Company -- contrasted its latest report with the inaugural edition five years ago.

"Most companies are capturing only a fraction of the potential value from data and analytics," the company said. "Our 2011 report estimated this potential in five domains; revisiting them today shows a great deal of value still on the table."

Those five domains include location-based services, retail, manufacturing, the public sector and health care. The location-based services and retail industries have done the best job of making Big Data progress, but the other domains have thus far lagged behind.

Specifically, the MGI report said, while location-based data (think GPS in smartphones) has achieved 50 percent to 60 percent of that potential value -- and the U.S. retail industry 30 percent to 40 percent -- the other industries have reaped less than 30 percent of the identified benefits.

"The biggest barriers companies face in extracting value from data and analytics are organizational; many struggle to incorporate data-driven insights into day-to-day business processes," MGI said. "Another challenge is attracting and retaining the right talent -- not only data scientists but business translators who combine data savvy with industry and functional expertise."

That Big Data skills shortage is a familiar culprit in keeping organizations from fully utilizing large-scale analytics and data science to glean business insights. For years now, we've been covering various reports about this problem in articles such as "Survey: Big Data Implementations Lag Behind Interest" and "Big Data Skills Getting Harder to Find."

Despite numerous remediation efforts on the part of industry and academia, the skills shortage is still holding enterprises back, indicates the MGI report, which cited "a shortage of technical skills" as a prime factor in the Big Data underperformance in the U.S. health care industry in particular.

"Human capital has proven to be one of the biggest barriers standing in the way of realizing the full potential of data and analytics," MGI said.

"Our 2011 report hypothesized that the demand for data scientists in the United States alone could far exceed the availability of workers with these valuable skills," it continued. "Since then, the labor market has borne out this hypothesis."

AtScale, the self-service, BI-on-Hadoop specialist, also noted the Big Data skills shortage, saying "skillset continues to be the top barrier to providing self-service access."

Despite the continuing skills shortage, AtScale noted that "Big Data is growing fast" in reporting on its maturity survey that polled more than 2,550 users.

Some of the survey highlights include:

• 97 percent of respondents to the new survey will do as much or more with Big Data over the next three months.
• Two-thirds (66 percent) view Big Data as "Strategic" or "Game Changing," while only 19 percent of respondents consider it experimental.
• 72 percent of respondents plan on doing Big Data in the cloud.
• Governance is the fastest growing area of concern year-over-year (21 percent YOY).
• 95 percent of respondents have achieved positive value with Big Data or are anticipating they will.
• 73 percent of respondents are now in production (versus 65 percent last year).
• 95 percent of respondents have achieved positive value or are anticipating they would.
• One in five have more than 100 nodes.
• 25 percent have Apache Spark in production today, while 21 percent plan to use Spark in production.

"From traditional players like Teradata, to open source Hadoop, to new cloud players like Google Big Query, the Big Data space is doing more to help companies manage and gain insights from their exploding and morphing data than at any other point in history," AtScale said.

For its summation, MGI stressed the corporate incentives to get onboard with Big Data: "Data and analytics are already shaking up multiple industries, and the effects will only become more pronounced as adoption reaches critical mass.

"An even bigger wave of change is looming on the horizon as deep learning reaches maturity, giving machines unprecedented capabilities to think, problem-solve and understand language. Organizations that are able to harness these capabilities effectively will be able to create significant value and differentiate themselves, while others will find themselves increasingly at a disadvantage."