On the Death of Big Data

Big Data died when I wasn't paying attention.

Recalibrate the hype machine: It's "all data" now. I learned this while talking to industry experts to research an upcoming article on SQL Server 2014 and how it fits into the new-age world of what-used-to-be-called-Big-Data.

Analyst Donald Feinberg at research firm Gartner Inc. doesn't even like to use the term. Don't get him started.

"'Big Data' is going to disappear in the next two or three years and it's going to be called -- guess what? -- 'data,'" Feinberg told me. "Once every vendor has a Big Data something, it no longer gives me a way to distinguish one vendor from another."

I also talked to David Torres, senior director of Data & Analytics at Avanade Inc., which provides Microsoft-focused consulting services. "People ask, 'OK, what are you going to do with Big Data?’ Torres said. "I say, 'What are you doing with all data? What are you doing with your relational data? What are you doing with streams -- dark data? Are you taking advantage of all your data and really kind of using it as a corporate asset and trying to figure out how it can help you innovate your business?' Beyond Big Data, it's really, in my opinion, it's all data. You need to have a strategy for all your data."

Microsoft's strategy is to instill a "data culture" within an organization where everybody has access to vast new stores of analytical information of varying types and can use it to improve business processes and products. For what-used-to-be-called-Big-Data, Microsoft is providing several products and services for leveraging the traditional Apache Hadoop ecosystem while also taking advantage of the democratization of the data and increased access to it from technologies such as SQL. The company is actually baking a lot of the "BD-word" functionality into the company's flagship RDBMS, SQL Server 2014.

For Pivotal Software Inc., the strategy is a new term: "data lake."

"They don't talk about Big Data anymore, they talk about data lakes," Feinberg said. "It's something new. They coined the term."

Feinberg doesn't like that term, either -- "we hate it" -- though it does serve to distinguish Pivotal from other vendors in the space. "But the reason they coined that term, is because Big Data doesn't mean anything anymore, since everybody has something."

Another expert, Andrew Brust, doesn't seem so sure about Microsoft's term: data culture.

"Well, you hear lots of discussions about making businesses more data-driven, and I think that 'data culture' is kind of Microsoft's and [CEO Satya Nadella's] little variance of that same message," said Brust, a SQL Server MVP and research director at Gigaom Research. "I mean, it's real. If we can get businesses to the point where it's just a matter of course and practice that decisions are based more on data and metrics and less on hunch, that's I think an efficient thing and good thing. And it's obviously easier rhetorically than in practice to get there, but it's absolutely what's been driving all the changes in data in the last five years or so."

It certainly has been driving changes within Microsoft and SQL Server. Alex Barnes, a DBA on the front lines actually using SQL Server, noted this while talking about the 2014 version's new delayed durability capability, which concerns when a database write operation is acknowledged by the system. NoSQL databases play "fast and loose" with some of these traditional database rules for the sake of speed, said Barnes, who works at Charleston, S.C.-based BoomTown, a company that provides a platform for real estate professionals.

"SQL Server in 2014 has actually implemented an option to turn on delayed durability, which is basically, you could call it, like, 'NoSQL mode.' It's a way to say, 'I don't need write-ahead logging, I don't necessarily care if my system crashes and I lose a little bit of data. I'd prefer to have the speed up front. They've really kind of added these options in there to compete with these guys who are playing fast and loose with these rules. You can basically, now, by jamming a bunch of data into memory, creating these hash indexes, or range indexes, and turning on this delayed durability, you can basically turn SQL Server into what people think of traditionally as a NoSQL database."

Ted Neward, an expert database developer, author and consultant, noted how the NoSQL and traditional relational SQL worlds have been evolving and reacting to one another. The highly scalable NoSQL databases were an upstart technology taking on the traditional RDBMS vendors with new capabilities and agility. Neward noted how you can download, install and start running some NoSQL databases in less time than it takes to read about how to do that for SQL Server.

Microsoft has responded, Neward said, trying to provide some of the advantages of the NoSQL movement while not leaving its traditional customers behind. And now the "other guys" have responded in kind, opening up more to the SQL world.

"There was certainly a movement in the NoSQL world that basically said, 'dump your SQL databases, dump your relational databases -- they can't handle this new Internet world," Neward said. "Which I think was always a misnomer and always a mistake, and we see them changing their tune now, now that NoSQL supposedly stands for 'Not Only SQL'" instead of "No SQL."

"That's B.S.," Neward said. "That's revisionist history. We're not stupid."

Well, I may be stupid, but it's becoming increasingly clear that Big Data and "traditional data" are merging into one coherent, integrated data analysis concept. A quick Internet search shows terms like "Big Data is dead" have been out there for years now, propagated by forward-thinking pundits. Who knew?

Noel Yuhanna knew, for one. The analyst at Forrester Research Inc. is paid to know these things ahead of dolts like me. He agreed with Avanade's Torres about the need for organizations to adopt a coherent, overarching data strategy encompassing technologies of all types.

"Enterprises that have a more complete data platform story, as well as a vision, are more likely to succeed in the coming years and also have a competitive advantage if they get onto this bandwagon of data platform, which includes Hadoop, Big Data, NoSQL as well as traditional databases -- all integrated," Yuhanna said. "Because that's where you see customers that are more successful, having all those data types together and managed together and provided together in a manner that will be helpful for businesses to operate."

So, goodbye, "Big Data." R.I.P.

Now, about this "cloud" thing ...

Posted by David Ramel on September 8, 2014