News

MIT Machine Beats Human Intuition in Big Data Analytics

• By David Ramel
• October 19, 2015

Just as IBM famously developed machine-learning supercomputers to conquer the best players of the TV quiz show Jeopardy! and beat chess grandmasters, MIT has come out with its own technology designed to take the human guesswork out of Big Data analytics, more than holding its own in three data science competitions with people.

And while the Data Science Machine didn't fare as well as IBM's Watson or Deep Blue supercomputers, which topped all human foes, it more than held its own in three initial competitions with humans.

The Data Science Machine was developed to remove human intuition from the Big Data equation that chooses what specific features of data to analyze in order to come up with hidden patterns that can lead to predictive insights translating into business initiatives.

"MIT researchers aim to take the human element out of Big Data analysis, with a new system that not only searches for patterns but designs the feature set, too," MIT announced on Friday. "To test the first prototype of their system, they enrolled it in three data science competitions, in which it competed against human teams to find predictive patterns in unfamiliar data sets. Of the 906 teams participating in the three competitions, the researchers' 'Data Science Machine' finished ahead of 615."

While the machine was at least as accurate as 96 percent of winning human entries in one contest, 94 percent in a second contest and 87 percent in a third, MIT said it also provided speed advantages, taking between two to 12 hours to come up with predictive algorithms that human teams typically labored over for months.

"We view the Data Science Machine as a natural complement to human intelligence," said Max Kanter, whose MIT master's thesis in computer science led to the technology. "There's so much data out there to be analyzed. And right now it's just sitting there not doing anything. So maybe we can come up with a solution that will at least get us started on it, at least get us moving."

Kanter will present his findings at the IEEE International Conference on Data Science and Advanced Analytics being held today through Wednesday in Paris.

Another data science expert from academia -- Margo Seltzer, a computer science professor at Harvard University who was not involved in the work -- had high praise for the project. "The Data Science Machine is one of those unbelievable projects where applying cutting-edge research to solve practical problems opens an entirely new way of looking at the problem," Seltzer said. "I think what they've done is going to become the standard quickly -- very quickly."