/data/universe/

I am bullish on the potential of increasingly pervasive data storage and analysis (one sense of “Big Data”) to improve outcomes in business, government, education, and our personal lives.  The cost of storage is plummeting (though providing useful access to that data has a nontrivial cost).  Faster computers, better algorithms, and increasingly experienced data scientists are arriving to take advantage of the bounty.  Even small gains in efficiency can be valuable for large industries, and Kaggle has repeatedly shown that modern machine learning can beat industry benchmarks.

This author provides a skeptical take on the size of the analytics market, though, highlighting a range of assumptions which may not hold:

The basic hypothesis goes like this: ‘It’s becoming ever cheaper to collect and store data due to ever increasing networking, new database structures and rapidly falling data storage costs. Companies are collecting mountains of data. Now they want to extract value from all this data they have, but don’t know how to do it. Companies that help them extract the insights will be well rewarded.‘

There are a number of sub-hypotheses. A) This data collection is a recent phenomenon B) Value is not already being extracted from whatever data is being collected C) Companies will need outside help to extract insights D) Outsiders can help companies extract insights without having deep industry knowledge E) The insights gathered from ever larger data sets have more value and are more accurate than insights gathered from smaller data sets F) Unstructured and cross functional data have huge value waiting to be extracted.

Notably, Netflix never implemented the million-dollar algorithm that won the Netflix Prize, as the cost of implementation didn’t justify the improvements over the intermediate results from the competition.

The history of artificial intelligence research is filled with boom and bust periods driven by the hype cycle.  Ultimately, measurement and quantitative assessment are vital to achieving the ends we desire–but let’s not overpromise the potential returns of Big Data.

Update: Other skeptical reactions on Hacker News.