Telruptive

Big Data is a hype right now. Everything that comes close to Hadoop or NOSQL turns into gold! Unfortunately we are getting close to Gartner’s “Peak of Inflated Expectations”. Hadoop does an excellent job at storing many tera bytes of data and doing relatively complex Map-Reduce operations. Unfortunately this is just the tip of the Big Data requirements iceberg. Doing intelligent Big Data analytics requires more than counting who visited a web site. Map Reduce is able to do complex machine learning but it is not really made for it. The Mahout project has to jump through too many hoops to convert matrix-based analytics algorithms into Map-Reduce enabled versions. Map-Reduce just is not an easy way of doing matrix-based operations. Unfortunately most machine learning algorithms rely on matrices. Also real-time and batch often go together in real live.  You need to pre-calculate recommendations or train a neural network but you do want recommendations, predictions and classifications to be in real-time. Unfortunately Hadoop is only good at one of the two.

So when the majority of investors and business analysts realize that Hadoop has limitations, what will happen?

Answer: Nothing unexpected. Hadoop will continue to be used for what it is best. A new hype will arrive as soon as somebody solves the real-time distributed analytics problem…