The Hadoop Elephant in the Room

Ok, how many of you have been approached in the hallway with the question, “…what’s our big data strategy…?” or “…what are we doing with Hadoop?”. With any new, sexy technology, the hype is bound to get the attention of others. That means that well-meaning executives don’t want to be left out.

The reality of it is that tools used to get value out of big data are still very new and many IT professionals have not worked with these yet. Just because someone can use MapReduce correctly in a sentence does not mean that he/she can instantly make the company more competitive by using Hadoop or other no-SQL databases. Its like the guy who goes out to buy a table-saw because “…I can make my own cabinets…”; when in reality it ends up gathering dust or just cutting 2×4’s. Contrary to all the hype, the data warehouse and other more traditional analytics are not dead. It’s like the Mark Twain quote: The rumors of my death have been greatly exaggerated.

Anyone who asks you about Hadoop or other big-data tools is asking the wrong question. The real question is: Is the organization’s approach to analytics mature enough to benefit from Hadoop? Many organizations struggle with just basic metrics for their business. So spending time and effort to make the developers learn MapReduce without a real sense of how to use it, will take a lot of time (and money) with very little chance of true value coming from it.

While its not sexy, the real value for many (but not all) organizations is to incrementally deliver richer analytics to the business through the databases and reporting tools that you likely have at your disposal. For example, anyone who has SQL Server running in their organization also can use the SQL Server Reporting Services (SSRS) and Analysis Services (SSAS) without much more investment in time, effort or money. Especially SSAS – by creating some OLAP cubes – can shine a light into areas the business has not looked. No, it’s not a “shiny toy” to show your execs, but more insights from existing data and tools will definitely catch their attention.

I’m not saying that big data doesn’t have intriguing potential. What I am saying is: without a clear vision of what is to be done with these new tools, it’s dangerous to simply jump in to a new initiative…especially since there will be significant time and effort (and therefore money) needed to get something working well. So first-things-first: Work with the business to get on solid footing with traditional analytics and that will naturally lead into a constructive conversation about how to leverage big data. Easier said than done…but it’s the right thing to do.

Business Cliche of the Day (followed by the real meaning):  We need to go back and sharpen our pencils – “We really have no idea what we’re talking about right now and we’re just trying to buy time”


Veteran technology professional and manager

Posted in data warehousing

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: