The conversation, it seems, never changes. Whether talking directly to customers or conversing with experts at industry conferences, the database platform conversation always seems to revolve around how technology X is dead and technology Y is taking its place. One is old, the other new. One represents the past, the other the future. One is rapidly losing share, the other poised to dominate the market. Song as old as time, as the famous lyrics say.
Now, if you were to guess which database technologies are playing which roles in today’s version of this conversation, you’d likely guess that the part of the deader-than-dead technology X is played by relational databases and data warehouses, and that the part of the star-in-the-making technology Y is played by Hadoop and other big data platforms.
Surprisingly, you’d be wrong. The debate I illustrated above pertains to relational databases versus online analytical processing (OLAP). During the late 1990’s and early 2000, these two types of data environments were dueling for market share, with relational databases initially seen as the new, progressive, and innovative technology, and OLAP seen as the stodgy, soon-to-be obsolete technology. The outcome seemed predetermined. OLAP environments, along with the developers specializing in them and the applications built on them, would fade, and relational databases would become the new king, with all things data being housed in a data mart or enterprise data warehouse.
And yet, here we are, 15 years later, and the debate has subside regarding Relational vs OLAP but now rages on between Relational and “modern” big data platforms. Why? What happened to the supposed demise of OLAP? OLAP lives on to this day. And aren’t we supposed talking about the demise of relational databases in favor of the “modern” big data platform? The answer, as always, is that technology trends never play out in black and white terms.
The engineers on the front lines – the folks in the trenches getting their hands dirty with the real work of building systems – are not driven by what’s trending, but by what’s practical. When I talk to customers about whether they’d consider moving onto a newer, more modern platform, the answer is never a clear cut yes or no, but rather, “it depends.” It depends on the specifics of each customer, each use case, and each environment. For while there will always be certain technologies that emerge in the court of popular opinion as the new “it” technology, the reality is that no one technology or group of technologies can meet the needs of all the world’s uses cases. It’s just not possible, and anyone who tells you otherwise is either lying or selling.
The needs of real-world customers more often than not are mirrored in the needs of a good toolbox. No good toolbox has just one tool or one set of tools, but instead has everything that you need for your specific projects. Power drills might be the new “it” technology, but there are many cases in which you still might need – or prefer – the old fashioned screwdriver. Such is the case with all the different database platform technologies today, and it will be case with relational and big data technologies tomorrow. They co-exist; each with their place in the collective tool box of technology solutions, deployed based not on what’s trendy, but on what best fits each specific use case.
There’s also the issue of stickiness to consider. Stickiness – not to be confused with inertia – refers to the fact a given solution was tapped in the first place because the use case demanded it. And if the use case demanded it, well, then there’s often a good reason to stick with it, focusing on supporting, augmenting and enhancing the technology rather than replacing it. In addition to stickiness, there’s also the issue of skill set. One of the biggest misconceptions in IT is that you can just update the technology. That’s only half the battle. If you’re updating the technology, you have to update the skill set of those in organization supporting it, and that’s far easier said than done. Often times, organizations make the reasonable and viable decision to remain on given system because the skill set of their team is tied directly to the support, maintenance and enhancement of that system. It’s not about inertia, or “settling for what we have,” but instead about making the best use of the skill set, expertise and resources you have available.
Ultimately, technologies of the past do get ushered out, and technologies of the future do become the norm. One way or another, this always happens. But there’s a right way and a wrong way for organizations to make the shift. The right way is on your own terms, when it’s right for your company, and just as importantly, the people working for it. The wrong way is to rush to change because some pundit like me says you should. After all, in the real-world, anything “hot” usually also comes with caution sign. You’d be wise to heed it.