Guest Post by Kunal Agarwal, founder of Unravel, a provider of Application Performance Management (APM) for big data
Big Data is evolving in a way that resembles the growth of the cloud. At the beginning of 2017, people were still navigating the basic Big Data ecosystem, technologies like Hadoop or Spark or Kafka. As 2017 ends, those conversations have finally been settled, and the industry is ready to move to a new era where the focus shifts to use cases from systems.
Big data applications will rise as a clear solution to address and solve challenges as we move toward this next step. In 2018, enterprises will shift their focus from traditional big data technologies more toward mission-critical big data applications as they begin to see their true value.
In the past, people were focused on learning the various big data technologies: Hadoop, Spark, Kafka, Cassandra, etc. It took time for users to understand, differentiate, and ultimately deploy them. There was a lot of debate and plenty of hype. Now that organizations have cut through the noise and figured all that out, they’re concerned about putting their data to use – this is where the applications come in.
Take the recommendation engine, for example, a critical app for most all web companies. Consider Netflix: Their recommendation engine isn’t just a nice add-on that enhances the user experience, it’s fundamental to the experience and to Netflix’s bottom line. Their platform depends on the ability to accurately suggest relevant movies and TV shows to people – otherwise, it’d be almost impossible for viewers to dig through their enormous library.
Netflix/the enterprise doesn’t really care about the technology being used. It’s not important which distribution or database or analytics they’re using, what matters is the result. The enterprise has realized this and we can expect to see an increased adoption of an application-centric approach to big data in the coming year
In 2018, with the use of these big data applications, enterprises will learn how to apply their systems and tools to get the most out of them. This next phase is about driving value from big data, a next step that will give companies a considerable edge as we look toward the year ahead.