Halcyon days for Hadoop just around the corner
By Benjamin Cher December 28, 2015
- Enterprises increasingly finding more uses for Hadoop
- There are still obstacles to overcome, including simplifying it
HADOOP, the open source framework for storing and processing big data, has proven itself useful for businesses around the world, giving them new insights into customer behaviour and helping them reduce churn and speed up turnaround times.
But realising these business opportunities is not as simple as just plugging datasets into Hadoop and crunching numbers.
“There’s a really standard Hadoop adoption curve,” says Chris Harrold, global chief technology officer of Big Data Solutions at EMC Corp.
“About the time they [businesses] start doing something meaningful, they find that it gets unwieldy to manage from an IT perspective.
“Some of the staggering statistics out there is that nine out of 10 big data initiatives have failed because people built it before they found a use for it,” he adds, speaking to Digital News Asia (DNA) in Singapore.
This leads to companies giving up, and thus unable to get a return on their investment, according to Harrold.
“It’s all about identifying what you want analytics to do for you,” he says.
It doesn’t help that Hadoop still has the image of a do-it-yourself project, according to Neil Mendelson, vice president of Big Data and Advanced Analytics at Oracle Corp.
“People tend to download a distribution onto a white box and try to piece it together,” he says.
“We’re seeing more and more of that, leading to a failure, or spending more time and money trying to get it up and running,” he adds.
Yet for those who do manage to work through the adoption, the benefits of Hadoop can be transformative.
“There is a media company with traditional print and online content, and it was struggling with people not renewing their subscriptions,” says Mendelson (pic above), declining to name the company concerned.
“It went ahead and used predictive analytics to determine the propensity for customers to drop their newspaper subscriptions,” he adds.
Insights from the use of analytics allowed the media company check in on subscribers who might be dropping out, and to give them another offer for subscription.
“It was able to pick, with about 92% accuracy, which customer would drop the newspaper, and boosted its revenue by 64% in doing so,” Mendelson claims.
Even aircraft-makers like Airbus have found uses for Hadoop, according to Mendelson, noting that airplanes have to go through a very rigorous process before being handed over to customers.
“There are no corners to cut, you have to go through it and you have to justify both to yourself and the regulator that the plane is in good working order,” he says.
The turnaround time for this process is often very long, and it is better to get the plane off the manufacturer’s books as quickly as possible.
“Airbus was previously ingesting data and then processing it,” says Mendelson.
“After it picked up Hadoop, it was able to process data a lot faster and turn around a plane a lot quicker.
“The time between test flights was reduced because Airbus could process data faster,” he adds.
Roadblocks on the Hadoop journey
Yet for all the benefits companies can get from Hadoop, many are unable to make it past the teething stages.
“Most companies will run into the fact that it is painful, and a lot of them give up and go back to standard data warehousing, which is unfortunate,” says EMC’s Harrold (pic).
“There is a lot of ways to fix it, better ways to work with Hadoop,” he adds.
A key part of overcoming this roadblock is in understanding Hadoop, and its limitations and strengths, according to Harrold.
One of these strengths is the ecosystem being built around Hadoop, with the community developing features like real-time analytics.
Enterprises have embraced Hadoop because they have the resources to figure it out, Harrold argues. But the real breakthrough will be in making Hadoop approachable for the small guys – the traditional small and medium enterprises (SMEs) which were not born in the digital age.
“Once we make it so that Hadoop is consumable by everybody, you will see explosive growth,” he says.
“There’s a huge pent-up demand sitting in the mid-market range waiting for Hadoop to get easier to use.
“Once we as an industry get that right, the sky’s the limit on what Hadoop adoption will look like,” he adds.
A conversation with the ‘father of Hadoop’
Maximising the value of big data analytics
Rising Hadoop use, but deployment challenges remain
For more technology news and the latest updates, follow us on Twitter, LinkedIn or Like us on Facebook.