Why 85% of Big Data projects fail
By Sharala Axryd April 16, 2019
- People from the top must define clear problem statements
- Deciding to become data driven can be a long, difficult process
MCKINSEY Global Institute Study shows organisations that harness Big Data and Analytics are: 23 times more likely to acquire customers; Nine times more likely to retain customers and; 19 times more likely to be profitable
But a year or so, Gartner estimated that 60% of big data projects fail. As bad as that sounds, according to Gartner analyst Nick Heudecker Gartner was "too conservative" with its 60% estimate. The real failure rate? "Closer to 85%."
Since then, the tweet Heudecker sent has been deleted.
This is a hard truth that illustrates that the problem isn’t the technology, it is you!
Leadership and strategy work hand-in-hand. In the past, Big Data Analytics would be a strategy in itself but with the maturity of the technology and abundance of data, the strategy is to sit and look at the crucial problem statement.
It is true but very upsetting to know that the biggest failure in Big Data implementation lies in the C-suite underestimating their involvement in carving out the problem statement.
The key decision-makers in C-suites should sit and discuss the primary pain points the company is facing. They will need to work on a few that would make the biggest impact.
These issues are usually left to the chief data scientist (CDS) (who is usually a very intelligent geek) or the chief data officer (CDO) (again – a very smart person who knows how to manage the company’s overall data governance and sees the big picture for data priorities and strategy).
I remember having a conversation with an ex-chief financial officer (CFO) of a multi-billion dollar company, who suggested that we should churn out more business-minded data scientists.
I believe that we are looking at this all wrong. If a data scientist can address the business problem and the business strategy, the work of the chief executive officer (CEO), chief marketing officer (CMO) and CFO would become irrelevant. This statement can anger the C-suite. It is hurtful but it is important to remind the management that strategy runs from top to bottom.
When this is left to the data science group and line managers or individual departments, you are then left with 200-300 Big Data projects from which, I predict, more than half will fail. After all, failing to establish order and governance over Big Data projects, leads to chaos and poor business decisions and places businesses at a severe disadvantage in today’s data-driven world.
Harvard Business Review indicates that a data strategy helps organisations “clarify the primary purpose of their data and guides them in strategic data management.” Astoundingly, according to management consultants McKinsey, 30% of banks have no data strategy.
Deciding to become data driven can be a long, difficult process that once decided upon, can spur a rush to try to attract data specialists and make scientific inferences before knowing the real problem.
This may not seem like a problem because after all, we need data and these specialists know how to handle it. But do we stop to think what data we need? This is where a data strategy can be overlooked and is therefore crucially missing from a business’ overall strategy.
‘Communications’ are often still taken for granted by management as something that just ‘happens’ through email, WhatsApp messages, calls or spreadsheets. However, without a well-defined communication strategy, many companies face monetary wastage with every project they attempt to execute.
Poor communication is the primary contributor to project failure one third of the time, and can have a negative impact on project success more than half the time.
Once the C-suite has identified crucial strategic initiatives, they must communicate these initiatives down to the line managers. Now, it is extremely important NOT to pass it down without equipping these managers with the relevant technology and talent.
Being in this vulnerable state can lead to ambiguity, noise and complexity, especially if teams aren’t ready to discover, interpret and use the data in decision-making.
Line managers from various department need to come together and exchange notes on what data is required to help achieve the problem/strategic statement defined by the C-suite. Having a clear mission will help to recognise data that is going to support business objectives and the sources that might distract from achieving goals.
Everybody says critical thinking is a must-have skill but never has this been more true than when it comes to investigating insights. Don’t purely and blindly take data as fact without ensuring its accuracy or assessing the potential impact this data-driven decision could have on your company.
The tricky part is getting them to identify, store, collect (if needed) and run analysis on the data. This brings us to the next topic.
Lack of skills
The lack of skills in organisations contributes 30% of the failure. This affects or takes effect on several levels:
- C-suite not having the digital leadership mindset to drive strategy
- Line managers not understanding the data they have within them
- Rest of the company, not understanding the lingo of analytics
The danger lies in there not being a culture that normalises embedding analytics in their daily work. Usually this would affect 80% of the company’s population. No initiative from above can be driven because the rest of the organisation is still stuck in the past and everything that is being spoken would be perceived as ‘back to the future’, therefore perpetuating resistance to change.
Unfortunately, dataphobes are likely to squander promising business opportunities and often fail to see problems until these problems become full-blown crises. As we know, it is human nature to fear what we don’t understand. Most people, let alone companies, are not prepared to adopt radical changes and become data driven.
The challenge lies in ensuring Big Data projects perform reliably and efficiently enough so that organisations can flip their mindsets from considering Big Data only as a defensive tool for current activities to using it as a catalyst for business growth.
Nearly all companies that embark on becoming data-driven organisations or digital transformation initiatives are too ambitious. They either spend millions of ringgit on infrastructure or claim a framework for analytic or digital transformation that might not be wholly sustainable or stable.
Having observed this for years, it is clear that this has to be tackled from above and below.
The C-suite may try to achieve sweeping change throughout the company to go data driven — which can lead to counterproductive overreaching.
Organisation are expected to give a quick ROI because of the investment made. The issue here is that the investment can be made with a lot of assumptions. The reality is the investment was made based on how a traditional business would do it. They will tender and buy equipment, then assume that with a click of a button, it will solve the company’s problems. I usually say that the selling point is this – press that button and it will solve world hunger.
Data is based in reality by examining what is actually happening. Therefore, decisions should be grounded in facts as much and as often as possible.
Emerging victorious in this landscape of digital transformation will not be possible by making huge bets. Winners of the digital age must be agile, pragmatic and disciplined. They follow a carefully devised transformation roadmap to optimise performance in the functions and operations that create the most value, while building the technical proficiency and resources to sustain the transformation.
Sixty percent or 85% is a big number and cannot be brushed aside. But this could be rescued by having some simple but hard measures put in place.
People from the top must define clear problem statements. They need to have a data-driven session to thrash out strategy and priorities.
From there, the C-suite have to parallelly initiate the whole organisation to be data ready. This would be a top-down approach where the journey will allow you to meet in the middle, thereby allowing quick wins and long-term initiatives to be driven clearly.
Another important thing is that, based on the maturity of the industry, companies could strategise on whether they need to quickly build their use cases or have a team, either internal or external, to operate on the problem statement so that they can hit the ground running. This helps to get the buy-in from the management quicker.
Always build a talent strategy around whatever problem statement is produced to ensure there is an opportunity to inherit the solution. Companies can then take it and run it on their own.
Sharala Axryd is the founder and CEO of The Center of Applied Data Science (CADS).
Related Stories :