Premise
Both customers and vendors need to prioritize how they address adoption barriers. As with all emerging technologies, a full solution will often require extensive 3rd party participation, such as “data wrangling” and SQL data access tools.
SQL data analysis offerings are maturing rapidly in the areas of application performance under greater user and data volume loads both from the Hadoop distribution vendors as well as 3rd parties. Other barriers, such as a skills gap across many roles, are more intractable. Smaller customers with fewer specialized practitioners in each role should include in their evaluations cloud-based solutions that are fully managed services.
Summary of Survey Results
Based on additional analysis of data from a survey of 300 Big Data Analytics customers, Wikibon has identified the top barriers to successful adoption and deployment*:
- Among evaluators: skills gap, integration with existing infrastructure, uncertainty regarding which vendors to use
- Among those conducting proofs of concept: performance problems as data volumes grow, performance problems as concurrent users grow, integration with existing infrastructure
- Among those deployed: skills gap, performance problems as data volumes grow, difficulty getting data ready for analysis
*(for full details of barriers, see Table 1 below)
Survey Results and Discussion
The barriers are listed below in Table 1 in the order they were identified as the “Most Significant Barrier” across the entire sample. For instance, Lack of Skilled Big Data Practitioners and Difficulties in Managing and Merging Data are foremost in inhibiting Big Data Analytics adoption, but there were some differences by phase of adoption. Users and providers should both be aware of just when a barrier is a barrier and when it can be best anticipated and be best addressed.
Evaluation Phase
During the Evaluation phase where users are trying to determine how to get the value promised by Big Data Analytics, the perception of a Lack of skilled Big Data practitioners is the #1 barrier (as it is overall), followed closely by the Difficulty integrating Big Data with existing infrastructure. However, Confusion/uncertainty regarding the vendors/technologies to use rises to the #3 position as the most significant barrier. This makes sense since the Evaluation process is one of discovering the skills, tools, and partners required to move forward, while also considering how to leverage the existing infrastructure required to make a project work.
There are considerable complexity concerns at this stage – giving rise to fear, uncertainty, and doubt. The premium should be on identifying the internal sponsors who can mobilize investment, providing the consulting support and expertise (internal and external) to work through the complexity at all levels (skills, technology, and organization), and helping to support the enterprise to mount a Proof of Concept project as a method to demonstrate value to the C-Suite.
Line of business users will believe there is a significant amount of value to be had from this technology, but they don’t have the wherewithal to progress to operationalizing even the basics of a POC. Getting them launched with a simple project with the right level of visibility that shows the value should be the primary goal for prospects at this stage.
Proof of Concept Phase
Those in the midst of a Proof of Concept have begun to demonstrate value and have largely moved beyond the FUD of the Evaluators. The challenge of Integrating Big Data with the existing infrastructure is closer at hand, but operational concerns such as Maintaining application performance as data volumes (and concurrent users) increase beyond a more modest POC begin to show up.
The challenge is to move beyond the demonstration to something that is actually operational – both in terms of the value it can produce for the user as well as being sufficiently compatible with existing business processes and methods to be seen as a useful complement to and accelerator of better business decisions. While it is not the “most significant barrier”, Big Data technology is too raw and difficult to use is a factor more frequently during the POC phase of adoption, when the users begin to actual grapple with the complexity of the tools and the data.
Deployment Phase
Those in the midst of deployment also identify Lack of Big Data Skills (when the rubber really meets the road) and Maintaining Application performance as data volumes increase. However, it’s during deployment where the Merging of multiple disparate data sources and Data transformation for suitable analysis become dominant concerns. Merging data is at the heart of the value of Big Data Analytics, so when enterprises are deploying a real mission critical Big Data Analytics application at an operational cadence, the daunting challenges related to actually getting control over the data become a priority. Some of the concern about Big Data Skills is probably related to data management and merging datasets for analysis.
Conclusions
Big Data Analytics prospects and users face significant barriers as they continue on their journey. Evaluators need a lot of support just getting on the right path and need to feel confident that they (and their partners) can move forward successfully. Those in the Proof of Concept stage need to feel confident that they can move forward with a real business process deployment. The key barriers for those currently in deployment are around handling and managing the data for its potential business use. Providers of Big Data Analytics products and services need to be able to provide appropriate support and tools at the right time to assist prospects and customers to continue moving forward effectively.
Methodology
Wikibon recently completed an extensive survey of Big Data Analytics (n=300 web survey interviews in the US) focusing on current practice and barriers to successful deployments. of Big Data analytics projects. Big Data Analytics projects are those that (1) leverage non-traditional data management tools and technologies such as Hadoop, NoSQL, or MPP analytic databases and/or (2) involve the analysis of multi-structured and/or unstructured data such as clickstream, text, log file, and social media data. An example of such a use case would be the use of Hadoop to store, transform and analyze mobile sensor data. Big Data projects do not include projects involving the use of relational databases to analyze traditional structured data associated (e.g., CRM, ERP, Finance, etc.)Our Big Data clients have received a report of our initial conclusions.
This report digs more deeply into the many barriers inhibiting adoption of Big Data Analytics by focusing on the relative importance of these barriers as a function of the stage of deployment (Evaluating, Proof of Concept, and Deployment) of at least one Big Data Analytics application.
Stage of Deployment Definition
Of course, there are many barriers faced by those adopting Big Data Analytics. Some barriers are troublesome and consistent throughout the entire journey to adoption, while others are less troublesome overall. Moreover, some barriers are more or less challenging at different stages of the adoption process.
For our purposes, we had each Big Data Analytics respondent class themselves on the following scale:
- We are currently evaluating Big Data Analytics use cases and vendors/technology.
- We have at least one Big Data Analytics pilot/proof-of-concept project underway
- We have at least one Big Data Analytics deployment in production supporting mission critical business processes and/or applications.
- We have at least one Big Data Analytics pilot/proof-of-concept project underway and at least one Big Data Analytics deployment in production supporting mission critical business processes and/or applications.
