Premise
Across all the primary roles involved in adopting Big Data applications, there are basic gaps in product maturity. However, IT leaders and practitioners should keep in mind that Big Data databases are part of a relatively immature ecosystems that requires advanced skills and integration technology in order to operate successfully. The ecosystem is evolving and maturing rapidly and there is a tremendous proliferation of technologies to augment early products.
The largest customers have the greatest ability to manage an ecosystem that is still maturing. For them, all five of the primary constituents Wikibon has identified need to participate in the evaluation and proof-of-concept phases that will involve multiple vendors. Smaller customers without access to all the skills to deploy and operate a still-maturing platform should also consider cloud-based options that operate as fully managed services.
Summary of Survey Results
Based on additional analysis of data from a survey of 300 Big Data Analytics customers, Wikibon has identified the top technology barriers to successful adoption and deployment:*
- Business user: data accessibility (37%), uncertainty regarding vendors/technologies (31%), security (27%)
- Business analyst: skills gap (47%), data accessibility (34%), uncertainty regarding vendors/technologies (32%)
- Application developer: security/backup/HA (83%), data transformation (35%)
- Data scientist: integration with existing infrastructure (36%), concurrent user performance (36%), too difficult (31%), lack of backup/recovery (31%)
- Infrastructure Administrator: data transformation (38%), performance with data volume growth (31%)
*Totals are greater than 100% because each respondent picked their top 3 issues. For full details of barriers, see table below
Perceptions of Technology Barriers by Different Roles
Given that many projects were in the Evaluation and Proof of Concept stages, most respondents claimed to have more than one role (in fact they averaged 2 roles each). We also asked respondents to identify their Primary Role, which is what the data in Table 1 is based on. Given the small scale of many of these Big Data Analytics projects, participants were wearing multiple hats in effecting solutions. It’s likely that the differences by roles is attenuated somewhat by the fact that the average respondent had multiple, overlapping roles in the Big Data Analytics environment. But the differences seen in Table 1 do seem to reflect the primary role each respondent defined for themselves.
In order to make the table easier to read, we’ve color-coded the responses to show how the roles react differently to each barrier. Green is a tendency for the role to cite the barrier more frequently (>5% pts.) than the average, red a tendency to be lower (<5% pts) than other roles. The barriers are ranked ordered by the frequency of their mention as one of the Top 3 Barriers to Big Data Analytics in their enterprise.
Differences in Perception of Importance of Technology Barriers to Big Data Analytics by Role in Project/Deployment
Application Developers – Most barriers ranked similarly to the aggregate, but Application Developers identified a number of problems having to do with enterprise-grade IT infrastructure practices. The following were especially important barriers as seen by Application developers:
- Transforming data into suitable forms for analysis – difficult data transformations
- Technology lacks enterprise-grade backup and recovery – general IT concerns
- Technology lacks enterprise-grade security capabilities – compliance issues
They were a little less concerned than other roles with:
- Difficulty integrating Big Data with existing infrastructure
- Difficulty merging multiple, disparate data sources
- Lack of skilled Big Data practitioners – they have the skills
- Confusion/uncertainty regarding the vendors/technologies in use – this is their bailiwick
Business Analysts – The Business Analysts cited lack of skills and knowledge as the most significant barriers for them.
- Lack of skilled Big Data practitioners
- Confusion/uncertainty regarding the vendors/technologies in use
- Difficulty merging multiple, disparate data sources – the practical matter of performing analytic tasks requiring the merging of various data sources
They tended not to be concerned with IT issues surrounding the details of the Big Data Analytics, per se:
- Technology lacks enterprise-grade backup and recovery – general IT concerns
- Technology lacks enterprise-grade security capabilities – compliance issues
- Big data technology too raw and difficult to use – for them it’s not their problem – but rather that of others. They just do the analysis.
Business Users – Like Business Analysts, Business Users cited lack of skills and knowledge as the most significant barriers for them as well as the daunting nature of merging multiple, disparate data sources.
- Lack of skilled Big Data Practitioners
- Confusion/Uncertainty regarding the vendors/technologies in use
- Difficulty merging multiple, disparate data sources – the practical matter of performing analytic tasks requiring the merging of various data sources
Similar to Business Analysts, they tended to be less concerned with:
- Technology lacks enterprise-grade backup and recovery – general IT concerns
- Transforming data into useful form for analysis – not their job
Business Users were the single largest group in our survey (~33% of our sample). They tended to have high expectations of using Big Data Analytics as a new source of competitive advantage. These users were likelier than others to be in the Evaluation stage of deployment, suggesting that their expectations were more likely to be aspirational and ambitious.
Business users expressed great confidence that they were on the right track (they disproportionately characterized the status of their “projects” to date as a “Success”) while respondents in other roles were more modest and tempered in grading how well things were going with their projects and deployments. This is to be expected at the early stages of adopting a new technology where adherents are imagining all the potential benefits and not seeing the practical challenges and barriers to that end. But they are a source of enthusiasm to draw on for the sales motion.
Data Scientists – Being analytics sorts by nature and immersed in technical details, Data Scientists enumerated more barriers overall than those in other roles. Key for them were barriers related to operationalizing the Big Data Analytic application and some IT related considerations. Aside from respondents declaring their primary Big Data Analytics role as IT Administration, Data Scientists were much more likely to belong to IT than other enterprise departments.
As such, their perspectives regarding barriers can be attributed in part to their viewing barriers through the lens of operationalizing the applications up to IT Dept standards. So integrating and maintaining application performance and scalability are key considerations for these folks. By the same token, they don’t see the lack of skills or vendors as a problem – but do believe far more than others that Big Data technology is too raw and difficult to use. Given that they are the users of the tools that vendors are providing, one must conclude that this is a critical barrier to vendors to overcome. Data scientists, as such, are the ones most in need of persuasion and support.
IT Admins – IT Admins have the challenge of integrating Big Data analytics into the operational technology infrastructure. As such, Transforming data into suitable forms for analysis and Maintaining application performance as data volumes increase were seen as the critical barriers for IT Admins.
Perceptions of Non-Technology Barriers by Different Roles
- Application Developers – Application developers were concerned about how to make this all work in an uncertain decision-making and regulatory environment – Unsure of regulatory compliance implications, lack of executive buy-in, and disputes over project ownership are the key non-technology barriers according to Application developers.
- Business Analysts – Business Analysts tended to see the important non-technology barriers revolving around issues with data. They reported a higher rate of concern about Lack of data or inability to access data sources they need for their work as well as the Poor data and information quality they must rely on.
- Data Scientists – Data Scientists were more concerned than most with establishing the solution space and Selling the value to end users, Getting stakeholders to agree on data definitions, and the Difficulty operationalizing insights.
- Infrastructure Admins – Infrastructure Admins identified Getting stakeholders to agree on data definitions, as well as Poor data and information quality and Initial projects being too ambitious as being their most significant barriers – probably a function of their view of the effort looking much more disorganized and open-ended than the sorts of IT projects they were used to seeing.
Perceptions of Visualization Barriers by Different Roles
Visualization presents problems to all the roles – with the nuts and bolts of End user training on visualization tools and Integrating visualization into existing applications being the most significant barriers – much more so than responding to New types of visualization requests by end-users or Expanding visualization data or new types of delivery (i.e., mobility). End user training is seen as the most significant barrier for Application Developers, Business Users, and Infrastructure Admins.
Data Scientists and Business analysts are less concerned about End user training for visualization as a barrier. Data Scientists tend to see Making more data sources available for data visualization for end users and Responding to new types of data visualization requests from end users as being more critical concerns than other roles. Business analysts want More mobile data visualization tools for end users, which makes sense given their role in analyzing data whenever it’s necessary.
It wasn’t surprising to see Infrastructure Admins seeing mobile delivery with much less urgency – probably a function of the challenge IT already has in delivering even basic mobility services to users with IT standards and security guidelines. Mobility would only make matters worse.
Perceptions of Security Barriers by Different Job Roles
All the security barriers had relatively high levels of concern. However, Big Data Analytics security barriers resonated somewhat differently depending upon the role of the respondent. Infrastructure Admins had a higher level of concern than others about the Challenge of understanding/complying with security/privacy regulations but were a little less concerned about securing Big Data Analytics deployments from hackers and the lack of fine-grained security controls needed to support large, heterogeneous user populations.
Business analysts seem to be more concerned about Securing Big Data Analytics deployments from hackers – basically a concern about theft of the insights and conclusion of their work by competitors. Application developers have a higher level of concern about the Lack of fine-grained security controls needed to support large, heterogeneous user populations.
Perceptions of Compliance Barriers by Different Job Roles
All of the compliance barriers showed up as being important to Big Data Analytics practitioners. Application Developers were especially concerned about Monitoring and tracking compliance with data governance rules and Determining and assigning data ownership (as were Data Scientists), important responsibilities for their function. They were less concerned about Adapting governance-related rules and procedures to quickly changing end-user requirements and requests – and would probably prefer to see as little of this as possible.
Owing to their focus on the use of data for decision making, Business Analysts overall showed less concern about compliance issues overall and had especially lower concerns than others about Determining and assigning data ownership and Auditing user access and tracking data lineage.
Conclusions
Ascertaining the role and perspectives of the primary actors when selling into the Big Data Analytics market is critical, since the actors – even on small teams – see the barriers and issues from different perspectives. Depending on their role, they are likely to focus on different aspects of vendor offerings and what they mean for different facets of the project. Given that many enterprises are feeling their way through the maze of Big Data Analytics goals and objectives and tools and methods, successful partners will meet the decision makers where they are and tune their messaging appropriately to each. From the customer’s perspective, they should calibrate their concerns against the mainstream, as represented by the aggregates in this survey. Then they can understand their maturity and place along the customer journey in order to focus on technology and process improvements for successful deployments.
Methodology
Wikibon recently completed an extensive analysis of the results of a Big Data Analytic survey (n=300 web survey interviews in the US) focusing on current practice and barriers to successful deployments. of Big Data analytics projects. Big Data Analytics projects are those that (1) leverage non-traditional data management tools and technologies such as Hadoop, NoSQL, or MPP analytic databases and/or (2) involve the analysis of multi-structured and/or unstructured data such as clickstream, text, log file, and social media data. An example of such a use case would be the use of Hadoop to store, transform and analyze mobile sensor data. Big Data projects do not include projects involving the use of relational databases to analyze traditional structured data associated (e.g., CRM, ERP, Finance, etc.) Our Big Data clients have received a report of our initial conclusions.
This report digs more deeply into the many barriers inhibiting adoption of Big Data Analytics by focusing on the relative importance of these barriers as a function of the individual primary roles (Application Developer, Business Analyst, Business User, Data Scientist, and IT Administrator) of the actors in the process. Of course, each see the challenges of Big Data Analytics from different perspective and lenses. It’s important for vendors to understand the concerns of those in various roles in the adoption process. Moreover, at an early stage of technology adoption, many participants where multiple hats and have multiple responsibilities. Vendors should understand who they’re talking to and when as they position themselves and their products with Big Data Analytics users.
Definitions
Respondents classified themselves in terms of their roles in the Big Data Analytic process in their enterprise.
- Business user (i.e. A line-of-business professional who uses dashboards and other visualizations to understand data).
- Business analyst (i.e. A departmental power-user who conducts analysis of various data sets with tools such as Excel and SPSS.)
- Application developer (i.e. A developer who builds applications that leverage Big Data Analytics such as predictive models and algorithms.)
- Data scientist (i.e. An advanced analytics professional who conducts sophisticated analytics and develops predictive models/algorithms on large volumes of “messy” data.)
- Infrastructure Administrator (i.e. A data center professional who manages infrastructure/hardware associated with Hadoop and NoSQL database deployments that support Big Data Analytics projects.)
