Big Data Vendor Revenue and Market Forecast, 2013-2017

The Big Data market as measured by vendor revenue derived from sales of related hardware, software and services reached $18.6 billion in calendar year 2013. That represents a growth rate of 58% over the previous year.

Broken down by type, Big Data-related services revenue made up 40% of the total market, followed by hardware at 38% and software at 22%. Such a breakdown is due in part to the open source nature of much Big Data software and related business models of Big Data vendors, as well as the need for professional services to help enterprises identify Big Data uses cases, architect solutions and maintain performance.

Big Data Growth Drivers

Several important growth drivers fueled the Big Data market in 2013. They include:

• Both mega-IT-vendors and pure-play Big Data vendors took steps to better articulate their product & services roadmaps and larger visions for Big Data in the enterprise, creating greater confidence from enterprise buyers.
• The products and services related to Big Data continued to mature from a features perspective in 2013, further spurring adoption. These include the advent of YARN, which lays the foundation for Hadoop as a true multi-application framework, and the continued evolution of cloud-based Big Data services for large-scale analytics and application development.
• Big Data technologies also took important steps towards greater enterprise-grade capabilities in 2013, critical for mass enterprise adoption. These steps included better privacy, security and governance capabilities, as well as improved backup & recovery and high-availability for Hadoop specifically.
• Partnerships also played an important role in maturing the Big Data landscape in 2013. Of particular importance are a number of reseller agreements and technical partnerships between Big Data vendors and non-Big Data vendors, the results of which that make it easier for practitioners to adopt and integrate Big Data technologies.

Big Data Adoption Barriers

While the Big Data market experienced healthy growth in 2013 thanks to maturing technology and vendor support, barriers to adoption in the enterprise remain. While not an exhaustive list, these barriers include:

• A lack of best practices for integrating Big Data analytics into existing business processes and workflows.
• Concerns over security and data privacy in the wake of numerous high-profile data breaches and the ongoing NSA scandal.
• Continued “Big Data Washing” by legacy IT vendors leading to confusion among enterprise buyers and practitioners, as well as “political” factors that make it difficult for enterprise buyers to engage new vendors.
• A still volatile and fast developing market of competing Big Data vendors and, though to a lesser degree in 2013, competing technologies and frameworks.
• A lack of polished Big Data applications designed to solve specific business problems.

Big Data Vendor Revenue

As part of its market-sizing efforts, Wikibon tracked and/or modeled the 2013 Big Data revenue of more than 70 vendors. This list includes both Big Data pure-plays – those vendors that derive close to if not all their revenue from the sale of Big Data products and services – and vendors for whom Big Data sales is just one of multiple revenue streams.

The complete list is below:

Research Methodology

Regarding methodology, the Big Data market size, forecast, and related market-share data was determined based on extensive research of public revenue figures, media reports, interviews with vendors, venture capitalists and resellers regarding customer pipelines, product roadmaps, and feedback from the Wikibon community of IT practitioners.

Many vendors were not able or willing to provide exact figures regarding their Big Data revenue, and because many of the vendors are privately held, Wikibon had to triangulate many types of information to determine its final figures. We also held extensive discussions with former employees of Big Data companies to further calibrate our models.

Information types used to estimate revenue of private Big Data vendors included supply-side data collection, number of employees, number of customers, size of average customer engagement, amount of venture capital raised, and age of vendor.

Big Data Definitions

It is critically important to understand how Wikibon defines Big Data as it relates to the market size overall and to revenue estimates for specific vendors in particular. Wikibon’s definition of Big Data contains two equally important parts.

First, from a technology perspective, Wikibon defines Big Data as those data sets whose size, type, and speed-of-creation make them impractical to process and analyze with traditional database technologies and related tools in a cost- or time-effective way.

Second, Wikibon believes Big Data requires practitioners to embrace an exploratory and experimental mindset regarding data and analytics, one that replaces gut instinct with data-driven decision-making, and exchanges stubbornness for a willingness to question long-held assumptions. Projects whose processes are informed by this mindset meet Wikibon’s definition of Big Data, even in cases where some of the tools and technology involved may not.

Based on the above definition, Wikibon includes the following products and services under the umbrella of Big Data:

• Hadoop software and related hardware and services;
• NoSQL database software and related hardware and services;
• Next-generation data warehouses/analytic database software and related hardware and services;
• Non-Hadoop Big Data platforms, software, and related hardware and services;
• In-memory – both DRAM and flash – databases as applied to Big Data workloads;
• Data integration and data quality platforms, tools and services as applied to Big Data deployments;
• Advanced analytics and data science platforms, tools and services;
• Application development platforms, tools and services as applied to Big Data use cases;
• Business intelligence and data visualization platforms, tools and services as applied to Big Data use cases;
• Analytic and transactional applications and services as applied to Big Data use cases;
• Cloud-based Big Data services including infrastructure, platform and software delivers as a service.
• Other Big Data support, training, and professional services.

2013 Big Data Market Highlights and Trends

2013 was an important year in the evolution of Big Data technology. The concept of Hadoop-based Big Data analytics and applications moving beyond MapReduce-style batch analytics existed before 2013, but this was the year that the structural foundation to such a transition was laid in the form of YARN.

YARN, or Yet Another Resource Negotiator, has been in the works for more than three years and made its official debut in October 2013 as part of Hadoop 2.0. While the technical architecture of YARN is outside the purview of this report, the important point is that YARN enables Hadoop to operate as a true multi-application framework. Developers now have the structural underpinnings to build real-time and streaming data applications, interactive SQL-style query applications, graph analytic apps, and more.

YARN is critical to the future of Hadoop. It ensures that Hadoop will not be relegated to backroom data science projects but will take a prominent (and potentially starring) role in the modern data architecture.

YARN was an indirect growth driver for the Big Data market in 2013. As stated above, in 2013 vendors began to crystalize their visions for Big Data in the enterprise. The pending arrival of YARN, among other technology advances, enabled vendors to credibly position Hadoop at the center of their Big Data plans.

Complimenting YARN were a number of moves by Hadoop and non-Hadoop vendors to better integrate the open source Big Data framework with existing data management infrastructure and legacy databases. These included:

• Cloudera’s Impala, Search and its Enterprise Data Hub;
• Hortonworks’ technical partnerships and reseller agreements with Teradata, Microsoft and SAP;
• HP’s HAVEn reference architecture and Vertica’s new FlexZone feature;
• Pivotal’s HAWQ and Data Dispatch offerings for Hadoop;
• IBM’s BigSQL feature and BLU Acceleration release;
• Microsoft’s PolyBase data-processing framework.

While each of these releases and features is still relatively immature, they served to bolster confidence in Hadoop and related Big Data technologies as a core part of the modern data architecture. This confidence translated into significant investment by Fortune 1000 enterprises in 2013, though the fruits of these investments won’t be enjoyed until 2014 and beyond. Other key developments in the Big Data market in 2013 included:

• Market leader IBM, while slowing in overall Big Data revenue growth, did not sit on its lead. The company released several important Big Data-related products and services in 2013, including PureData System for Hadoop, BLU Acceleration, BigSQL and the Watson Developer Cloud. The company also acquired SoftLayer, which will play an important role in IBM’s Big Data from the cloud strategy.
• From the number two positions in terms of Big Data revenue, HP likewise took important steps to improve its Big Data portfolio in 2013, particularly its software portfolio. Namely,the company unveiled HAVEn, a modular reference architecture designed to integrate (and make more consumable to enterprise customers) Hadoop, Vertica, Autonomy, enterprise security and application development.
• Professional services made up the largest segment of the Big Data market in 2013, with IBM, Accenture and HP leading in total professional services revenue for the year. This is not unexpected. While Big Data technologies continued rapid maturation in 2013 (as explained above), enterprise practitioners still require significant help from professional services organizations to identify use cases, design and deploy systems, and integrate the technology and output (read: analytic insights) into business processes and workflows.
• Big Data and the cloud continued their courtship in 2013. Amazon Web Services releasedKinesis, a streaming data framework for real-time applications, and RedShift, its large-scale data warehousing service. Meanwhile, Hortonworks and Microsoft released HDInsights, which delivers HDP on Microsoft’s Azure cloud. It is still very early days for Big Data in the cloud, however, with the vast majority of current use cases focused on test-and-development, not production deployments. The future for Big Data deployments is clearly hybrid, with cloud and on-premise deployments living (hopefully) in harmony.
• There were a number of important improvements to Hadoop’s enterprise-grade capabilities around security, privacy and reliability in 2013. These include WANdisco’s Non-Stop NameNode for high availability that removes Hadoop’s single-point-of-failure, IBM’s InfoSphere Data Privacy for Hadoop offering that de-identifies sensitive data in Hadoop, and Sqrrl Enterprise that enables cell-level security in Hadoop via Accumulo.
• 2013 saw a handful of new entrants to the Big Data market. Of the most significant werePivotal, the EMC/VMware spin-off that is dedicated to building a data-centric platform for agile application development; Intel, which debuted its Hadoop distribution to the U.S. market, with its focus on security and performance improvements at the hardware level; and General Electric (of all companies), which released a new Hadoop-based database and a slew of related applications to help its customers make sense of data streaming off industrial equipment.
• Speaking of General Electric, the concept of the Industrial Internet came to the fore in 2013. In addition to analyzing data from individual pieces of equipment, the larger goal of an Industrial Internet (sometimes called the Internet of Things) is to orchestrate multiple industrial applications to work intelligently in order to optimize entire operational environments. Industrial Internet will play an increasingly important role in the world of Big Data analytics.
• Data visualization continued its hot streak in 2013. Tableau Software went public in 2013and racked up impressive customer wins, revenue growth and partnership announcements in its first few months as a publicly traded company.

Wikibon’s Big Data Market Forecast

Wikibon forecasts Big Data market growth to slow slightly in 2014 to 53%, reaching $28.5 billion for the year. Looking ahead, the Big Data market is currently on pace to top $50 billion in 2017, which translates to a 38% compound annual growth rate over the six year period from 2011, the first year Wikibon sized the Big Data market, to 2017.

As the market matures through 2017 and beyond, Wikibon expects Big Data applications and cloud-based services to play an increasingly important role. As the underlying infrastructure solidifies, Wikibon believes mainstream and late-adopters will look to service providers to deliver polished applications and services that sit on top the hardened Big Data infrastructure and target specific, high-value business challenges.

While Wikibon believes over the long term Big Data practitioners will generate significantly more value than Big Data vendors, there is significant opportunity for those vendors that can deliver Big Data solutions that speak to business rather than technical value. This is still a work-in-progress for many vendors, despite progress that was made in 2013.

Finally, Wikibon believes the biggest growth inhibitors for the Big Data market are security and privacy concerns. The NSA revelations clearly illustrate that data security and privacy are hot button topics for both the American and international public. This, despite the public’s relative lack of understanding about just how much personal data is available on the web and how it often unwittingly provides this data with the click of a button. The current ire directed at the NSA is likely to turn its attention to the commercial sector in 2014, as the public comes to better understand how social networks, retailers, banks and other businesses are using its data. As Wikibon has urged before, it is critical for the industry to be proactive and address these concerns sooner rather than later. It makes good business sense to do so, but is also the responsible thing to do.

Below is Wikibon’s Big Data market forecast broken down by market component through 2017.

Action Item: The value of Big Data is in its potential to help practitioners make better strategic and tactical decisions, run more streamlined and efficient organizations, and deliver better products and services to customers. Vendors would be wise to remember that it is such business value, not technology features per se, that will drive revenue in this market. In order to propel the Big Data market forward and entice early mainstream adopters, Big Data vendors must align not just their marketing messages but product roadmaps to this reality.