Is VMware Building a DevOps Framework for the Masses?

Premise: While Cloud Native Applications and DevOps are generating massive amounts of hype, the ability of IT organizations to execute on this vision outside of Silicon Valley is often being questioned. VMware’s Cloud-Native Apps group is putting together an infrastructure framework that might just be the right model to bring DevOps interactions to the masses.

For the past 2-3 years, VMware has been going through a strategic transformation from a dominant player in a single industry sector (server virtualization) to a company that is engaged in a multi-front battle across multiple areas of disruption – public cloud, containers, software-led infrastructure, etc. Some of these battles, such as software-led infrastructure, are being driven by macro-economic shifts that are driving the cost curves of hardware and software to marginally different rates. This is fundamentally an infrastructure operations challenge. Other battles, such as public cloud, are being driven by a shift in the pace at which business groups need technology to accomplish their goals. This is fundamentally an application developer challenge. At the intersection of those challenges is a concept called DevOps, which is an emerging operational model where companies are trying to synchronize and harmonize the interactions between the Development teams and the Operations teams.

DevOps is not a new concept. It’s been around since the 2010-2011 timeframe and it has spawned a massive community that hosts global events such as VelocityConf and DevOpsDays on a frequent basis. Many of these were started by open-source software communities, so it took awhile before the attention reached the mainstream markets, whose businesses were less aligned to technology-centric product offerings.

If anyone were to attend enough of these events, one might be lead to believe that all future applications will be based on microservices, and all IT groups will merge into a team or sets of teams that practice DevOps principles of collaboration. But the reality is that this evolution faces two significant barriers:

1. The evolution of technical skills to understand automation, containers and operations in these new models will take many years to evolve and become widely available.
2. Most companies are not greenfield start-ups and they must manage the life-cycles and evolution of their existing application portfolio, while simultaneously planning for the Cloud Native applications that could differentiate their business in the future.

This evolution, which probably applies to 90-95% of all businesses, is why the Cloud-Native Apps announcements from VMware at VMworld 2015 have the potential to create a DevOps foundation that may apply to the masses much more than approaches driven by start-ups.

The elements in [Diagram 1] that are highlighted in blue show the components of the VMware Cloud-Native Apps architecture. Not all of these are new (NSX, VSAN, vSphere, vCloud Air, vRealize), and some have moved from pre-announced to generally-available (AppCatalyst, Code Stream). Of particular importance is the vSphere Integrated Containers and Photon Platform, which are actually made up of several technology elements:

vSphere Integrated Containers provides the infrastructure foundation for companies that have an existing installed base of VMware vSphere and associated management tools.  It does three important things:

1. Within an ESXi host, managed by vSphere/vCenter, it now includes technology that emulates all the speed of running containers on a native Linux machine. From a developer’s perspective, it operates exactly like any other Linux machine and Docker container would operate, from an API, packaging and deployment perspective. From an operations perspective, it’s an ESXi host that has a really fast version of Linux inside. This is using technology from Project Bonneville, Instant Clone and Photon OS.
2. Outside the ESXi host, it natively connects to both NSX and VSAN for networking and storage. Instead of introducing completely new technologies and products for those elements, the operations teams can use the native VMware tools that they already understand.
3. It clearly defines the demarcation point for Developers and IT Operations, while allowing Operations to provide a more container-friendly environment for the application containers. It removes the need for Developers to be hassled with the nuances of container management.

While VMware Integrated Containers will provide a bridge between existing VM-centric environments and Container-Enhanced environments, the overall VMware architecture will need additional enhancements and frameworks to move towards a Container-Centric model. This is where Photon Platform comes into the architecture. Just providing Development teams with a container-friendly host won’t be enough for the IT Operations team. They will need greater visibility and control of the overall environment to manage containers at large-scale. vSphere is not the right platform for this, as it has limitations on the number of hosts it can manage, and it conceptually looks at the world through VM-centric lenses. The Photon Platform provides two important elements:

1. Photon Machine delivers a lightweight, container-centric host platform to run containers. Instead of the full ESXi hypervisor, it uses a stripped-down “microvisor” (or ‘Just Enough Virtualization’). Photon OS is layered on this to deliver a container-centric host that is IT Operations friendly.
2. Photon Controller acts in a similar role to what vCenter does for VMs – centralized control, scheduling and resource management. It also provides native API interfaces to application-centric scheduling frameworks such as Kubernetes, Mesos, Cloud Foundry, OpenStack and others.

Photon Platform again provides a distinct line of demarcation between the tools, frameworks and APIs that Developers expect to consume, and the tools that IT Operations needs to maintain high levels of performance and availability for the overall environment.

It is these unique points of demarcation that will help many IT organizations more easily adopt not only container-centric environments, but allow clear areas for collaboration between Application Development and IT Operations teams. In addition to the areas of demarcation, VMware is making a strong push to natively integrate security technologies (Project Lightwave) into the overall architecture. The architecture also highlights that the on-going vendor battles will not necessarily be between VMs and Containers, but between the infrastructure platforms that can manage both, and the Cloud Native application platforms that are attracting developments.

The Basics of Cloud Native Infrastructures

For the past several months, Wikibon has been exploring the architectures of Cloud Native applications [here, here, here]. There are several characteristics that all of the Cloud Native architectures have in common:

1. The use of containers as the packaging and deployment model for the applications.
2. The need for container orchestration to manage larger-scale placement and deployment of containers.
3. The ability to interconnect containers, via dynamic networking and storage awareness, as well as discover new services that are added into the architecture.
4. The ability to apply various types of policies to not only the container infrastructure, but also the user-level policies related to the Cloud Native applications.

The reason that containers are at the core of each of these platforms is the nature of the applications. They are often designed to be stateless, and may have usage patterns that mandate more rapid upward and downward scaling than applications in the past. Containers allow the applications to start faster and be more consistently deployed. This creates a model called immutable infrastructure, where applications can declare the durability of their environments in a more granular way.  In essence, the Cloud Native applications can tell the infrastructure platform how many resources always need to be available.

At VMworld 2015, our panel discussion on the Evolution of Containers give insight into how this transition is happening from the perspective of hardware, operating systems and systems management.

For any technology company creating a strategy to deliver infrastructure technology that can support Cloud Native applications, it needs to meet all of these basic criteria. But the requirements extend well beyond those technology basics.

Aligning DevOps Organization to Business Organization

When thinking about these architectures in the context of DevOps, it’s important to understand that successful IT organizations not only align their platform capabilities to the needs of the business applications, but also to the skills of their organization (or outsourced partners). This is frequently referred to as “Conway’s Law“. This is often overlooked by new technology companies, as they often underestimate the challenges of organizational inertia and the costs of change.

One of the initial pieces of feedback that IT organizations often give when learning about DevOps is to ask, “Who does what function in that model?“. While this central question might be biased by a concern about an individuals long-term job security, it’s ultimately asking how the distinct functions of application development and operations are supposed to operate in a more modern and collaborative way. While the concept of a full-stack engineer is frequently discussed, someone who understands/builds/troubleshoots the entire stack from applications down to infrastructure, it’s a difficult reality to fulfill for most businesses. So the question remains for the bulk of companies – where can the lines be drawn to align the IT organization to this new DevOps model?

VMware’s Cloud-Native vision for DevOps

As VMware’s CEO Pat Gelsinger pointed out, most of the DevOps movement is being driven by infrastructure and operations teams that are trying to keep up with the increasing need for agility from application development teams. “The container trend becomes even more important than OpenStack in the future for developers, because it’s an application proposition.”

In order to be relevant in this this space, VMware needs to be able to bring value to several areas for the infrastructure and operations teams:

• VMware must create direct awareness and visibility within the technology of containers, such as Docker.
• VMware must create ways to provide secure environments for containers to run.
• VMware must make it simple to network containers together in a dynamic and automated way so that the infrastructure doesn’t become a bottleneck to application development teams.
• VMware must adapt their management tools to understand the concept of containers, and manage insight into these container infrastructures at the scale needed for Cloud Native applications.

Given the massive installed base that VMware has today, it must also be able to create an environment that will allow existing infrastructure and operations teams to be able to manage both VM-centric and Container-centric environments. They do not want to be forced to create separate bi-modal silos for legacy and modern applications.

This is where VMware’s Cloud-Native Apps vision has a unique advantage over many of its competitors.

Managing Open-Source and Commercial Offerings

While VMware has made significant progress over the past 12 months in understanding and embracing both containers and open-source software, it still has a ways to go to understand the balance between these two communities and types of users.

Announcements aren’t Code – Code wins: In the open source communities, code wins. They don’t tolerate early announcements and no software available for months to test it and provide feedback. VMware needs to improve their cadence between announcements and delivery. Right now, they are still treating it as a way to control buying patterns (pre-announcements) rather than driving innovation with the community.

To Open-Source or Not Open-Source – That is the Question: Some of the announcements were clear that the software will be made available as open-source. Other announcements were vague and hinted at “maybe”. Open-source communities succeed when roadmaps and plans are transparent and community members can easily provide feedback. VMware needs to do a better job of understanding that dynamic.

Eliminate the Friction: Trying to find information about the new VMware announcements and the actual software is complicated and frustrating. Some of it is on the VMware GitHub page. Some of it is on a project-specific landing page. Some of it is embedded within an ELA or some password-protected download page. Developers and Operators have many choices for Cloud Native applications, and they will reject friction if the next-best option is easier to find or consume.

Action Item: While not all elements of the VMware Cloud-Native Architecture are available today, VMware has laid out a compelling framework to help many companies move to an environment that not only preserves today’s skills and investments, but gives them a clear bridge to adopt containers and Cloud Native applications.  VMware customers that are being pressed by their business leaders to enable application faster should be aggressively looking at these new architectural enhancements and determine how they can get them integrated with their existing VMware environments.