Will 3D XPoint make it against 3D NAND?

Premise

Wikibon projects that 3D XPoint will not make it in volume in either the enterprise or consumer markets; it may reside in some specialist application niches. The areas of uniqueness claimed for 3D XPoint will be addressed by DRAM, 3D NAND, and innovative non-volatile combinations of DRAM and 3D NAND.

Claims and Analysis of 3D XPoint

Introduction

3D XPoint (pronounced CrossPoint) was announced by Intel & Micron in 2015. The vendors claimed 3D XPoint is 1,000 times faster, and 1,000 time more durable than NAND technology, and 10 times denser than DRAM (see Figure 1 below).

The announcement created great interest in system vendor, enterprises, academia and IT practitioners.

Previous Wikibon Research on 3D XPoint

In July 2015, Wikibon published  research titled “Intel and Micron take the Plunge with Non-volatile 3D XP“, which advised practitioners to wait and see.

Wikibon has consistently pointed out that unless 3D XPoint can establish itself in the consumer market, it is very unlikely to ship sufficient volume in the enterprise memory market to reach critical mass in the market to justify future investments. At the end of 2016, Wikibon research entitled “Wikibon 2017 Disruptive Infrastructure Predictions” projected that 3D XPoint would stall in 2017.

Analysis of 3D XPoint Technology Claims

Figure 1 shows the initial claims made 3D XPoint at the April 2016 Intel developer forum in Shenzhen. There was no explanation of what kinds of NAND and DRAM were being referred to.

The claim of 1,000 faster is probably a translation of the fact that 3D XPoint is byte addressable, while NAND storage is accessed in blocks. The reality is that byte addressable storage would be very expensive to build because of the additional three-dimensional wiring; no 3D XPoint technology has been built or demonstrated with byte addressability in a DIMM form factor.

The 1,000 times more durable claim was probably due to over-pessimistic evaluation of 3D NAND technologies. A few years ago a chorus of engineers from outside of NAND (especially engineers working on new memory projects and magnetic disk drive manufacturers) were declaring that new NAND technologies would not be durable, and the end of NAND was nigh. As is often the case, they underestimated the ingenuity of engineers, underestimated the investments that would be made in NAND, and underestimate the size and growth of the consumer NAND market.

The DRAM and NAND markets are very well established markets requiring high fabrication investments, with initial new technology deployments and most volume shipping to the consumer markets.  Before 3D XPoint there have been a number of memory technologies and architectures have been announced with similar acclaim. They include RDRAM (Rambus Dynamic), MRAM (magnetoresistive), ReRAM (resistive), PCM (phase-change memory), and HMC (hybrid memory cube). None have successfully challenged DRAM for system memory or NAND flash for persistent storage.

Bottom Line: There is little room for new technologies break into these established markets. There is practically no room to break into them with an enterprise-only technology such as 3D XPoint.

Detailed Technical Comparison between 3D XPoint and 3D NAND

Comparing 3D Consumer Storage Technologies

For 3D XPoint to have any chance of thriving, it has to offer unique characteristics that are important to buyers, and will justify a higher price point before volume shipment. Both 3D XPoint and 48-layer/64-layer 3D NAND technologies are only shipping to consumers in the 1H 2017. The methodology for comparing the two technologies in the real-world is therefore constrained to consumer products. The comparison methodology is to compare the technical specifications for each product, and the specific use case for each product.

The two comparison products chosen by Wikibon are the Intel 32GB Optane Series PCIe M.2 Memory Module and the Toshiba XG5 M.2 2280 SSD. There are detailed specifications available for both products.

• Toshiba is chosen because it has the first demonstrable product with 64-layers and deploying TLC technology, and the densest technology. Similar results are likely found with 64-layer Western Digital TLC technology, 48-layer Samsung technology, future 72-layer SK Hynix TLC technology, and similar technologies from Micron/Intel; Wikibon will report on them as real products are announced.
• Intel 32GB Optane Series PCIe M.2 Memory Module was chosen because it is the only technology with full specifications and able to be ordered or pre-ordered. It is a component of Intel’s Memory Drive Technology
  • It is fundamentally a paging mechanism, which enables the DRAM and the Intel Optane SSD to combine to emulate a single virtual, volatile memory pool. There are a number of alternative paging systems available that provided virtual paging solutions.
  • The Optane module appears to the system like DRAM to the OS and applications. Physically, it is attached to the systems as a SSD storage unit using NVMe on the PCIe Bus, behind Intel software and firmware in latest generation Intel chips. It is not a retrofit to increase the for previous generations of intel processors and HDDs).

Intel 3D XPoint Optane vs. Toshiba 3D TLC NAND

Table 1 below shows a detailed technical comparison between the Toshiba XG5 M.2 2280 consumer form factor SSD, based on 3D NAND 64-layer TLC technology, and the Intel 3D XPoint M.2 2280 form-factor (Optane) consumer SSD technology. The 3D XPoint M.2 technology is the only Intel Technology that is shipping at the moment. There are examples of 48-layer 3D MLC NAND SSDs that are shipping in volume (e.g., Samsung 960 Pro SSD)

As discussed earlier, the initial claims for PCM-based 3D XPoint technology are 1,000 times faster than NAND, and 1,000 time more reliable. As can be seen in Table 1, in the real-world of storage, it is at best 2-4 times faster for some applications requiring faster IO, and already matched in bandwidth.

In terms of reliability, the mean time to failure (MTTF) metrics are about the same. The lifetime writes endurance rating  for 3D XPoint is 7 times higher than for 3D NAND.  However, endurance can always be improved by using more storage! Table 1 shows that if a very active 32GB segment of 3D NAND requires a lifetime write of 182.5 Terabytes (same as Optane), this can be achieved by allocating 182.5 Terabytes of 3D NAND.  This is 23% of the 1TB of 3D NAND. This still leaves 77% (618 TB) of lifetime writes for the remainder of the storage, more than sufficient for most environments.

The NAND manufactures have focused significant engineering resource on NAND durability and reliability, driven by the enormous volume of solid state storage solutions for mobile, tablets, laptops and PCs to meet the practical needs of consumers. In particular, the use of sophisticated algorithms and additional bits for preserving data integrity, and the development of specific ARM processors in the SSD controllers have led to efficient use of storage and high levels of availability. These capabilities can be developed for both 3D XP and 3D NAND; the overwhelming volume in favor of 3D NAND means that durability will not be a significant differentiator for 3D XPoint.

Bottom Line: There is little differentiation between 3D XPoint and 3D NAND for use as storage. Serial performance is a wash. Endurance is a wash. The potential byte access is not relevant to application performance except in some edge areas; average IO block sizes have increased over time and are now 32K. The price difference is overwhelmingly in favor of 3D NAND, and will increase as vendor are investment in 3D NAND fabs and research turns into very high volume later in 2017.

3D XPoint as a Memory Device

3D XPoint as a replacement for DRAM

Wikibon is not aware of any ongoing project for 3D XPoint in place of DRAM memory with any vendor. Both 3D NAND and 3D XPoint from a PCIe connection are still much, much slower than DRAM; microseconds vs nanoseconds. If there was going to be any place where 3D XPoint might be used, it would be with entry-level ARM processors such as a Raspberry Pi, and used in low-cost consumer storage and networking.

In theory, there is a potential use of 3D XPoint (or other persistent technologies) in ultra-small complete devices with no external power supply. These devices might be used, for example, medical applications such as blood stream monitoring. Wikibon is not aware of any research projects in this area, and if viable would require a decade of development.

The Wikibon projections for enterprise NAND flash and 3D NAND flash can be found in previous research entitled “Wikibon Flash Enterprise Adoption Projections“.

Bottom Line: The volumes for both performance and capacity 3D NAND to replace HDDs are growing very fast, and 3D XPoint cannot compete on price.

System Class Memory: Intel 3D XPoint SSD 

The Intel M.2 2280 consumer form-factor implementation is a hybrid setup, using Intel 3D XPoint Optane Memory as a 16 or 32GB cache in front of magnetic hard disks. This is offered on the latest generation x86 chips, with PC support on 64 bit Windows 10 only.

This offers some potential benefit when portions of the operating system are held in cache and are accessed faster. It will offer speed-up of frequently accessed small files. However, this hybrid setup will offer little or no performance benefit to infrequently accessed, large-file data or new data. Large file copies, application installations, gaming, video editing, and similar random or large file workloads will most times perform on the hybrid much like a standard HDD.  As Wikibon has discussed before, the problem of HDDs is the “long tail effect” (the impact of the occasional very long waits for IO), which has a very negative effect of application performance, particularly databases. 

Intel  has also announced Intel Optane SSD DC P4800X, a 375 GB SSD on a PCIe card, without shipment dates. This is also part of Intel’s Memory Drive Technology, which makes the SSD appear like DRAM to the OS and applications.

Bottom Line: System Class Memories will use Hybrid DIMMS (see next section).

System Class Memory: 3D XP or 3D NAND Hybrid DIMMs 

For functions which need non-volatility, small buffers and caches can deploy DRAM with capacitor protection, allowing time to transfer the data to a non-volatile media following failure. These are used extensively in write caches in SAN controllers.

An early example of hybrid DIMMs technology is Diablo’s hybrid DIMM device, which creates a NAND storage SATA drive behind DRAM on a DIMM format.

A recent example of a non-volatile memory is the Netlist’s HybriDIMM (hybrid of DRAM & 3D NAND), based on the industry-standard DDR4 LRDIMM interface. Netlist claims this will operate in current Intel x86 servers without BIOS or application change.

The issue of endurance for 3D NAND, as stated before, can be overcome by more NAND storage, at an overall lower cost and higher density.

Bottom line: Hybrid DIMMs with slightly more DRAM and denser and much larger 3D NAND will be cheaper and perform better than Hybrid DIMMs using 3D XPoint for almost all applications.

The Changing Market with 3D NAND

3D NAND is Decimating the HDD Market

The whole market for consumer magnetic HDD is falling fast, with only enterprise HDD maintaining volume. The additional cost of power (~15 Watts) and weight (400 grams) and volume of HDDs makes this technology unacceptable for mobile, tablets, and laptops, and even desktop PC are also seeing a strong push to NAND flash.

The impact of 3D NAND flash with low latency and high performance will increase the amount of IO that travels from storage to the processor. This has the potential to increase the utilization from about 10% to 15%-20%. Larger DRAM sizes will be required, but the overall system cost will be much lower. Larger DRAM is far more cost effective than much slower NAND or 3D XPoint caches for almost all applications, as the processors are still an expensive part.

The rapid advances of 3D NAND and DRAM densities driven by the insatiable demands of the consumer market, the wholesale collapse of the magnetic hard disk market (HDD) in the consumer area, and the lack of adoption of 3D XPoint in the all-important consumer area has led to 3D XPoint being unable to keep pace in the consumer world. The lack of volume, lack of clear performance advantage and high cost (Table 1 shows Intel Optane is 6 times more expensive that 3D NAND) is inhibiting the adoption of 3D XPoint in the enterprise. 3D NAND is set to become denser, higher performing and lower cost as consumer and enterprise volumes increase.

3D NAND Enables “UniGrid” Architecture

The good news is that the groundbreaking work that Intel and SNIA drove in creating great NVMe and NVMe over Fabric specifications for 3D XPoint has provided the basis for much faster computers based on multiple processor nodes, large DRAM, and 3D NAND interconnected with very fast 100GbE networks. Micron recently announced such an architecture, call SolidScale, using technologies from Excelero for NVMe over Fabric, Mellanox for offloading 100GbE networks, HP servers, and Micron’s own SSDs.

Wikibon is coining the work “UniGrid” to describe this architectural approach. UniGrid allows very fast access to data, and very little difference in access time between data attached to a processor or any other processor in the UniGrid. This allows applications to access any data in the UniGrid with uniform latency.

Unigrid is based on multiple processor nodes with large NVMe addressed 3D NAND storage, large DRAM or hybrid NVDRAM/3DNAND, and fast 3D NAND storage.  The 3D NAND storage uses NVMe over Fabric to interconnected with very fast 100GbE networks (mainly from Mellanox) using RDMA and other protocols. The key to this UniGrid architecture is that access to any node’s data, locally (same processor) or remotely, can be accessed with only a 5 microsecond difference between local and remote.

The UniGrid architecture will allow multiple applications to run on sub-clusters of processor nodes, and for data to be shared between each other in real-time (e.g., systems of record accessing big data fraud calculations in real time).

Wikibon believes is set to revolutionize the next generation of applications. Future Wikibon research will cover this architecture in detail.

3D XPoint Conclusions

3D XPoint was announced by Intel & Micron with pomp and fanfare in 2015, claiming that it was 1,000 times faster, and 1,000 time more durable than NAND technology, and 10 times denser than DRAM. Today, with the delivery of 3D 64-layer TLC NAND technologies, those original claims are pipe dreams. The Wikibon research determines that 3D XPoint is struggling to stay ahead of 3D NAND for performance and endurance, and failing to compete with DRAM. 3D XPoint is failing to compete in volume or price against 3D NAND. The future looks even bleaker for XPoint, as all the NAND vendors focus on new fabs and research, and 3D NAND volumes explode in late 2017.

The announcement of 3D XPoint was a calculated risk by Intel and Micron. The business bet was that 3D NAND capability and volumes would be slower. Its failure is a sadness to most lovers of technology, including the author. Wikibon expects that Intel and Micron will significantly reduce investments in 3D XPoint technologies.

Intel and SNIA did excellent work in setting up great “NVMe” and “NVMe over Fabric specifications”. These were originally set up for 3D XPoint, but will work just as well with 3D NAND. These standards have provided the basis for much faster computer architectures, named as “UniGrid” architectures by Wikibon.

Bottom Line: Wikibon projects that 3D XPoint will not make it to volume production in either the enterprise or consumer markets; it may reside in some specialist application niches such as wall street arbitrage and some military uses. The areas of uniqueness claimed for 3D XPoint will be addressed adequately and at much lower cost by DRAM, 3D NAND, and innovative non-volatile combinations of both DRAM and 3D NAND.

ACTION ITEM

CIOs and CTOs should assume 3D XPoint will NOT be important to the enterprise, and not evaluate it further. The focus should be on deployment of 3D NAND technologies, and expect a rapid cadence of increases in performance, density, form factor, and volume.  Hybrid DIMMs with DRAM and 3D NAND should also be evaluated as the technologies mature, as well as UniGrid Architectures.