Brought to you by:
Enterprise Strategy Group | Getting to the Bigger Truth™
ESG WHITE PAPER
Intel, Lightbits Labs, and VMware Accelerate Private Clouds
A High-performance Solution for Virtualized Environments Leveraging NVMeTM/TCP
By Scott Sinclair, ESG Senior Analyst and Monya Keane, ESG Senior Research Analyst
OCTOBER 2021
Introduction
For a modern business, competitive success requires scalable, highly efficient IT operations. Nine out of ten IT organizations surveyed by ESG report that they now have to move faster than three years ago, with 41% accelerating their response speed by more than 50%.1
Complexity, however, is getting in IT’s way. Sixty-four percent of ESG research respondents believe that the complexity of their current infrastructures is slowing ongoing operations and strategic digital initiatives. A big reason for this ever-mounting level of complexity is the rapid growth of both data and infrastructure components. It’s a tough situation: IT organizations are struggling to modernize their IT infrastructures at the same time they are trying to simplify operational management activities enough to allow them to react as fast as they need to.
An essential part of IT modernization (and IT simplification) involves eliminating traditional infrastructure silos and transitioning to a more cloud-like, scalable infrastructure. Such environments often also leverage a distributed architecture that can reduce TCO.
One excellent way to accomplish IT modernization/simplification efforts is to use the specification known as non-volatile memory express (NVM Express™ or NVMe™) on top of the TCP/IP software stack. Using NVMe over TCP (NVMe/TCP) supports efficient, streamlined block storage optimized for today’s multi-core application servers. It maximizes the value of high-performance NVMe-based flash storage systems by extending the low-latency benefits of NVMe across the network to the application. Leveraging standard TCP transport simplifies network infrastructure management.
At this point, NVMe over TCP is an industry standard with a well-established ecosystem. That is fortunate because it ensures vendor interoperability and provides benefits such as a more consistent experience, better toolchains (programming tool sets), and overall, more options for storage buyers.
Recently, VMware announced support for NVMe over TCP with VMware vSphere(R) 7 Update 3, elevating TCP to the upper tier of options to provide NVMe over Fabrics capability. Deployment is a simple, standard migration path using VMware storage vMotion, and ongoing consumption is easy and nondisruptive.
Intel® Lightbits Labs® (Lightbits®), and VMware, have collaborated to deliver a software-defined storage solution that can further accelerate application performance, simplify infrastructure management, and reduce TCO. Lightbits software-defined storage is optimized for 3rd Generation Intel Xeon® Scalable processors, Intel Optane™, and Intel Ethernet. It offers performance, availability, resilience, and scalability for VMware vSphere while helping to lower costs.
The Transformational Value of NVMe over TCP
ESG research validates the value of NVMe, including NVMe over Fabrics (NVMe-oF), in overcoming the high-level challenges seen in on-premises data center storage environments. On average, organizations expect their on-premises data environments to grow 35% annually in capacity, and they are on pace to double capacity in less than three years.
But as environments scale, silos can proliferate—or at least management, optimization, and automation of data placement on the right storage tier or media can become a problem. That is one of several storage challenges in block storage environments, alongside the cost of the infrastructure (cited by 26% of respondents), difficulties in meeting performance challenges (cited by 25%), and trouble keeping pace with the rapid growth rate of data (cited by 24%). These businesses need highly scalable, highly available, high-bandwidth, low-latency storage disaggregation over the Ethernet network.
The rise of NVMe-based storage and NVMe-oF technology are central elements of modernization to ensure that storage environments can keep pace with the growing need for high-performance data access. The NVMe protocol was designed for high-performance PCIe solid-state drives. It provides a higher level of performance and lower latencies than traditional controllers such as SAS or SATA offer.
According to ESG research, 28% of IT organizations already use NVMe-over Fabrics, and 42% are poised to deploy NVMe-over Fabrics technology in the next 12 months. NVMe-oF support has emerged to support a hyperscale, remote pool of SSDs extending the low-latency benefits of NVMe across the network.
Figure 1 depicts the benefits that organizations have realized by adopting NVMe technology—not just performance benefits for existing and future workloads, but also NVMe’s ability to improve utilization and reduce both capital and operational costs.
Figure 1. Top Benefits of NVMe Adoption
Which of the following benefits has your organization realized as the result of deploying on-premises NVMe-based flash storage technology? (Percent of respondents, N=119, multiple responses accepted)
Source: Enterprise Strategy Group
VMware has added support for NVMe over TCP with vSphere 7 Update 3, and now IT organizations should start considering NVMe over TCP for their virtualized environments.
Intel, Lightbits, and VMware Collaborate to Transform the Data Center
For their part, Intel and VMware partner Lightbits are bringing to on-premises data centers a storage architecture similar to what the big cloud providers have implemented. The intent is to simplify management while improving scalability in both performance and capacity—all leveraging an SDS-based design. An SDS architecture simplifies an IT environment by eliminating silos, improving scalability, and reducing the overall cost of storage.
Lightbits and its partners have a good sense of timing in regard to this effort. ESG research shows that 59% of respondents expect to accelerate their spending on SDS technology over the next 24 months. When ESG asked respondents to identify the top benefits their organizations realized, or expected to realize, through SDS adoption, simplicity for both storage and hybrid cloud environments, along with greater flexibility and the ability to reduce/control costs, were common themes (see Figure 2).
Figure 2. Top Benefits of Software-defined Storage
What benefits has your organization realized, or does it expect to realize, as a result of deploying software-defined storage technology? (Percent of respondents, N=308, multiple responses accepted)
Source: Enterprise Strategy Group
Intel, Lightbits, and VMware Deliver NVMe/TCP-based, Scale-out SDS Storage for VM Environments
Lightbits is a storage technology innovator offering NVMe-optimized, software-defined elastic block storage. They are the creators of NVMe/TCP standardization and ecosystem development. The Lightbits scale-out storage solution is built on NVMe over TCP and is designed to be easy to deploy at scale and provide application performance similar to local flash (see Figure 3).
Figure 3. Lightbits Topology Diagram
Source: Lightbits Labs
Several Intel-based innovations coexist within Lightbits technology to modernize and accelerate data center environments. Specifically, Lightbits leverages:
• Intel® Xeon® Scalable Processors—delivering higher platform performance for storage software and data services and efficient utilization of more cores.
• Intel® Optane™ technology for high-performance storage—the fast, non-volatile write buffer and large memory capacities lower TCO without requiring costly battery technology.
• Intel® Ethernet 800 Series with ADQ Technology—delivering high performance for low-latency NVMe over TCP.
Benefits of the Collaboration
The benefits of the collaboration among Intel, Lightbits, and VMware are clear. As a VMware Technology Alliance Partner and Day 0 design partner, Intel and Lightbits collaborated extensively with VMware on development and testing of the new vSphere NVMe/TCP feature. Lightbits’ LightOS fully optimized on Intel’s platform breadth of high-performance technologies for the data center such as Intel Xeon Scalable processors, Intel Optane Persistent memory, and Intel Ethernet 800 Series Network Adapters is now a VMware certified Day 0 software-defined storage solution for VMware vSphere 7 Update 3 release. Lightbits’ storage solution, running on Intel technologies, is also integrated into VMware vCenter Server(R), which simplifies management. Using VMware vSphere ® vMotion(R), organizations can easily migrate their existing VMs to the Lightbits environment, thus simplifying IT modernization efforts. Organizations with private clouds and hybrid clouds, as well as cloud service providers can now realize the performance, scalability, and cost-efficiency benefits of a combined solution from Intel, Lightbits, and VMware.
The combined Intel, Lightbits, and VMware solution delivers performance, scalability, and cost-efficiency benefits for virtualized environments. IT can quickly deploy and scale an SDS-based storage environment optimized to meet the needs of a VMware private cloud environment. The combined solution is:
• Simple to configure and deploy: It requires no changes to the network, and it fits seamlessly into the VMware environment with vCenter integration for single-pane-of-glass management.
• Flexible and scalable: IT can scale compute and storage independently—it is easy to deploy over standard TCP/IP networks as well.
• High performing: With Intel Xeon Scalable processors with Intel DL Boost, Intel Optane PMem, and Intel Ethernet 800 Series Network Adapters with ADQ technology, LightOS delivers high-IOPS, low-latency, NVMe-based architecture with line rate compression configurable per volume.
• Designed to reduce total cost of ownership: Coupling Intel’s high-performance hardware platform with LightOS optimized for low-cost QLC SSDs delivers a cost-efficient storage solution. The solution also assists with data reduction, as all volumes are thin provisioned. Additionally, no hypervisor is required on storage nodes.
• Highly available: It comes with individual SSD-level Elastic RAID and per-volume replication.
• Able to support essential data services: It offers volume snapshots (read only) and clones (read/write), all thin provisioned.
The integration of LightOS with Intel technology, especially for VMware environments, provides private-cloud infrastructure and cloud service providers customers with an optimized cost-efficient, lower TCO storage solution while maintaining high performance and very low latency.
Evaluating the Performance of the Intel, Lightbits, and VMware Solution with NVMe over TCP
To show the benefits and performance of using vSphere together with LightOS and NVMe/TCP, Lightbits and Intel collaborated to test a sample set-up and then shared the results with ESG. This test, however, was not conducted by the ESG Technical Validation team.
For this performance test, Lightbits and Intel leveraged the following configuration.
• 8 servers (“clients”) running VMware ESXi 7.0.3 build-18321446.
• Each ESXi server is equipped with dual Intel Xeon Platinum 8280L CPUs, 384GB of RAM, and Intel(R) XXV710 dual port 25GbE network adapter.
• 3 servers running CentOS 7.9.2009 with kernel 4.14.216, running LightOS (LOS) 2.3.8.RC1, creating a cluster that provides storage for all the clients.
• Each LightOS server (LOS) equipped with dual Intel Xeon® Gold 6338 CPU, 256GB of RAM, 16*128GB Intel® Optane™ Persistent Memory 200 Series, 16x 3.84GB Intel P5510 NVMe SSDs, and Intel E810 dual port 100GbE network adapter.
• Each ESXi server ran 8 VMs running CentOS Stream release 8 (kernel 4.18.0-338.el8.x86_64), with 12 CPUs, 32GB of RAM, and a 16GB hard disk to run the OS.
• Each VM had its own dedicated datastore created from a single 600GB LightOS volume, each datastore held 4 VMDKs size 140GB, and each VMDK was connected to the corresponding VM via a separate VMware Paravirtual SCSI controller.
For the test, Lightbits and Intel used fio (fio-3.19) on each VM to run various patterns of workloads using various block sizes. The chart in Figure 4 presents the number of I/O operations (IOPS) the solution was able to push from all the clients (ESXi servers).
Figure 4. Total IOPs by Block Size/Read Write Mix
Source: Lightbits Labs
The test results presented by Lightbits and Intel reveal that with 8k block size, the LightOS cluster delivered close to 2 million combined IOPS (70% read operations and 30% write operations). As the block size was multiplied, the number of I/O operations reduced accordingly.
The data in Figure 5 presents the read/write total bandwidth (throughput) that the tests recorded going to/from the LightOS cluster. With the larger block sizes (64k and above), the Lightbits and Intel team found that the LightOS cluster saturated the 2x100GbE connections. With smaller block sizes (8k), the expectation is that the addition of more ESXi servers connected to the LightOS cluster would be able to scale bandwidth further, suggesting that the LightOS cluster offers room to support additional scaling of ESXi servers at lower block sizes.
Figure 5. Total Bandwidth (in Gb/s) by Block Size and Read/Write Mix
Source: Lightbits Labs
These tests highlight impressive performance results using NVMe over TCP. While it is important to further evaluate solutions to ensure that it will meet with requirements of your specific application environment, these results support the assertion that NVMe over TCP needs to be a consideration when seeking to access the performance benefits of NVMe over the storage network.
LightOS vCenter Plugin
To simplify the management of virtualized environments, the vCenter LightOS plugin (shown in Figure 6) allows organizations to control LightOS clusters, create volumes, assign them to ESXi servers, upgrade LightOS, monitor cluster performance, and view alerts all from the known environment of VMware’s vCenter. The plugin allows a single vCenter instance to connect to several LightOS clusters.
Figure 6. LightOS vCenter Plugin
Source: Lightbits Labs
The Bigger Truth
NVMe over TCP is a fast interface. But if you put a fast interface on a slow storage system, it won’t help. That’s not the case here. The Lightbits storage system is designed to support high performance and low latency by design. This is a full-blown storage system that shines with high performance.
And that is going to be a real bonus to organizations that need to simplify IT, even as they modernize IT. NVMe over TCP absolutely needs to be part of their consideration criteria for their VMware environments. After all, VMware is expanding its NVMe over Fabrics capabilities by supporting this transport protocol because this protocol is important. NVMe over TCP is feeding a real market need.
Furthermore, Lightbits with Intel needs to be a storage consideration for organizations seeking more flexibility, ease, and cost control. This is all about low latency that is good enough for business-critical workloads and enterprise-level VMware environments. That’s a huge win.
NVMe over TCP
DOWNLOAD PDF REPORT
This ESG White Paper was commissioned by Lightbits and Intel and is distributed under license from ESG.
1 Source: ESG Research Master Survey Results, 2021 Data Infrastructure Trends, Aug 2021. All ESG research references and charts in this white paper have been taken from this research report, unless otherwise noted.
All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at 508.482.0188.
Enterprise Strategy Group | Getting to the Bigger Truth™
Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community.