Storage has gotten much more appealing over the last few years as cloud computing has found its way into the vernacular of the mainstream press. But at its core storage still operates in the same old boring way that it always has – at the SCSI layer. It is for this reason new features are needed from time to time so SCSI can continue to meet the new demands of the emerging virtualized infrastructure which is exactly what is being announced today by vendors like 3PAR and VMware.
The impetus behind introducing these changes into the SCSI stack is driven by the following problems that virtualized data centers are encountering:
- VMs competing for the same resources. In cloud computing environments, competition for system resources can limit scalability and performance. While this resource contention is rarely an issue in smaller environments, large ESX or vSphere servers that may host tens if not hundreds of VMs may run up against these system scalability limits.
In these situations what is known as the SCSI reservation bit locks a LUN when, for example, VMDK clones are made. This precludes large environments from putting large numbers of VMDKs on a single large LUN. The reason is that other VMDKS on that LUN are negatively impacted waiting for a SCSI reservation to complete while a clone is made of that VMDK file.
- Expediting the creation of VMware initiated VMDK clones. VMware also has the ability to create its own clones. However when it does so, it adds extra overhead to ESX or vSphere underlying physical server’s CPU, memory and network resources since the clone has to traverse the storage array, the host and then go back out to the storage array again.
- Host overhead associated with zeroing out previously allocated space. vSphere includes the ability to zero out blocks of data when storage is allocated to a VM. By first zeroing out these blocks of data, it prevents the new VM from accidentally accessing any of the data that may have been stored on that disk by a deleted VM that previously had access to it. However the new problem that results is that the newly created VM has to generate excessive amounts of write I/O and overhead on the physical host and network in order to zero out newly allocated blocks.
It is these storage specific issues that vSphere 4.1 and the latest release of 3PAR’s InForm OS tackle. By adding three new SCSI commands to the standard SCSI command set, VMware and 3PAR give virtualized data centers the more granular control that they need to continue to scale out their burgeoning virtualized infrastructure.
The vStorage APIs for Array Integration (VAAI) that is now part of vSphere is the driving force behind this announcement. VAAI was announced as part of vSphere 4.0 but vSphere 4.1 is the first practical application of VAAI with the introduction of these new SCSI commands.
The first of the three new SCSI commands that it introduces, ATS (Atomic Test and Set) and referred to by VMware Hardware Assisted Locking, solves this problem of locking the entire LUN on a storage array when a clone of a VMDK file is taken. Rather than locking the entire LUN, ATS only locks the blocks on the LUN that are allocated to the VMDK.
This ATS command is intended to help virtualized data centers in at least two important ways.
- First, if they are already using or want to use larger size LUNs and place multiple VMDKs on a single LUN, they can now do so and still make clones of individual VMDK files without negatively impacting other VMDKs also located on that LUN.
- Second, 3PAR’s implementation of the ATS command was done within the ASIC of its InServ storage servers to expedite processing of this command. While the performance benefits that this provides in small environments may be too negligible to notice, it is service providers with large virtualized infrastructures that need to quickly create large numbers of clones that are most apt to benefit from this feature.
The second SCSI command that VAAI introduces is the XCOPY command, or Full Copy per VMware’s naming conventions, which resolves the host overhead that is associated with VMware initiating and managing cloning operations. XCOPY facilitates the cloning of individual VMDKs while keeping the overhead associated with the copy off the host and on the storage array which should significantly improve the performance of host-initiated clones. In its early testing using XCOPY 3PAR reports that it has seen a 2X increase in performance versus when clones are created using the prior method that VMware provided.
The third and final new SCSI command to find its way into the vSphere 4.1 release is WRITE-SAME or Bulk Zero as known in the VMware vernacular. This command is intended to resolve the host overhead problem that results when VMware zeros out disk space – at run time for Thin and Thick VMDKs and at create time for Eager Zeroed Thick (EZT) VMs. .
Using the WRITE-SAME command, VMware now transfers the overhead associated with these writes to the storage array by instructing the storage array to assume the burden of writing the zeros on these newly allocated blocks.
3PAR arrays then takes this WRITE SAME command a step further when the blocks associated with the VM are initialized using EZT.3PAR’s ASIC and its Thin Persistence software recognize the zeros as they are written by the WRITE SAME command.
Since 3PAR tracks which blocks on its array are zeroed out and which ones have data in them, 3PAR only needs to zero out the blocks with data in them. Those blocks of data that do contain zeros 3PAR simply unmaps so no write penalty is incurred on the 3PAR system.
Of course, once organizations (especially virtualized data centers) understand the benefits that these new SCSI commands offer, the next logical question they are bound to ask is, “What do I need to do in order to take advantage of them?“
In the near term, it will take a little bit of coordination and effort to implement these new features. On the host side, they need to upgrade to vSphere 4.1 while on the storage side they need to upgrade their 3PAR InForm OS to version 2.3.1 MU2. 3PAR users will also need to download from 3PAR a plug-in that contains these three new SCSI commands. This plug-in is then installed as a VAAI plug-in on the vSphere host which it uses to communicate with the 3PAR array.
Longer term these implementation pain points will likely go away. It is expected that these new SCSI commands will eventually be approved by the T10 standards committee and become a part of the standard set of SCSI commands. Once they are approved none of these extra steps will be required as these commands will be natively available in both VMware vSphere and the standard 3PAR InForm OS.
The introduction of these three new commands into the SCSI protocol stack is admittedly getting down in the weeds of how server and storage virtualization work and interact. However it is these types of incremental advancements that are helping to close the current gaps and pain points associated with the implementation and management of virtualized data centers.
Announcements like this one are most applicable to data centers that are well down the virtualization path and need to put forth the extra effort required to implement these SCSI commands. Everyone else can look forward to the day when these commands are simply part of the standard SCSI stack so they can take full advantage of them without the extra work in their soon-to-be virtualized data centers.