Data deduplication has exploded onto the backup scene in the last few years. But now that data deduplication has firmly established itself, new options for its deployment and implementation are presenting themselves. One such option is last week’s introduction of Data Domain Boost from EMC.
Over the last few years, data deduplication has broken into three distinct camps: Source-based deduplication (EMC Avamar), media server-based deduplication found on enterprise backup products such as Symantec NetBackup or CommVault Simpana and target-based deduplication (EMC Data Domain).
There are valid use cases for each scenario. The best for source-based deduplication is applications with modest amounts of changes to their data while media server and target-based deduplication are best suited for enterprise shops that have applications with large amounts of changing data.
Deciding whether to use media server and target based deduplication in enterprise shops then comes down to tracing the backup data path. If the backup data first has to go through a media server before hitting the target device, it may make sense to deduplicate data on the media server (assuming the media server has sufficient resources to do so). If the media server has a lot of work to do in terms of deduplicating the data, then it was more logical to deduplicate the data on the target device.
Since many backup topologies have the backup data traversing the media server and more enterprise backup software now supporting media server deduplication, the case for moving the deduplication of backup data to the media server was becoming more pronounced. If anything, the advent of media server deduplication was starting to alter the use cases for target-based deduplication solutions.
The recent introduction of DD Boost from EMC changes this perception. Now to do data deduplication with a Data Domain solution, the backup data no longer has to wait to be deduplicated by the Data Domain target device but can be deduplicated while traversing the media server. To accomplish that, the Data Domain Boost software library that is needed to execute part of the deduplication process is distributed to the backup software media server.
In this sense, EMC tips its hat to backup software providers that already offer deduplication. By offering this new DD Boost feature, it acknowledges that it can reduce local network traffic by 80 – 99 percent between the media server and the Data Domain solution since redundant data segments do not need to be sent to the target. EMC has even found that doing deduplication on the media server reduces the media server’s overall overhead by 20 – 40 percent since it is sending less data to the target device.
So the question may arise – why not just use the deduplication feature that is natively found on some backup software products and circumvent the need for DD Boost altogether? EMC showed a little foresight in this area. By making DD Boost available for media server deduplication, it one ups the native deduplication features found on some of these backup software products in three important ways that enterprises should find especially attractive.
- First, these 20 – 40% reductions in overhead are possible because of how DD Boost is implemented. The typical inline segmentation and generation of the hash by the Data Domain system now takes place on the backup media server using DD Boost. Now that DD Boost has the hash, it communicates with the back end Data Domain system to determine what segments are unique.
Once those are identified, the backend system communicates back to the DD Boost Library on the backup server as to what segments the media server needs to send. Once it has that information, DD Boost compresses the segments and sends them. This preprocessing minimizes the amount of the data that gets transferred over the network.
- Second, DD Boost enables the backup media server to aggregate multiple interfaces on a Data Domain system. This aggregation creates a load balanced infrastructure so DD Boost can dynamically move data from the backup media server to the Data Domain system.
This aggregation is important for those organizations that cannot afford 10 Gb Ethernet technology and want to leverage their existing 1 Gb infrastructure. Using DD Boost in conjunction with a Data Domain system, its multiple 1 Gb interfaces can be aggregated together and presented as one logical interface to the backup media server.
This aggregation also eliminates the need for organizations to do load balancing at the network level. Rather the Data Domain system communicates with the DD Boost plug-in to do the load balancing of the various jobs sent to it from the media servers that are using it as a target. This functionality also enables organizations to deploy systems in environments that are network constrained.
- Third, it provides link failover. In the event one link goes down, DD Boost can failover to another link and restart any of the backup jobs that were using the failed link. This is again done at the Data Domain software level rather than at the network level.
DD Boost is now immediately available for use in Symantec NetBackup and Backup Exec environments that have the OpenStorage API (OST) feature installed. Further, everyone that has purchased the OST option in the past will have access to this feature as a free upgrade.
The next place to look for DD Boost functionality is in forthcoming EMC NetWorker upgrades which will distribute DD Boost to EMC NetWorker storage nodes. EMC NetWorker will also introduce new wizards that will step backup administrators through the setup and configuration of DD Boost and Data Domain within NetWorker. Finally, NetWorker will integrate with Data Domain Replicator software so that NetWorker clones can be replicated between Data Domain systems and then individual retention policies can be set on each copy in each location.
EMC’s decision to move some of its deduplication software into the backup media server in the form of its DD Boost is in one sense a tip of its hat to media server-based deduplication. However EMC Data Domain does more than just acknowledge this form of deduplication. It leverages its strength in deduplication to one up it that gives users compelling performance and management reasons to choose DD Boost over native media server based deduplication.