How target-based deduplication solutions are implemented going forward may have just been permanently altered after this week’s announcement from EMC. While EMC simultaneously announced a number of new Data Domain features, its implementation of global deduplication in the GDA through its heightened integration with backup software changes some assumptions as to how one should think about target-based data deduplication architectures going forward.
The newly announced Data Domain GDA from EMC is bigger and faster than its prior models while its existing DD880 added new support for encryption of data at rest as well as new data replication options. These enhancements are what one might expect as part of a broader product release. However it was the new global deduplication technique in this Data Domain system that specifically piqued my interest in this release.
Global deduplication technology is taking on increasing importance as enterprise organizations scale out data deduplication. While data deduplication certainly reduces the size of their corporate backup data stores, as they scale data deduplication out in their environments, they encounter new challenges that may include:
- The creation of multiple deduplicated data stores that may have like data
- Insufficient network bandwidth resulting from the increase in backup traffic
- Ensuring that each backup software job is sent to a deduplicating solution that has sufficient storage capacity and adequate performance to process the job
Further, enterprise organizations are starting to recognize that the adoption of a solution that supports global deduplication can increase the data deduplication ratios they achieve. This results in more storage capacity savings and can even reduce the management overhead that may be required to direct multiple backup data streams to the appropriate backup target and could even increase deduplication ratios.
The trick from a target-based deduplication perspective therefore becomes how to deliver on these objectives without needing to also create a more complex multi-tiered deduplication solution that can become costly to implement, difficult to manage and complex to scale.
The EMC Data Domain GDA does a pretty good job of striking a balance in delivering on these competing priorities. To accomplish this feat, the most notable change in what EMC does with the GDA is that it moves some of its data deduplication technology upstream from the GDA into the media server of the backup software. This new functionality works as follows:
- Data Domain takes advantage of the Open Storage API (OST) feature found in Symantec’s Backup Exec and NetBackup backup software products and installs a plug-in on the backup software’s media server.
- As backup streams come in from the client to the media server, the Data Domain system analyzes the content of the data in the backup stream, makes a decision as to which GDA controller it will send the data and then notifies the plug-in of the proper routing.
- The decision as to which controller receives the data is based upon the patterns of data included in the chunk of data and what deduplicated data has already been stored on that controller. In this way, the deduplication effect is optimized.
The key benefit that Data Domain cites that this new process of data deduplication brings to the table is that deduplication now executes on the backup media server which should result in the utilization of the backup server actually going down.
While this seems like a paradox, the rationale is that even though the backup media server is now deduplicating the data, less data is moved between the backup server and the Data Domain system. Since the backup server does not have to work as hard sending duplicate data over the LAN, the net effect is improved operational efficiency on the backup media server.
Data Domain’s introduction of a multi-controller product that does global deduplication and operates as a single logical instance into its portfolio of solutions represents a notable departure from how Data Domain has historically deduplicated data. Immediately it eliminates the need for more costly, complex controller head configurations while still enabling organizations to realize the benefits that global data deduplication delivers. Longer term Data Domain delivers a new target-based deduplication architecture that can scale to meet tomorrow’s enterprise backup demands while giving organizations a new option in which to implement it.