Data deduplication itself is nothing new and some may argue that it has not yet delivered on its full potential. How appliances implement deduplication must change and become more refined due to the explosion in the amount and types of data being backed up and retained. Sepaton VirtuoSO has emerged to provide organizations with a hybrid deduplication offering innovation in deduplication to deliver the most efficient experience and one that optimizes backup storage capacity even as it ensures backups complete within ever-shrinking backup windows.
One Size Does Not Fit All
When it comes to deduplication especially in backups, one size does not fit all scenarios. “Only inline” or “only post-process” does not provide the best method to deduplicate all types of data as certain data types deduplicate better using a specific deduplication method. For example:
- Inline deduplication. This form of deduplication is optimized for file systems, web servers, and application servers.
- Post-process deduplication. This form of deduplication is optimized for databases (Oracle, SQ, DB2, and others) that use multiplexing, multi-streaming backup approaches, as well as data types that have massive high change rates such as Microsoft Exchange and Microsoft SharePoint.
The unfortunate part about the backup appliance market today is most backup appliances only offer an “either or” scenario. In other words, they only do inline or post-process but cannot do both at the same time. This forces organizations to choose the type of deduplication approach that best fits their needs when the purchase is initially made. This sets up a potential perfect storm of issues down the road, some of which include:
- Detailed information. Organizations may have insufficient information about their environment to make an informed choice about which deduplication method is “best” at the time they acquire their appliance or at any time during the life of the product.
- Changes to the environment. The “best” deduplication method for a particular organization may change over time, due to increases or introductions of a data type that is better suited for an alternative form of deduplication.
- Maximum flexibility. In some organizations a “best” form of deduplication may simply not exist. They need both, so choosing a particular type of deduplication forces them to make and manage the trade-offs from Day 1.
The Regulatory and Encryption Impacts on Deduplication
Regulations (both external and internal to a corporation) in many industries also continue to tighten, morph, and accelerate. In many cases these mean increased data retention periods for certain types of data. Deduplication is an absolute requirement when it comes to backing up data with long duration retentions. In addition to deduplication reducing the amount of backup storage capacity required to retain these types of data, longer retention rates tend to increase data reduction ratio which translates into less capacity required and the need to maintain fewer backup appliances.
Encrypting certain active data-sets also is becoming more of a factor in some shops as encryption negatively impacts the deduplication process. In these cases, deduplication of the data provides no space savings whatsoever in the backup stream.
However, organizations still need to retain, recover, and potentially replicate this data for disaster recovery purposes. Yet in the case of these data sets which are often encrypted by an application before it is sent to the deduplicating backup appliance, it is best if the appliance does NOT attempt to deduplicate this data since no deduplication benefits are provided.
The Rise of Intelligent Deduplication
People tend to grimace a bit when they think about machines taking over and no one enjoys the feeling that “Big Brother is watching.” Yet putting intelligence into deduplicating backup appliances that can distinguish between different types of data so the right form – or even no method – of deduplication is used during the backup process is a distinct advantage offered to organizations.
By adopting deduplicating backup appliances with a hybrid approach to deduplication and applying the right form of deduplication – inline, post-process or simply turning it off – at the appropriate time for different data types, organizations can simultaneously increase their deduplication ratios while backing up more data in less time. To accomplish this feat, the appliance needs to analyze incoming backup data to apply the correct form of deduplication for each data type and even have the flexibility to turn off deduplication for data that does no deduplicate well or at all such as encrypted data.
Sepaton VirtuoSO’s Hybrid Deduplication
Sepaton VirtuoSO with its hybrid deduplication brings this intelligence into the data deduplication process. It functions as follows:
- Dedicated file system. Each backup job is assigned its own file system share using VirtuoSO’s fully distributed file system. By default VirtuoSO performs inline data deduplication on all incoming backup data.
- Let the machine decide. Incoming backup data is analyzed using a “sliding window” algorithm. If VirtuoSO finds data that has historically deduplicated well using inline and is now better suited to post-process, VirtuoSO starts to deduplicate data using that method.
- Real time. Employing an alternative method of deduplication is automatically invoked to include turning deduplication off should encrypted data be detected while the backup is occurring. This ensures VirtuoSO uses the most optimal method of deduplication by taking into account the type of data being ingested, the need to complete the backup in a timely manner and achieving a high deduplication ratio.
- Human control. In cases where organizations know the nature of the backup data ahead of time, organization can still opt to choose the initial form of deduplication that a VirtuoSO file system share uses which may include disabling deduplication altogether.
- Sufficient capacity for long term data retention. In scenarios where data is subject to specific regulations or legal holds that need to be kept for years, VirtuoSO can scale-out to store petabytes of data on disk.
More Deduplication Options Means a Better Backup Experience
By employing hybrid deduplication, VirtuoSO delivers consistent backup throughput even as it employs the best deduplication method at the right time to deliver the best backup experience possible. Using inline deduplication for file systems and web servers, post-process deduplication for databases and even turning off deduplication for encrypted data, organizations can consolidate, expedite and simplify their backups without compromising on their ability to achieve higher deduplication ratios and certainty in meeting backup windows.
Yet what makes the hybrid deduplication feature on Sepaton VirtuoSO so compelling is that it is the only enterprise deduplicating backup appliance to offer this functionality. As enterprise organizations look to simplify their backup processes, streamline their IT operations, and optimize utilization of their IT infrastructure, VirtuoSO provides a means for organizations to realistically achieve these competing objectives using a single solution.
For a very slick white-board video of how VirtuoSO implements hybrid deduplication, please view: