Introduction to deduplication in Quest QoreStor

Hello, again. And welcome to another video on Quest QoreStor. In this video, I will be giving you a quick introduction into the different types of deduplication and, more importantly, highlight the type of deduplication administered by QoreStor, which is variable block deduplication. So without further ado, let's get started. So we know that QoreStor provides deduplication, but what does that mean? There are several approaches to deduplication to help combat increased need for data storage. The most basic method is single-instance or file-level deduplication. In this method, identical files are tracked, referenced, and then only the unique files are stored. However, if a file is changed, it is treated as a new file and stored alongside the previous version. Since this deduplication method is relatively simple, it is good for application use, but not very good for data backup and, typically, results in a less than 5% storage savings. Fixed-blocked deduplication is where sub-file matching occurs, but the block size never changes. This is because the block size is set by the software managing the deduplication. This method allows for changed parts of a file to be stored, rather than storing a unique instance of the whole file. This method is better than single-instance dedupe but is only OK for housing backup data and, typically, results in a less than 30% storage savings. Variable-block deduplication will change the size of the blocks to provide an optimal level of deduplication for storage utilization. Hash fingerprints are assigned via an algorithm. The data sets are then scanned for the matching fingerprints. If there are no matching fingerprints, then the data will be stored as unique blocks. If there are matching fingerprints, a reference will be made and the incoming blocks will be discarded. This method accounts for shifts in blocks within the file. New header information that will be smaller than a fixed block can now be captured as a new block at a smaller size. So let's look at an example of the difference between fixed-block deduplication and variable-block deduplication. Here, we have an original data stream that's comprised of the first line of Herman Melville's classic, Moby Dick. In the second stream, using fixed-block dedupe, you can see that the name was shortened from Ishmael to Ish. As you can see, all of the matched chunks of data to the Original stream are shown in orange. The only matched block is the first one as all the other blocks have shifted due to the removal of some data, even though that data did not change. Therefore, all the blocks in blue are unique blocks that have to be stored. In the next stream, still using fixed-block dedupe, Ish has now changed to Izzy and, consequently, all data, again, has shifted and the one matched block has also changed and thus, all of the blocks are treated as unique. With variable-block, and compared to the original data stream, there are significantly less unique blocks that need to be stored as boundary changes allow that reduction.