Shredded storage and Storage Maximizer with SharePoint RBS

With the release of SharePoint 2013, there’s been a lot of curiosity about one of its new features, Shredded Storage. And for good reason. During the past decade, enterprises have struggled to cope with the increasing demand for storage capacity as SharePoint has become the go-to platform for document storage, management, and collaboration.

 

We've been doing a lot of internal research in our labs to tune performance of both traditional and shredded storage in 2013. We're going to be publishing those results shortly - best block size, best shred size, and the rest - but for now, let's introduce the basic storage design concepts.

 

SharePoint stores all its content, and all the associated metadata (information about the file, like who wrote it) directly inside a SQL database. Makes sense – up to a point. The problem comes when that content gets bigger. Keeping all those documents, and all those versions INSIDE SQL makes it easy for those databases to grow beyond Microsoft’s recommended maximum size for content databases. It also makes those databases slower and harder to backup, besides the usual challenges of putting any content into the SQL database that is over 2-3 MB.

 

Quick note – files, when stored inside SQL, are called BLOBs – Binary Large Objects. Microsoft introduced Remote BLOB Storage [1] to let administrators move those files OUTSIDE SQL to more efficient storage – file shares, NAS devices, or that snazzy new SAN storage with high performance and redundancy. Large files should always be externalized regardless of database size.

 

Now, in SharePoint 2013, Microsoft has introduced “Shredded Storage”. Shredded Storage is an additional option that allows you to break up Office documents into smaller pieces for more efficient storage. When a large file is edited, only the “shreds” that are changed need to be updated. This can really help reduce the size of highly edited, multi-versioned content. But it doesn’t address size and performance, and it only helps with versioned documents.

 

The great news is that Dell’s solution for managing RBS, Storage Maximizer, works with both traditional SharePoint storage, as well as shredded storage. Lets look at how.

 

For our example, lets assume we just have a single 100GB[2] Microsoft Word document in SharePoint 2013, running on SQL Server 2012 with a big Dell Compellent SAN sitting right alongside. Our SharePoint libraries are enabled for versioning[3] and collaboration, and our SAN has a lot of extra space. It’s a large, 100 page document. Lets start by saving that document to SharePoint 2013 for the first time, without RBS or Shredded Storage. (In the following charts, original file sections are shown in blue, and edits are highlighted in green.)

  SQL Storage SAN Storage
Initial save of 100GB doc

100GB:

0GB

Makes sense so far. Now, lets assume we go in and edit 20 pages of the document and save it again

  SQL Storage SAN Storage
Update 20% of the document and save again

200GB: 

0GB

Now, lets turn on SMAX and let it move those files out to the SAN for us. This shrinks our content database over 99% and moves the files to a more appropriate store.

20% File edit, SMAX/RBS enabled SQL Storage SAN Storage
100GB 1K (metadata)

200GB:

What happens with Shredded Storage? In Shredded Storage, that file gets invisibly broken up into dozens of storage shreds. Only the edits are updated, so the storage growth is lessened – but its still building up in SQL if RBS isn't being used.

  SQL Storage SAN Storage
20% File edit, no RBS, with shredded storage

120GB:

0GB

Finally, we can see the most efficient way to manage our storage growth, when we use SMAX alongside Shredded Storage.

  SQL Storage SAN Storage
20% File edit, with RBS and Shredded Storage 1K (metadata)

120GB: 

 

To sum up:

Scenario SQL Storage SAN Storage
100GB initial save

100GB:
 

0GB
20% file edit, no RBS or Shredded Storage

200GB:
 

0GB
20% file edit RBS only 1K (metadata)

200GB:    

20% file edit Shredded Storage Only

120GB:
 

0GB
20% file edit SMAX with Shredded Storage 1K (metadata)

120GB:

 

Its a clear message – the best solution to rein in those pesky SharePoint BLOBs is RBS using Storage Maximizer, along with SharePoint Shredded Storage.


[1] There was also an older version of this called Extended BLOB Storage (“EBS”) released with SharePoint 2007.

[2] This is much larger than normal for a single document, but many content databases are significantly larger. 100GB makes the math easier.

[3] Don’t forget – Dell Site Administrator for SharePoint reports on file version and allows administrator to set policies and thresholds.

 
About the Author