Why are exports so slow?

So this is something we have had issues with ever since getting our DL4300 appliance 2 years ago back when it was AppAssure, we are now running the latest version of RapidRecovery. Our exports typically cannot get any faster than around 10MB/s (MegaBytes), and over the years we have had numerous reasons from Dell/Quest as to why this is ranging from:

  • Slow network connectivity
  • Bad firmware on both DL4300 and our R730XD ESXI hosts
  • Issues with the ESXI kernal which are outside of Dell/Quest control
  • Underspec DL4300 Appliance
  • Datastore fragmentation
  • Firewalls & IPS
  • Encryption, De-Dup and/or Compression.
  • The list goes on....

So to give some info on our setup:

  • Appliance: DL4300 running RapidRecovery 6.2.0.17839
  • ESXI Hosts: R730XD - 256GB RAM - Dual Xeon E5-2630 CPU's - 24x 15k RPM 600GB Dell SAS Hard Drives - H730 Mini RAID Controller - ESXI 6.0
  • Network: 10Gb/s fibre between all servers - No Firewalls or IPS, all direct Layer 2
  • Firmware: All firmware on all servers are up to date with the Dell FTP site

We have addressed every reason (*almost) that has been given so far, and we are now completely lost for an answer other than the product is essentially useless for restoring backup data. Today I took one of our standby R730XD hosts (only 64GB RAM, all other spec the same) and did a bare metal install of Server 2016 Datacentre with the HyperV role and started an export so we can rule out ESXI as a cause, to no surprise we see the exact same behaviour with slow transfer speeds of sub-10MB/s. One thing I have noticed, monitoring the network traffic shows small regular "bursts" of a few megabytes and then nothing, and then the same again throughout the entire export. It's as if RapidRecovery is having to wait for the data to be read from disk and then sending what it has. *The one remaining step we have yet to try is disabling compression and de-dup, simply because support have warned us "it may cause some issues, but we can't say specifically". We do have the room in our repository to comfortably disable these features, and would be willing to do so if this has shown results for others.

I would be very interested to hear of any other suggestions, experiences or general feedback from other community members who are (or are not!) experiencing this issue and what steps you have tried to resolve it, successfully or not. Happy to provide more detailed information if needed.