Job hungs every night

Hi,

 

I have a problem with the backup jobs on one of my machines. The job alwasys hung at 6% and I have to manually cancel the job to try again with the backup. I need help to troubleshoot the problem and solve it asap. See attached pictures.

 

 

Best regards

Poke

  • Hi pokedoke:
    For a quick solution, please open a case with our support team. Unfortunately, the information you provided does not suggest the cause of the issue. For instance, we do not know if forcing a backup for that agent manually is successful or even if it is possible to back up drives D: and E: but not C:. In most cases corroborating RR and Windows logs on the Core and Agent gives a solid indication as of what the issue may be and some troubleshooting would pinpoint the solution.
  • Hi,

    Thanks for your reply.

    How do I change schedule to only take base image or snapchot of D: and E: drive?

    Here is some info from the log-files.

    Core-Log

    ERROR 2017-01-15T20:42:21 [13818] - Replay.Core.Implementation.Transfer.VolumeDataReaderStream (Agent=Server-name Volume=C:\ BlockOffset=0x7a721d BlockLength=0x200 Job Id = 171b89f5-d01b-4ca3-8ae5-8f74bf5d854b:TransferJob 0ef2736d-b689-43ff-8d6a-00b711556be4:VolumeTransferJob)
    Transfer data server responded to request for block offset 0x7a721d, block length 0x200 with server error: This volume instance was closed for IO operations. New instance should be created
    System.InvalidOperationException depth 0: This volume instance was closed for IO operations. New instance should be created (0x80131509)


    Agent-Log

    ERROR 2017-01-15T20:45:42 [84] - Replay.Agent.Implementation.Transfer.TransferDataChannel (RemoteEndPoint=172.18.1.1:58967)
    Sending of volume data at byte offset 0xd94b80000, byte length 0x400000 failed
    System.AggregateException depth 0: One or more errors occurred. (0x80131500) ---> (Inner Exception #0 depth 0) Replay.Common.Contracts.Win32Api.Win32ApiFailedException depth 1: I/O operation - The network connection was aborted by the local system (0x80131500)
    at Replay.Common.Implementation.Win32Api.NativeIOReader.EndTransmitFile(IAsyncResult asyncResult) +0xc3 600238e
    at Replay.Common.Implementation.Win32Api.NativeIOReader.<>c__DisplayClass4.<StartTransmitFile>b__3(IAsyncResult ar) +0x0 60029f5
    --- End of inner exception depth 1 stack trace ---
    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) +0x11 6003de5
    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) +0x43 6003e09
    at Replay.Agent.Implementation.Transfer.TransferDataChannel.SendRawData(IVolume volume, UInt64 offsetInBytes, UInt64 lengthInBytes) +0x69 60007b3
    at Replay.Agent.Implementation.Transfer.TransferDataChannel.SendData(IVolume volume, UInt64 byteOffset, UInt64 byteLength, UInt64 volumeSize) +0x6c 60007b0
    at Replay.Agent.Implementation.Transfer.TransferDataChannel.SendVolumeDataResponse(IVolume volume, UInt64 byteOffset, UInt64 byteLength, UInt64 volumeSize) +0x22 60007ad

    DEBUG 2017-01-15T20:45:42 [84] - Replay.Agent.Implementation.Transfer.TransferDataConnection ()
    Socket object disposed while handling requests from 172.18.1.1:58967; this probably indicates the server is shutting down. No further requests will be handled

    DEBUG 2017-01-15T20:45:42 [84] - Replay.Agent.Implementation.Transfer.TransferDataServer ()
    Removing client connection 172.18.1.1:58967 from connection list

    DEBUG 2017-01-15T21:04:50 [45] - Replay.Agent.Implementation.Metadata.AgentMetadataService ()
    BCD backup created with the following result: 'The operation completed successfully.
  • There is no way in the GUI to set specific volumes to only take base images. The Rapid Recovery software is designed and intended to have a single base image and then take incremental snapshots from then on. When going to restore you are then able to chose whichever recovery point you want to restore from and the software does all the work of merging the changes together and presenting you with the data as it was on the date of the recovery point. It's not necessary to have multiple base images.

    As for the transfer failure this error is often caused by a mismatch between the core and agent version. It might also be caused by a network error in your environment during transfer causing the agent and core to lose communication with each other. Additional work has been done in Rapid Recovery to improve the resiliency of the connection between the core and the agent. I highly recommend upgrading to the latest version of the Rapid Recovery software version 6.1 on both the core and the agent. That should resolve the transfer issue. If it does not, please open a support case so that our Support Team can look deeper into the issue.
  • Hi pokedoke:

    To remove (or add) drives from protection and take snapshots of the remaining ones, please take a look at the steps in the picture below.

    1. Select the machine to process

    2. On the summary page navigate to the volumes section and expand the volumes info

    3. Select the volume(s) you want to remove (or add to protection)

    4. Hit "Set a schedule" (a window pops up)

    5. Uncheck (to remove) or check (to add) Weekdays/Weekends checkboxes

    6. Hit "Apply"

    7. Force a snapshot

    It is important to find out if you can backup at least one volume. If you can, it means that there may be some issue at volume level (for the volumes you cannot backup). Anyway, the errors you posted suggest a network disconnection. However this may be the result and not the cause of the failure. Please check the Windows logs for any issues (i.e. no room for VSS snapshots to grow, NTFS Error 55 etc.). If you are able to backup at least one volume, you may be able to pinpoint the issue to one volume and we can take it from there.

    Please let us know how it goes.

  • Hi Tim,

    I have core Version: 6.1.0.645 and agent version Version: 6.1.0.645. I have four other machines running version 6.0.x and they works without any problem.

    Where can I open a support ticket?

    Best Regars
    poke
  • Here is the link to our page which details the ways to open a support case - support.quest.com/contact-support.
  • Hi Tudor,

    I succeeded to backup D: and E: drive without any problem. But when I try to backup my C: drive i hungs at 77% or 47-49gb, so the problem is only on the C: drive.

    Thanks for your help and I will open a case at the support

    Best Regards
    Poke
  • Hello Pokedoke , 

    can you share de solution. I have the same issue.

    regards