• Locked Locked
  • Replies 7 replies
  • Subscribers 27 subscribers
  • Views 5097 views
  • Users 0 members are here
Options
Related
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Intermitten Failure

mark_36585
over 5 years ago

I have a 2012 RDS server we are trying to backup.  Everyone once in a while it fails to perform the backup.  It will give an error like this..

The next time the machine tries to backup will be successful.  Unfortunately, this error is driving the tech support guys in house nuts, they don't like errors.

Any thoughts on how to resolve this?

The transfer of the backup of '(Volume Labeled 'System Reserved'); C:\; E:\' on 'gs-app3' failed

Agent is offline

Replay.Core.Contracts.Agents.AgentIsOfflineException: Agent is offline ---> WCFClientBase.ClientServerErrorException: Call to service method https://gs-app3:8006/apprecovery/api/agent/metadata/summaryMetadata PUT failed: Failed to call Create File on disk '\\.\PhysicalDrive45' - The system cannot find the file specified ---> Replay.Common.Contracts.Win32Api.Win32ApiFailedException: Failed to call Create File on disk '\\.\PhysicalDrive45' - The system cannot find the file specified
   at Replay.Common.Implementation.Win32Api.Win32.ThrowLastError(String message, Object[] args)
   at Replay.Common.Implementation.Win32Api.Win32.GetDiskAttributes(String diskName)
   at Replay.Common.Implementation.Storage.DiskInfoBase.get_IsReadOnly()
   at Replay.Common.Implementation.Metadata.CommonMetadataService.GetDiskInformation(ICommonSummaryMetadata metaData, IStorageMetadata storageMetadata)
   at Replay.Common.Implementation.Metadata.CommonMetadataService.GetCommonSummaryMetadata(ICommonSummaryMetadata metadata, Boolean includeNonSnapableVolumes)
   at Replay.Agent.Implementation.Metadata.AgentMetadataService.GetCurrentSummary(MetadataCredentials metadataCredentials)
   --- End of inner exception stack trace ---
   at Replay.Common.Implementation.Utilities.SingletonTask`1.Execute(Func`1 function, CancellationToken cancellationToken)
   at Replay.Core.Implementation.Agents.AgentClient.GetCurrentSummaryMetadata(MetadataCredentials metadataCredentials, CancellationToken cancellationToken)
   at Replay.Core.Implementation.Agents.AgentsMetadataHelper.GetAgentSummaryMetadataInternalClient(AgentDescriptor agentDescriptor, IAgentClient agentClient, CancellationToken cancellationToken)
   at Replay.Core.Implementation.Agents.AgentsMetadataHelper.GetSummaryMetadata(AgentDescriptor agentDescriptor, IAgentClient agentClient, CancellationToken cancellationToken)
   at Replay.Core.Implementation.Agents.ProtectedAgent.<>c__DisplayClass40_0.<GetSummaryMetadata>b__0()
   at Replay.Core.Implementation.Agents.ProtectedAgent.AgentClientSend[TResult](Func`1 func)
   --- End of inner exception stack trace ---
   at Replay.Core.Implementation.Agents.ProtectedAgent.AgentClientSend[TResult](Func`1 func)
   at Replay.Core.Implementation.Agents.ProtectedAgent.GetSummaryMetadata(CancellationToken cancellationToken)
   at Replay.Core.Implementation.Transfer.Validation.Implementation.ProtectedAgentTransferValidator.Validate()
   at Replay.Core.Contracts.Validation.ValidatorBase.AggregateValidator.Validate()
   at Replay.Core.Implementation.Transfer.Queuing.Implementation.TransferQueueService.StartTransfer(TransferQueueEntry entry)
---

About this event: The transfer of a new recovery point from the protected machine has failed

  • over 5 years ago

    One odd thing is the "\\.\PhysicalDrive45&#39" this just does not look right

    If you look at the drives it tries to backup (C:\ E:\ and &#39) I would guess there is some odd storage device (&#39) that is causing the issue

    VSS is trying to snap the device &#39 and this is failing.

  • over 5 years ago

    Hi Mark, 

    What is your Core Version and Agent version?

    As Emte states, that physical disk looks a little suspicious.

    What happens if you try to create a schedule to just backup C: and E:, does the backup complete?

    If the Agent version is different from the Core, it might be recommended to Upgrade the Agent.

    Also, is this a physical or virtual machine? Agentless protection maybe a alternate solution if this is a VM.

    Please let me know if you have any questions.

  • over 5 years ago

    You said, every once in a while it fails a backup. The error message says "Agent is offline" is there any consistency to when the backup fails? Is it at the same time each day or week? Does it correlate with something else going on with your network or that server? The fact that it is able to start a backup job tells us that the agent was online when it queued the job (otherwise it would throw an alert that the backup couldn't be started cause the agent is offline). So that means that the job is able to start and then the agent goes offline when it's trying to do a metadata call and a create shadow copy call. So Emte could be correct that something in VSS trying to shadow copy the device &#39 is failing, or it could be something causing the commands to be blocked (hence the agent offline error), or it could be something related to the agent service actually going offline or being stopped and restarted. 

  • over 5 years ago in reply to francis jeffrey

    the core is 6.2.1.100, the agent is 6.1.3.100

    Should I upgrade the agent?

    This is a vm, but it's also a RDS, and I think part of the issue might be the users virtual disks being created for their profiles...

  • over 5 years ago in reply to Emte

    That device is a RDP users virtual disk. 

  • over 5 years ago in reply to mark_36585

    Upgrading the agent probably wont help you as I don't think it is a Rapid Recovery problem per say. The reason I say this is because RR calls Microsoft VSS and VSS appears to be the one failing when it tries to create the cache file. Find out what is creating the odd drive and you should be able to fix it. I assume the backups that work don't try to backup this disk.

    If the drive does not exist when you go look, maybe create a quick script that runs the get-disk powershell command and dump the output to an appended file (so you can see all the results not just the most recent) Run this script every X minutes from Windows Task Scheduler

    I don't have any experience with a RDS server but it looks like you can set RDS to create user profiles on remote servers. If this is presented as a local drive, then that may explain what is happening but I am not sure how to fix it.

    - https://www.urtech.ca/2018/04/solved-how-to-setup-rds-user-profile-disks/

    So maybe the RDS server is not used very often, so backups typically work. But if anyone is logged into the RDS server and a remote profile is created, backups fail. Just a guess

  • over 5 years ago in reply to mark_36585

    That's the issue. You should not have a user profile disk selected for backup. Remove that disk from the backup configuration and then you shouldn't ever have that failure. The user profile virtual disks are stored on C or E right? If so, you're backing up that data when you backup the actual local disks. There is no need (and RR doesn't support) backing up a mounted virtual disk within Windows. So I'll bet what is happening is when that user is logged in and that user profile disk is mounted it fails the backup. When that user is not connected and that disk is dismounted the job runs properly and doesn't fail because the disk is missing and you probably get an alert about one disk being missing, but it's not a hard failure. Just a guess, but it seems like the most logical explanation.