• State Not Answered
  • Locked Locked
  • Replies 4 replies
  • Subscribers 27 subscribers
  • Views 12852 views
  • Users 0 members are here
Options
Related
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Replication - GetResponse Timed Out

pancraziop
over 6 years ago

When attempting to replicate from core1 to core2 the replication is failing.   Both cores are running Rapid Recovery 6.1.1.137

Error
GetResponse timed out
GetResponse timed out

The operation has timed out
GetResponse timed out
GetResponse timed out

The operation has timed out
 
I've completely rebuilt core2.   Server 2012 R2 - removed respository and setup new.   Moved the server to a different network location for better network throughput.  
 
What other information can I provide?
 
Phil
  • 0 over 6 years ago
    The full stack trace from the error is usually helpful. Get the stack trace from both Core1 and Core2 so that we can compare the errors.

    On the replication page for Core1, it should show Core2 under outbound replications. Is the icon green next to Core2? Meaning it has connectivity to it? So the jobs are able to initiate and start but then fail after a period of time?
  • 0 over 6 years ago
    The icon is green next to core2. The job does initiate and will run for a couple hours before failing.

    Server side:

    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)
    ______________________________________
    Core2 stack trace
    Server side:

    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    Microsoft.Http.HttpStageProcessingException: GetResponse timed out ---> System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    --- End of inner exception stack trace ---
    at Microsoft.Http.HttpStageProcessingAsyncResult.Complete(HttpStage stage, Exception e)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)
    at Microsoft.Http.HttpClient.Send(HttpRequestMessage request)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.SendWithClient(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.ExecuteStreamingRequest(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.GetResponseWithRetry(HttpClient client, Uri uri, Stream data, HttpMethod method, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.CoreReplicationClient.TransferMissingRecords(Guid agentId, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.RemoteReplicatedSlaveAgent.TransferMissingRecords(ICoreReplicationClient coreReplicationClient, Guid jobId, Guid sessionId, Stream data, ISpeedLimiter speedLimiter)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.SendRawDataToSlave(VolumeImageReplicatorPipelineItem item, IPerformanceCounter rawBlocksSentCounter, Action3 progressCallback)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.<>c__DisplayClass16.<InitializeWorkerTasks>b__f(VolumeImageReplicatorPipelineItem item)
    at Replay.Core.Implementation.Replication.VolumeImage.VolumeImageSenderDvm.DoActionWithItem(IDisposable item, Action action)
    at System.Threading.Tasks.Parallel.<>c__DisplayClass322.b30()
    at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
    at System.Threading.Tasks.Task.<>cDisplayClass11.b__10(Object param0)


    System.TimeoutException: GetResponse timed out ---> System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Http.AsyncResult.End[TAsyncResult](IAsyncResult result, Boolean throwException)
    at Microsoft.Http.HttpWebRequestTransportStage.EndProcessRequestAndTryGetResponse(IAsyncResult result, HttpResponseMessage& response, Object& state)
    at Microsoft.Http.HttpStageProcessingAsyncResult.NextRequest(HttpStageProcessingAsyncResult self)


    System.Net.WebException ---> System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)


    --- End of inner exception stack trace ---


    System.Net.WebException: The operation has timed out
    at System.Net.HttpWebRequest.GetResponse()
    at Microsoft.Http.HttpWebRequestTransportStage.HttpTransportAsyncResult.PopulateWebResponse(HttpTransportAsyncResult self, IAsyncResult result, Func`3 getResponse)
  • 0 over 6 years ago
    That looks to me like a straight timeout issue. That generally means that enough packets were dropped that the job couldn't continue and timed out. Usually that means networking connectivity issues, WAN stability, or possibly even NIC issues on one or both of the machines. One workaround in the software would be to try the changes in this KB - support.quest.com/.../121363.

    To isolate the issue it may be worth throwing Wireshark on both cores and then running a capture of all traffic on port 8006 to see what's happening to the packets. If you have lots of retransmits or out of order packets, that could definitely point to what's going on.
  • 0 over 6 years ago
    Thanks. I have something to try now.