This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Core Memory

Looking to start a discussion on Core memory usage and memory trouble-shooting in general

1) The first problem we are seeing is that the Core will often be at 0 free memory with a HUGE amount in standby. Yes I know low free memory is "normal" in current server OS's and that standby memory is available to be used by another application if it needs it. But that is certainly not my experience either and def not when RR is the one with all the memory in standby

I have a Core that has 125GB of memory, 25GB is in use from every process and 100GB is in standby. 0 Free

I see the TN below about write caching. But a few issues

https://support.quest.com/rapid-recovery/kb/119686/high-paged-pool-ram-utilization-on-systems-running-appassure-or-rapid-recovery

a) The Core is 2012 R2 so should not be having this issue.

b) The technote gives no indications of how to confirm if you are having this issue.

c) Without a way to confirm if I am having this issue, the technote may not even help

https://www.quest.com/community/products/rapid-recovery/f/forum/21016/core-memory-usage-in-hyper-v-vms-making-vms-unresponsive#

 

2) Rammap file summary will often show a HUGE amount (+100GB) of memory in standby

Is this normal, does it show a problem with write cache (or anything else)

Why does this memory only show up in RAMMAP file sumary and point to the dfs.records file of our repo.

What does it not show up as standby memory allocated to core.service.exe (or any process) in task manager/ resource monitor. This is not how process are supposed to act

Parents
  • I could be way off here, but I thought that standby memory is managed by windows and the whole point of standby memory is to allow Windows to try and "cache" pages that are no longer actively needed by a process. The intent of keeping it in RAM marked as inactive rather than zero it out, is to better server the processes running on the system that are using lots of pages of RAM. So for processes that are using the same pages over and over but are deallocating memory regularly, Windows is speeding that applications data processing time by "caching" those pages so that it doesn't have to call them from the disk when the process needs them again.

    The reason the RR repository files are the ones that are making up most of the standby RAM is that when write caching is enabled on a repository, RR allows Windows to cache writes to the repository in RAM. So lots and lots of pages of RAM are consumed by writes to those files and then marked as inactive once the write is complete. Over time you see standby memory grow to consume all RAM since Windows is leaving those pages in Standby and not zeroing them out to try and improve performance. More than likely it isn't improving performance because RR is a backup software so most of what it does is data writes to the repository. Reads only happen for tasks like replication, virtual standby, data checks, rollup, etc. So unless those tasks are happening immediately after a backup completes, standby memory is probably not improving performance.

    The KB article 119686 was written specifically for Server 2008 and 2008 R2 which both had issues with quickly releasing standby RAM. Windows memory manager didn't handle standby memory as efficiently as Server 2012 and newer do, and so when the system reached 0% RAM free due to active and standby memory, we saw performance degrade. We believe it has something to do with Windows Memory manager not being able to zero a page and return it to another process efficiently. By disabling write caching, Windows doesn't fill the standby memory with cached pages from the repository writes and doesn't end up in a situation where it is completely out of free memory and has to zero out standby memory to make it available to other processes.

    That said, in 2012 and newer we have seen negligible difference between write caching enabled and disabled in the core software. Again, because RR does mostly data ingest (since it's a backup software and that's its primary purpose), it doesn't take long for Windows to run out of readily available RAM for write caching. Especially when you have 3 or 4 concurrent backup jobs sending data to the core and maxing out resources. So if you're unhappy with all your RAM being in standy, disable write caching in the repository per that KB article and I'll bet you see significantly less RAM marked as Standby.
Reply
  • I could be way off here, but I thought that standby memory is managed by windows and the whole point of standby memory is to allow Windows to try and "cache" pages that are no longer actively needed by a process. The intent of keeping it in RAM marked as inactive rather than zero it out, is to better server the processes running on the system that are using lots of pages of RAM. So for processes that are using the same pages over and over but are deallocating memory regularly, Windows is speeding that applications data processing time by "caching" those pages so that it doesn't have to call them from the disk when the process needs them again.

    The reason the RR repository files are the ones that are making up most of the standby RAM is that when write caching is enabled on a repository, RR allows Windows to cache writes to the repository in RAM. So lots and lots of pages of RAM are consumed by writes to those files and then marked as inactive once the write is complete. Over time you see standby memory grow to consume all RAM since Windows is leaving those pages in Standby and not zeroing them out to try and improve performance. More than likely it isn't improving performance because RR is a backup software so most of what it does is data writes to the repository. Reads only happen for tasks like replication, virtual standby, data checks, rollup, etc. So unless those tasks are happening immediately after a backup completes, standby memory is probably not improving performance.

    The KB article 119686 was written specifically for Server 2008 and 2008 R2 which both had issues with quickly releasing standby RAM. Windows memory manager didn't handle standby memory as efficiently as Server 2012 and newer do, and so when the system reached 0% RAM free due to active and standby memory, we saw performance degrade. We believe it has something to do with Windows Memory manager not being able to zero a page and return it to another process efficiently. By disabling write caching, Windows doesn't fill the standby memory with cached pages from the repository writes and doesn't end up in a situation where it is completely out of free memory and has to zero out standby memory to make it available to other processes.

    That said, in 2012 and newer we have seen negligible difference between write caching enabled and disabled in the core software. Again, because RR does mostly data ingest (since it's a backup software and that's its primary purpose), it doesn't take long for Windows to run out of readily available RAM for write caching. Especially when you have 3 or 4 concurrent backup jobs sending data to the core and maxing out resources. So if you're unhappy with all your RAM being in standy, disable write caching in the repository per that KB article and I'll bet you see significantly less RAM marked as Standby.
Children
No Data