Here's some words from what we do, I hope it helps.
For SharePoint backup, we encountered the same problem due to a lack of any sort of process control/management. For now, we've written essentially a wrapper around the "metavis.exe" operations that we use, outputting the logs (-log) stdout and stderr to a set location. Once the process closes, the files are checked to ensure the operation was successful. Whilst this works, there are many disadvantages, two of which are mentioned below.
Software updates often change the outputs, resulting in the checks failing, even if the operation was a success. To capture this, we have a test written that runs a backup and performs all the checks. If the test fails, we have to update the checks. A recent example is that the –log flag stopped working, which this test should detect, so we’d revert to the previous version and raise a case to request this to be fixed.
Secondly, there are some errors where the metavis process does not terminate but the process has fallen over. A good example of this issue is where the Garbage Collection overload exception is thrown, which we often face when backing up large lists. This is something to do with the Java virtual machine spending too much time garbage collecting and can be mitigated by allocating more memory to the application (currently we're at 24GB and we still face this issue). To detect this, we essentially have a process that checks the logs and processes every day, it will error if the process has been running for over two days but no changes to the logs has been written. Often an engineer will terminate the process as there's no other choice.
Thanks Steven, you answered my question. We will set a few control points based on your advice. Cheers!
