Way back in 2020, these posts formed the building blocks of using the command line to automate alarm blackouts in Foglight:
In Part 3, we looked at adding objects (instances, hosts, etc.) to a service definition, and then having the blackout query apply to the objects in the service definition.
When we ran the fglcmd to set the blackout on the contents of the service, we need to understand that it takes effect at that point in time. If a new instance is added to the service definition (either manually, or dynamically via query), the blackout does NOT automatically apply to the new objects. We need to re-run the fglcmd to enforce the blackout on the newly added objects. Let's look at how we can use Foglight to automate that step. I really don't want you sitting there hitting up arrow + Enter over and over and over..
We will use a Foglight rule to automate the running of the fglcmd to set the blackout. Start by creating a simple rule and fill out the circled fields. In this example, I've set the rule to run every 2 minutes (for testing). You could also set it to use a schedule. A couple points that are critical - it needs to trigger without data, and there is no scoping query.
The condition for this rule is "true" - the rule fires when it evaluates to true.
On the "Email Notifications & Recover Actions" tab, we'll create a "CommandAction" and add it for the "entering" condition of the rule. Clicking on the CommandAction link in the Action Name column then lets us enter the necessary parameters.
The parameter needed in this case is the COMMAND_LINE. It's going to be "User Defined" and we simply point it to the script (.bat or .sh) that we are going to run.
This is the .bat text that I used in my example. You could of course have multiple fglcmd calls if there are multiple topology queries to blackout, etc. You could also run the fglcmd with "topology:blackouts" to list the blackouts and pipe the output to a file for later review.
I outlined a testing process in Part 3 of this blog series. You can set a blackout on objects in a service definition using interactive command line. Add a new instance to the service definition, clear some recurring alarms (eg. database hasn't been backed up) and watch them come back. Then go an create this type of rule with command_action. Wait a bit until you know the alarm has fired, clear those alarms and they should not come back.
To take a deeper dive with Foglight in your environment, you can try it out by downloading the installer here.
