During periods of scheduled maintenance, you may wish to prevent Foglight from sending alarms about the systems involved in the maintenance task. The use case is quite obvious. You know the system's down for maintenance and there's no need for you to be alerted to this fact. This is known as "blackouts" in Foglight.
The process is quite simple:
- Decide whether you want to stop collecting data or whether you want to stop processing rules
- Configure the blackout
Decisions, Decisions, Decisions
Foglight Blackouts can be configured to either stop collecting data or to stop processing the rules that are responsible for generating the alarms. The distinction is important. If you tell Foglight to stop collecting data, it absolutely will. While this is definitely one way to prevent those pesky alarms during the maintenance window, it might not be the ideal approach. If you choose this option, you won't have any data about your maintenance window. If you choose to stop processing rules, data is still gathered, but no alarms will be generated. The decision is yours.
Create the Blackout
Navigate to the Administration dashboard and select "Blackouts":
You have the option of creating scheduled blackouts or a one-time blackout:
The remaining options allow management of previously created blackouts. When you create the blackout (either scheduled or one-time), you have to choose whether to suspend alarms (the processing of rules) or suspend the collection of data. These are the options I discussed previously:
Agents vs. Topology Objects
Quite honestly, this topic could be a separate blog. I'll do my best to summarize the idea here, but I may butcher it. The Agents are what are responsible for gathering data. Topology objects are an essential framework within Foglight. Topology objects reflect the Foglight data model. In the context of this discussion, topology objects represent a very granular way of suspending alarms. In other words, if I want to suspend alarms for a specific database on my SQL Server instance, I can select said database as a topology object.
Suspend Data Collection
When you suspend data collection, you have the option to select a topology object. However, it's the agent that actually gets selected. Since it's the agent's responsibility to gather data, the agent is turned off. Let's see an example. If I choose to suspend data collection and click the Next button, I see the following:
I'll choose the option to "Select Monitored Elements" and click Next to expose the categories of topology objects. I'll highlight DB (for database):
**HINT: The above list is exactly the same that you see if you choose to suspend alarms. We're choosing to expose topology objects.
Clicking Next will expose database the category of topology objects within Foglight. The topology objects you'll see are based on the cartridges you have installed in Foglight.
I'll expand SQL Server and choose the DBSS_Backup_Locations topology type:
and click Next:
As you can see, Foglight is telling me that, because I've chosen to suspend data collection, I actually have to chose the agent. By selecting a topology object, Foglight will do the heavy lifting for me to discover the related agent and display it on the screen. If I had chosen to select the agent, as opposed to the monitored element, I would see the list of agents currently deployed in Foglight:
Clicking Next through the UI will allow you to select the blackout interval (for a one-time blackout) or to select the schedule (for a scheduled blackout).
Suspending Alarms
This process is very similar to suspending data collection. The difference is that, by suspending alarms, I have the ability to be very granular and select specific topology objects. This allows me to target very specific areas, within Foglight, where I can stop processing the rules and, therefore, prevent alarms from generating. When I choose to suspend alarms, I immediately see the categories of topology objects within Foglight:
Choosing the topology category exposes the appropriate topology categories related to the cartridges installed and the related topology objects (just as we saw previously):
To summarize, blackouts allow us to prevent Foglight from generating alarms. You can do this by suspending data collections at the agent level or by suspending alarms for very specific topology objects in Foglight (say a specific database on a particular SQL Server instance). Suspending data collections prevents Foglight from gathering data during your maintenance window. Suspending alarms allows Foglight to gather data, but prevents rules from processing their conditions that trigger the alarms.
