A customer asked the following:
"We had a scenario where we received an email alert in the overnight hours because a remote switch went down. However we realized that Foglight will not notify again once the switch goes back up. So we came to the office wondering if the switch was still down and once we got to the office and actually logged into the Foglight web client, we realized the switch was only down for 4 minutes. It would have been nice to receive a follow up email from Foglight when the switch actually came back up. So we would have seen the email at 1AM saying “Device is down” but then another email at 1:04AM saying “Device is back up” and we would have known we had nothing to worry about. Do you know if such a feature exists?"
By default, actions attached to rules (eg. an email alert) will fire upon entering the alarm severity. When the remote switch in the scenario above went down, Foglight entered either a warning or critical state and the email action took place.
There is a not so well known function in Foglight rules that lets you run an action when a condition causes Foglight to exit the severity level. At least that was my plan of attack for the scenario above. Upon further thought, it would be tedious to have actions when the alarm exits each alarm state (and perhaps gives a false positive because it just escalated to critical from warning).
For multiple severity rules however, there is a severity of "Normal". A-ha! We could just add an email action to that when it enters the Normal state and that should do it.
1- Start by navigating to Administration -> Rules & Notifications -> Rules. Select the NetMonitorAgent cartridge from the pulldown list. Then highlight the rule to modify, and select View and Edit.
Select Rule Editor from the popup. It is a good practice to make a copy of the rule first.
Notice that there are 2 severity levels for this particular rule (Warning and Critical). Expand the Warning section and click on the Action tab to check for existing actions. There is an email action already setup for Warning. Most defaults are used, however, there are some variables as well.
Click the Severity Level Variables tab to check out the variables and their values.
The "Subject2" variable is self-explanatory. The Text2 variable (which is the placeholder for the message text) is a bit more involved. It references @deviceName and @monitoringHost. Those don't appear in this tab, which tells us it is somewhere else.
The "somewhere else" is on the Rule Variable tab. These are variables that can apply to anything in the rule, regardless of severity.
Now that we see how other parts of the rule are built, expand the Normal severity. Foglight has an easy way to setup the action and variables - just copy them from another severity level.
Once the values are copied over, modify the ones that make sense. In this case, change the subject and text of the email message. To do that, click on the name of the severity level variable, make modifications in the Expression/Message window, and then click <<Add. Click ok to any warning that comes up.
Finally, click the Save button to make sure it takes the changes.
If you have an "expendable" device to test against, you could add it to NetMonitor, let it collect data for a bit, and then go pull the plug. You should get the warning or critical alert. Plug the device back in, and hopefully you'll get the "all clear" email.
For completeness, you could also add text from the "Text2" severity variable to the Alarm Message tab to confirm that the message is appearing properly on the Alarms dashboard. [Update: This step is not necessary. Green/Normal state alarms will not show up anywhere. You can however add the email action as above (even if email server is not configured) and you will see the event in the FMS log file.]
