A customer asked how to implement and email that indicates an alarm has been cleared. In Foglight, there are typically several ways to do that.
In this post, we'll look at creating a new rule that fires when an alarm event occurs. The event we'll focus on is when the alarm clears. This could be the system clearing the alarm because the condition that raised it has gone away (eg. backup missing alarm is raised, DBA takes a backup, the alarm condition is no longer true and the alarm self clears.) This could also be when someone manually clears an alarm - whether or not the underlying condition is resolved. And it could also be if a alarm is programmatically cleared - the Database Expansion Pack, for example, has a rule that clears any alarm that is older than 7 days. That's also a "clear" event.
We'll start by creating a new rule. It needs to be a "simple" rule with no scoping query and it will be event driven - when an "AlarmSystemEvent" occurs. [Note you can create multiple rules of this type to suit your needs.. it doesn't need to be one all encompassing rule.]
On the "Conditions & Actions" tab, we'll expand the "Fire" option, and enter the condition for the rule to fire. If you wanted this for every rule in Foglight, you could just enter:
@event.get("isCleared")
You can expand the criteria to include only alarms of a certain severity that have cleared, for example, if you want just critical and fatal alarms, you can add:
@event.get("severityLevel") >= 3
Or maybe you want to narrow it down to only SQL Server (DBSS) or Oracle (DBO) or DB2 (DB2) alarms. In that case you could add:
@event.get("ruleName").startsWith("DBSS")
@event.get("ruleName").startsWith("DBO")
@event.get("ruleName").startsWith("DB2")
In the example in the screenshot, our new rule will fire when any SQL Server (DBSS) alarm is cleared:
Next we'll add some Severity Level Variables which can then be used in our email message (or other action).
Here is the list of variables used in this example:
Next we'll add an EmailAction when the rule fires. This will allow us to build the email subject, message, etc.
When you open the EmailAction, there are parameters to complete. "mail.recipient" is the "To" value of the email, "mail.subject" is the subject line and "mail.message" is the email body. We'll use the "User Defined" option to add some descriptive text along with referencing our severity level variables that were previously created.
In the case of mail.recipient, we may want to use a registry variable. This would allow us to change the value in one spot (vs. every rule), put it on a schedule (different value for weekday vs. weekend) or scope it to specific objects (Instance1 emails go to Joe; Instance2 emails go to Kelly).
You'll simply add a value and then provide an email address as the value of the variable:
Back in the Rule Editor, you can then set the value of mail.recipient to the Variable that was just created:
We're almost there. You can define the mail.message value (again using descriptive text and severity level variables.) Here is an example:
Now it should be ready to test. Pick an existing alarm that would match the rule criteria (in this case, any DBSS alarm will do.) Then wait a few moments and check the email of the recipient. It should have something like this:
As a bonus for getting this far, here's the code for the "eventHost" severity level variable:
eventScope = @event_scope;
eventHost = null;
if(server.get("TopologyService").getObject(eventScope).get("monitoredHost") != null)
eventHost = server.get("TopologyService").getObject(eventScope).get("monitoredHost").getName();
return eventHost;
To take a deeper dive with Foglight in your environment, you can try it out by downloading the installer here.
