Foglight administrators and other Foglight performance monitoring users who are asked to configure alarms to send email notifications sometimes struggle if they miss a step or two in the process. This post is meant to take you through the basics, and help you be successful the first time.
No matter which Foglight cartridges you have installed and have put into use, a few requirements need to be met to send email notifications to people when alarms are raised:
- Supply the SMTP host name (and possibly other company-specific parameters)
- Decide who will receive email notifications
- Decide on which Foglight rules, and on which severity levels, email notifications are necessary
First, let’s set up the email server so Foglight knows how to send emails.
- The Administration >> Administer Server >> Email dashboard is the place to go to set up the Global email settings.
- SMTP host is required before Foglight can send any emails.
- Everything else on the screen are ‘defaults’ and used globally in Foglight.
- Some mail servers are going to require a username and password to send emails…check with your mail server administrator to find out.
Use of Registry Variables
You probably don’t want to type in the destination email addresses on every Foglight rule. Variables are the answer – Foglight registry variables. Registry variables can be assigned default values, and certain kinds of override values (see this post for an example of value scoping: https://www.quest.com/community/b/en/posts/setting-different-alarm-thresholds-for-different-hosts).
Each Foglight cartridge comes with its own default email destination variable set up: for example, j2eeadmin, dbadmin, sysadmin, or vmadmin. You can make changes to the values assigned to these variables:
- You can add email addresses to these variables
- You can add values that are scoped by topology objects (host, db instance, etc.) or scoped by schedules
Remember, a best practice is to copy the variable to a new name, change that one, and then change the original to use a registry reference to the new variable name. This technique has the advantage of preserving your customizations of the variable through a Foglight upgrade, and you’ll be able to safely make those changes in a copy of the variable and not have to change the rule(s) that reference it. The same, out-of-the-box registry variable name will, through the registry reference, allow the rule condition to use the correct (perhaps scoped) variable value every time.
Below is an example of a copied registry variable, with suffix added to name, and a schedule-scoped value:
You will sometimes find it useful to create new variables. Maybe for special groupings of things, if you don’t want to scope values all in one variable, for instance – and use descriptive names for the variables so it’s obvious which one to use for a particular alarm.
Alter a Rule to Send an Email Notification
You’ll alter rules to include email notifications at selected severity levels – warning, critical, and/or fatal. To do that you’ll use the “Rules” dashboard.
Please note: For database cartridges SQL Server SQL Azure, Oracle, and DB2 – Agent Administration from the “Databases Dashboard” is the place to configure email notifications!
If you set up an email notification, make sure SMTP host name is valid and email addresses are valid.
Here’s the Rule Editor, and the “Action” sub-tab where Email actions are added:
And, you’ll be able to supply some information so the email goes to the right addresses (see screen below) – the “mail recipient” is the field where you’ll use the registry variable. The others indicated by the red arrows – the message text, etc. – can all be customized as you wish. More on that in another blog post soon (but look up Registry Variables in the Foglight online Help for more information or post questions on the Foglight Community forum, the link is provided at the end of this post).
If you’re monitoring databases, you’ll need to set up email notifications on rules/alarms from the Databases dashboard’s ‘Configure Alarms’ screen:
Determine if Emails are Being Generated
To some extent, the only way to be sure that emails are being generated correctly is to check with the intended recipients. However, there are some clues that will help your investigation. Whatever type of alarm you are examining for valid email-generation, you can see the times that the alarm has fired duiring a time period, and which severity levels have had "hits" (the condition was true). If email actions are set up on those severities for the specific rule, the email should have gone out according to the way you've configured that alarm (Rule Behavior settings can have some affect on actions, remember). But let's take a look at some of those clues you get from Foglight.
On the Rule Management screen, you’ll see “Alarms” (number of time the alarm has fired in the time range).
The ‘Rule Diagnostics’ screen shows the last time a rule fired an alert, and the number of times each severity of that rule was ‘hit’:
If email actions have failed, you can see when that happened by looking through the Management Server Log file on the Foglight Management Server (FMS). Emails can fail because of communication problems between the FMS and the mail server, invalid mail server name, etc. You can view this log directly from the Foglight console:
Here’s the message to look for in the log:
Foglight won’t know that the organization’s mail server has, or has not, sent an email. The mail server logs would need to be checked (and possibly monitored for these failure messages using the Foglight Log Monitor agent).
Please check out other posts in this Foglight Skills 101 series, where various topics related to Foglight core functionality are explored.
For more information visit the Foglight performance monitoring product page.
You can ask questions about the material in this post, or offer your insights and feedback, by joining us on the Foglight forum.
