How to reduce alert notification noise with database monitoring tool

Your database monitoring solution is only as good as its ability to let you know about problems – hopefully before application users and revenue-generating systems are affected. But, too often, monitoring tools get in their own way by firing too many alarms, or the wrong ones, for an organization’s needs. Configuration of alarm frequency, thresholds and more contributes to the effectiveness and viability of the monitor. Are alarms configured properly for your environment? Default settings are rarely enough. Every environment is unique. So, the question becomes, “How can we customize our monitor to eliminate the noise so we see actual problems that need our attention.”

Alarm noise is the result of your database monitoring solution firing alarms that might be technically correct (a threshold violation has occurred, for example), but are insignificant or otherwise unworthy of investigation. Alarm noise is a common problem with many monitoring solutions. Controlling and eliminating the noise requires accurately setting alarm conditions at varying severities for your key performance metrics, and turning off alarms for those conditions that are not urgent.

What if you could quickly eliminate or significantly decrease the volume of needless alarms that result from your heterogeneous database monitoring or physical/virtualized infrastructure monitoring?

Quest Foglight has always had robust alarm management capabilities, and release 6.0 includes a new feature called Alarm Templates that takes your ability to control alarm noise to a new level.

Alarm Templates are sure to become a popular feature among those responsible for setting up Foglight alerting for database monitoring. Its ease-of-use and broad reach allow either a default or customized template of alarm configurations to be shared quickly across multiple monitored database instances.

Figure 1: Create a new template from a default template: The new template will be created for a chosen domain and becomes a template you can then alter.

Foglight cartridges are distinct from one another, with differing lists of out-of-the-box rules and alarms. Each installed cartridge has its own alarm template for its unique domain.

To make changes to the alarm template for a domain (such as databases), rather than using the default domain template (which you cannot change), create a new template. Simply give the new template a name and choose the domain identifying the default template to use as the model for the new template. You can then make changes such as enabling/disabling alarms or altering alarm threshold values for particular severity levels. Finally, you can assign agents (of the corresponding domain) to the alarm template.

Figure 2: Editing a template involves setting chosen alarms either “on” or “off,” and utilizing only desired severity levels (fatal, critical, warning). Thresholds for each selected severity level can be changed here as well.

Figure 3: Assigning agents to alarm templates quickly sets standard alarm behavior across the desired environment.

When additional database instances need to be monitored, you can select the appropriate alarm template.

Figure 4: Above: Simple mode.

Figure 5: Above: Advanced mode

While you can assign a template to an agent when creating the agent, you can also assign a different template to the agent later.

Figure 6: Assigning an existing template, or a new one, for a monitored instance.

Figure 7: Create a new template based on an existing configuration in the database monitoring tool

After upgrades of database cartridges, all rule configurations work the same as before – although changes made to the rule definition outside of the database instance “Administration” screens may not survive an upgrade. The suggested approach is to always make changes in the instance Administration screens if possible.

Alarm “Sensitivity Level” is no longer in effect after a template is assigned for a selected agent or agents.

Figure 8: With an alarm template assigned to the agent, Sensitivity Level is not used.

Summary

Database and infrastructure professionals’ time is precious, and too much of it has been spent investigating alarms from their database monitoring tool that turn out to be of lower urgency or, worse, not even a true problem on a particular database, for example. Customizing alarm configurations for every monitored database instance or infrastructure object has been a time-consuming task that historically has been avoided or done manually, risking mistakes and inconsistencies.

Alarm templates in Foglight 6.0 are designed to give time back to IT administrators and others who manage performance of critical databases, virtualization hypervisors, virtual machines, containers, storage and Kubernetes across the hybrid cloud. Easy duplication of alarm configurations across many monitored items in a domain means that a standard approach is followed based on what’s right for monitoring and alerting in the organization’s unique environment. 

True problems with key performance indicators, availability issues and other conditions are met with quicker action when those alarms notify the right people as soon as possible. With those requirements defined and proliferated, the database monitoring tool can add value more broadly in the organization.

Anonymous
Related Content