Getting rid of those pesky 'NULL's in your Foglight notifications

One of the pitfalls of creating custom rules for Foglight is failure to take into account assumptions being made when crafting a custom rule.  As a result this can cause unexpected results when different assumptions are made, for example when the individual who created the custom rule is different than those responsible for adding new agents to the system who are either unaware of their site's naming conventions or the naming conventions are not explicitly stated.  This can create havoc when custom rules are used and assumptions are made based on those either explicit or implicit naming conventions.

To take an example, a customer was receiving notification emails from one of their Oracle monitoring agents in which the name of the database was being reported as "null" whereas notifications from other Oracle agents were correctly reporting the name of the database.

 

 The name of the database was being assigned to a custom rule variable via the following groovy code:

 agentName = scope.get("agent").name;
def tokens = agentName.split("-");
SID = tokens[2];
return SID;

 

The code above makes the assumption that the agent name conforms to the format "DBO-host-SID"; retrieving the SID by splitting the name based on the second "-" delimiter.  Most of the customer's database agent names were in the form "DBO-host-SID", but the agent that was returning the null database name had just the SID with no delimiters in the agent name.  Since there were no "-" delimiters, the split didn't occur and the reference to tokens[2] was simply out of bounds, causing the returned value to be null.

One way to solve this would be to rename the agent so that it conformed to the "DBO-host-SID" convention.  But this would have left open the possibility that it could occur again if the naming convention wasn't adhered to in the future when adding yet more agents.  Instead, another option is that the code can be adjusted in order to account for variations in the naming convention.  This is tricky because you would have to anticipate every naming convention.  For our purposes, we will assume that the naming convention is either "DBO-host-SID" or just "SID".

The following code should work no matter how many dashes are in the agent name.  It should always grab the text after the last dash, or in the case of no dashes just take the agent name itself:

 agentName = scope.get("agent").name;
if (agentName.indexOf("-") > -1)
  {
    def tokens = agentName.split("-");
    def count = tokens.length;
    SID = tokens[count-1];
  }
else
  {SID = agentName;}
return SID;

 

Here is the explanation of the code.  First the agentName is retrieved from the rule scope.  We then scan the agent name for "-" delimiters.  If the index returned is -1, there are no "-" delimiters in the string.  If the index returned is greater than -1, this is the location in the string of the first occurrence of the delimiter.  We then split the string based on the delimiter.  The tokens are the substrings extracted from the full string.  We get a count of the tokens, and select the last token extracted.  It is a zero-based array, so we subtract 1 to get the last token.

This accounts for if the name of the agent is just the SID of the database itself.  There is of course, still the possibility that we could grab the wrong token if an agent is created with a different naming convention altogether, for example "DBO-SID-host".  Such are the pitfalls of creating custom rules.  The lesson here is that care should be taken when creating custom rules to avoid over reliance on arbitrary naming conventions where possible.

Anonymous