Gateway Rules, Actions, and Alerts

Data Types

There are a number of data types that can be used with the rules. These are string, integer, double (floating point/decimal), Boolean and null. Severity can also be used.

"the quick brown fox"

"he said \"I like foxes\" and \"I like cats\""
"C:\\Program Files\\ActiveConsole"
"/usr/local/geneos/gateway"

0
234
-45

0.1
15.1876
23.42

true
false

null

undefined
ok
warning
critical

if severity > ok then ...

Data Items and Paths

Paths can also be used to refer to the properties of other data-items, termed secondary variables. The paths themselves are defined separately in the path aliases section of the rule, and are each given names.

if value > path "mypath" value then...

The paths may be absolute, in that they refer to an exact item in the system. They can also be relative, for example "the same row, but column X". More information about properties and paths can be found in the XPath User Guide.

Other live system data

It is also possible to check the state of other parts of the Geneos system.

It is possible to test against the time of the gateway:

if within activetime "time1" then...

This may be used for changing the thresholds at different times of the day, for example:

if value > 70 and within activetime "trading hours" then
...
elseif value > 90 then
...
else
...
endif

Variables

Managed Entity variables are accessible from within rules. If a variable does not exist, an empty value will be returned (the empty value converts to 0 if used as a numeric value). In most cases using a non simple variable will not work in rules. The simple variable types are boolean, string, integer, and double. The inList() function is a special case as this function can reference stringList variables as well as simple variables.

if value > $(variable) then ...

Local variables can also be set, which are then accessible as per the above.

set $(variable) "literal string value"

If repeatedly using a path lookup value (e.g. in a sequence of if statements) it is more efficient to store the looked-up value in a variable and reference this instead.

set $(variable) path "aliasName" value

Variable Active Time

It is possible to use variable active time within rules.

if within activetime $(vTime) then...

where "vTime" may be defined as a variable of type active time and have different values per managed Entity.

This may be used for changing the thresholds at different times of the day for each geographic locations, for example:

if within activetime $(vTime) then
if value > 90
...
elseif value > 90 then
...
else
...
endif

Target Names

Parts of a unique name can be extracted from the target data-item when a rule is executed. These names can then be used for comparison as for a normal value. For example, to set a variable to the row name of the target item, use the following:

set $(row) target "rowName"

The names which can be extracted are as follows:

• gatewayName
• netprobeName
• netprobeHost
• netprobePort
• managedEntityName
• samplerName
• samplerType
• dataviewName
• rowName
• headlineName
• columnName

Operators

There are a number of operators that can be used to manipulate the data. Operators typically appear between two expressions (called the left hand side and right hand side), for example

5 + 3
true and false

Some operators only operate on a single expression, for example:

not true

5 + 3 * 6

(5 + 3) * 6

Functions

A range of functions exist to manipulate the data. Functions take one or more parameters, depending on the function being used. The parameters are enclosed in parentheses and separated by commas:

function(parameter1)
function(parameter1, parameter2)
function(parameter1, parameter2, ...)

For example:

abs(-8)

The list of available functions is available in the Functions configuration section.

Control Statements

if [condition] then
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
endif

if [condition] then
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
else
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
endif

if [condition 1] then
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
elseif [condition 2] then
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
else
[optional additional if/else statements]
[updates and actions] or [set variables]
[optional additional if/else statements]
endif

if value <> previous value then
set $(changed) "true"
if value > 75 then
severity critical
else
severity warning
endif
endif

Updates and Actions

If the condition of a rule is true then a number of updates or actions can be applied to the system.

[property] [value]

severity critical

run [action name]

run "email support team"

if value > 10 then
severity warning
run "email support" throttle "ten a minute"
run "email managers" throttle "three in five mins"
endif

if value > 10 then
severity critical
run "email support" throttle "twenty a minute"
run "email managers" throttle "ten in five mins"
endif

name / value pairs			

userdata "variable" $(var)

DataItem userdata "dataItem" value

userdata "dataItems" %%//cell value

[false, 1, 2.3, "four"]

userdata "view_of" target "dataviewName"

userdata "active" within activetime "timeActive"

userdata "failover_attribute" attribute "FAILOVER"

userdata "usual-at-this-time" timeseries "MyTimeSeries"

delay [length] (seconds/samples)

delay 60
delay 60 seconds
delay 2 samples

Targets

Rules must specify one or more targets that they apply to. A target is an XPath expression which specifies one or more parts of the system, for example individual cells, all cells in a column, the same cell on all managed entities, the samplingStatus headline of every dataview, etc. Rules that change the value of cells or headlines should only target computed data items you have created using the Compute Engine.

Rules may optionally have additional XPath expressions, called contexts, which are typically used to restrict the targets of a rule to particular managed entities or types. These are particularly useful when set using rule group defaults (see rule group defaults below).

For example, if a rule's target was "all headlines named samplingStatus" and it had a context selecting "all headlines in the FixedIncomeLondon managed entity", then the rule would apply only to sampling status headlines in that entity. It could be extended to other managed entities by adding more context XPaths or to other named headlines by adding more target XPaths.

Briefly, if no contexts exist, a rule applies to a particular item if it matches this expression:

(target1 or target2 or target3 ...)

If contexts do exist for a rule, the rule applies to items that match this expression:

(target1 or target2 or target3 ...) and (context1 or context2 or context3 ...)

So at least one target and at least one context must match an item for the rule to apply to it.

Priority

Multiple rules can apply to a single item. Priority is used to determine the order that the rules are evaluated. Rules with a higher overall priority are evaluated first. Every rule must set a priority and may optionally specify a priority group. 1 is the highest priority that can be set, 2 is lower than this, etc.

The overall priority of a rule is determined first by its priority group and then by its priority. If no group is set then it is treated as 0, i.e. the priority will be higher than any rule where a priority group is set. Priority groups may be set using rule group defaults (see rule group defaults below).

An optional setting on each rule, Stop further evaluation, allows lower priority rules to be ignored altogether. The 'Show Rules' command shows all rules applying to an item in order of priority; if the 'Stop further evaluation' setting applies, a dividing line with the text 'EVALUATION STOPS HERE' is shown so that you can see which are being used and which are not.

When priority is displayed (for example by the 'Show Rules' command), it will be shown as group:priority, so a rule with a priority group of 7 and a priority of 3 will be shown as 7:3.

Here are some examples of some priorities, shown with the highest priority first and lowest priority last:

0:2 0:5 0:6 1:1 1:8 3:4 4:2

If two rules have the same overall priority (that is the same priority and priority group) then the system may evaluate them in any order; this order may vary between target items and may change when the configuration is changed or reloaded.

ActiveTime

The active time of a rule determines when it applies to the system. Outside of the active time it will be as if the rule did not exist at all. Multiple active times can apply to each rule, and these will be combined together using the same rules as other active times. See Active Times for more details.

By default the active times set on rules also affect the active state of the rule's target cells. The target cell will go inactive when all rules that apply to it go inactive (the cells active state is a logical OR of the active state of all rules that have activeTimeAffectsCell set true). This can be turned off on a rule by rule basis using the activeTimeAffectsCell setting, or overridden by explicitly setting the active state of a cell from within a rule.

Single Property Updates

Consider a rule like:

if value > 20 then
severity warning
endif
if value > 30 then
severity critical
endif

If the value is greater than 30 then the two updates conflict. The first 'if' will set the severity to warning; the second 'if' will also try to set the severity, but will not be able to, because it is already being set. (There is also a problem here in that the rule has no 'else' clause to reset severity to 'ok' or undefined when the values falls below 20, see if without else)

Transactions

The transactional element means that updates and actions that are grouped together will either all be applied, or none of them will be applied. Expanding on the problematic rule from the previous section:

if value > 20 then
severity warning
run "email support"
endif
if value > 30 then
severity critical
run "email boss"
endif

Since the severity has been set to warning, the action "email support" can fire, but since the severity cannot be set to critical, the action "email boss" will not fire.

Applying Updates

Take the following rule

if severity = critical then
severity warning
endif

if severity = critical then
run "email support"
endif

If the severity turns critical then it will be changed to warning, but only after the rule has finished evaluating. That means that the second condition will also still be true, and support will be emailed.

Re-Evaluating Rules

When writing rules, it should be assumed that they can run at any point, and will be run whenever the state of the system changes. Take the following rule:

if value <> previous value then
severity critical
else
severity ok
endif

When the value of the item changes then the severity will be set to critical. As soon as the rule has finished and the state of the system has been changed then the rule will be re-evaluated, and since the value has not changed, the severity will be immediately returned to ok. In this case the severity change may not be visible to the user, but an event will be generated that can be seen in the event ticker.

Actions can also be applied to the above:

if value <> previous value then
	severity critical
	run "email support"
else
	severity ok
endif

In this case, as well as the ticker event being generated, the action will fire whenever the value of the target changes.

Disabling Rules at Start Up

In a large Gateway setup, the overall time taken for the Gateway to start up can sometimes be reduced by delaying the application of rules. This is controlled by the startup delay setting, which allows you to specify an interval in seconds between the Gateway becoming active and the rules being applied.

This may be useful because, when a data item is added to the Gateway, all existing rules must be checked to determine whether they apply to that item. Conversely, when a rule is added, all existing data items must be checked to see whether the rule applies to them. In some cases - for instance if there are a huge number of data items and many rules, most of which only apply to a few items - it is faster to check each rule against all the data items than vice versa. The startup delay setting lets the gateway start with no rules and wait until most of the data items are present before applying them.

The length of the delay interval, and whether an interval is useful at all, must be determined by experiment in a non-production environment. This is because it is not possible for the Gateway to detect when it has finished connecting to all available Netprobes, since it does not know which Netprobes are about to come up, or to determine when all samplers have finished sending their initial data, since this depends on the nature of each sampler and its configuration.

When rules are enabled after a startup delay, the 'fire on configuration change' settings for Actions, Alerts, Ticker Events and database logging are respected. The initial application of a rule is considered to take place in the context of a configuration change so that, by default, any actions, events or alerts triggered as the rule is first applied will be suppressed.

Rule defaults

Defaults can be set for contexts, priority group and active times. These will each be set on any rules that do not have these set already.

For example, if a set of defaults is configured, with default priority group 5, active time "London business hours" and a context of 'managed entities with attribute "Region" set to "London"';

• A rule in the group with priority group 3, no active time and no context would get the active time and context specified in the defaults
• A rule with no priority group, active time "London evenings" and no context would get the priority and context specified in the defaults
• A rule with priority group 2, no active time and the context of 'managed entites with attribute "Division" set to "Fixed Income"' would get the active time "London business hours" from the defaults. Since the default context would not apply, this rule would apply to all managed entities in the "Fixed Income" division, regardless of their "Region" attribute. This may not be what was intended: contexts should be specified either via defaults or directly on rules, but not both ways in the same rule group.

Transaction defaults

Transaction defaults can extend some or all of the transactions in all the rules in the group. The specification for transaction defaults has two parts: a 'match' section and a 'data' section; both consist of statements in the rule language.

The 'match' section specifies which transactions will be extended, based on the updates they perform. For example, it could specify "severity critical" or "active false". If no match conditions are specified then the defaults will apply to all transactions. If multiple match conditions are specified then the defaults will apply only to transactions which meet all the conditions.

The 'data' section specifies one or more statements, each of which will be added to the matching transactions, as long as it does not conflict with a statement already present in the transaction.

• An update statement specified as a default will be added to matching transactions which do not update any properties.
• A delay statement specified as a default will be added to matching transactions which do not set a delay.
• A run action statement specified as a default will be added to matching transactions which do not run an action.
• A userdata statement specified as a default will be added to all matching transactions. If a transaction sets the same user data variable, its setting will take priority over the default.

For example, consider a transaction default which matches "severity critical" and sets "run "email support"": this will add the "email support" action to any transactions that set the severity to critical and do not already run any actions.

A set of defaults can contain multiple transaction defaults, each with their own 'match' and 'data' sections. If a transaction in a rule matches more than one transaction default, each 'data' section is applied in turn.

Nested rule groups

Rule groups can be nested and sets of defaults can be specified at each level of nesting. Defaults on groups at an inner level of nesting are not merged with defaults on the groups that contain them: if a set of defaults defined on a group has the same name as a set defined at an outer level, the innermost set of defaults apply; otherwise it as if the sets of defaults defined at the outer level were copied to the inner level.

if without else

It is not necessary to put an else in an if statement, but in most cases it is sensible to do so. A rule such as the following may cause problems:

if value > 10 then
severity critical
endif

If the value starts off as 5 then the cell will be grey. If the value becomes 11 then the cell will go red. If it goes back to 5 then it will still be red, because it has not been told to do anything different. In fact, it will continue to be red until the gateway or Netprobe are restarted, or the rule is changed. What was probably meant was:

if value > 10 then
severity critical
else
severity ok
endif

Overlapping defaults

If two or more sets of rule group defaults apply to to the same rule, it is as if the rule was specified twice, once for each set of defaults.

For example, suppose Rule Group G contains these two sets of defaults and a relevant rule:

• Default "x": In transactions that set severity critical, run action X
• Default "y": In transactions that set severity critical, run action Y
• Rule 1: When value > 90 then severity critical

In this scenario, both defaults will apply to Rule 1, effectively creating two instances of rules within the gateway:

• Rule 1, defaults "x": When value > 90 then severity critical, run action X
• Rule 1, defaults "y": When value > 90 then severity critical, run action Y

Both the rules will apply to the rule targets, and the "Show Rules" output for a given target will show both rules. Because both rule transactions set severity, only one will run. If the priority group is the same in each case, then the choice of rule is unpredictable.

To avoid this ambiguity, if multiple sets of defaults apply to a group of rules, each set of defaults should specify a different set of contexts. As long as these contexts do not overlap and as long as none of the rules to which they apply specifies any contexts, at most one rule will apply to each item.

Alternatively, (if some overlap of contexts is unavoidable) each set of defaults could specify a different priority group, or they could have different (and non-overlapping) active times. As long as none of the rules specifies a priority group (or an active time), this will ensure that at most one rule applies at a time.

Using functions with ranges of items

Some useful functions exist to help summarise data, such as average, total, max, min, count and standard deviation. As well as normal parameters, these operate on wildcard paths which return sets of items rather than a single item.

value total(wpath "line throughput" value)

or

value count(wpath "lines up" value)

These functions can take multiple parameters if required:

value average(path "cell a" value, path "cell b" value)

or

value average(wpath "line throughput" value, path "additional" value)

Available functions are listed in the Statistical Functions configuration section, and include maximum, minimum, average, total, count, standard deviation and rate.

Using historic time-based functions

Using the normal rule syntax, previous values can be compared against the current value. The time-based functions allow a summary to be created from a period of time.

For example, to populate a cell with the highest the CPU has been over the last minute:

value maximum(path "cpu" value for "1 minute")

The "1 minute" refers to a named History Periods. These are defined at the top level of the rules section - at the same place as rule groups.

All the functions mentioned under sections Statistical Functions and Duration-Weighted Statistical Functions can be used in this manner.

Local Variables

In addition to Managed Entity variables, temporary local variables can be set and then accessed using the same syntax:

set $(temp) value + 1
if $(temp) > 0 then ...

Local variables override Managed Entity variables. Once a rule sets a variable with the same name as a Managed Entity variable then the Managed Entity variable will no longer be accessible from within that rule. Local variables have global scope but cease to exist at the end of each rule evaluation.

Conditions

It is also possible to use the conditional functionality of rules when setting values. For example:

if path "other cell" value > 10 then
value path "other cell" value
else
value path "third cell" value
endif

Setting other properties in addition

Although this section has dealt simply with values, it is possible to set other properties at the same time in the same rule, just like with normal rules. e.g.

value path "other cell" value
severity path "other cell" severity

if true then
value path "other cell" value
severity path "other cell" severity
endif

History Periods

Distribution Lists

Each notification effect has three distribution lists: To, CC (Carbon Copy), and BCC (Blind Carbon Copy). When adding a user or user-group to a distribution list the information type to add can be specified, this corresponds to the generic user information defined in the Authentication Technical Reference. Be default this will be "Email" but any information can be specified and passed to the effect, and because an effect can be a user defined script any information can be interpreted.

Effects

When a notification fires it calls an effect (as defined in the effects section), passing it information about the alert, the distribution lists, and about the data-item the alert has been created for.

Severity Transition Example 1

1. OK -> Warning
   • WARNING alert fires. Repeats and escalations scheduled.
2. Warning -> Critical
   • WARNING alert repeats and escalations cancelled.
   • CRITICAL alert fires. Repeats and escalations scheduled.
3. Critical -> Warning
   • CRITICAL alert repeats and escalations cancelled.
   • CRITICAL clear fired.
   • WARNING alert fires (at first escalation level and with a repeat count of 0). Repeats and escalations scheduled.
4. Warning -> OK
   • WARNING alert repeats and escalations cancelled.
   • WARNING clear fired.

Severity Transition Example 2

5. OK -> Warning
   • WARNING alert fires. Repeats and escalations scheduled.
6. Warning -> Critical
   • WARNING alert repeats and escalations cancelled.
   • CRITICAL alert fires. Repeats and escalations scheduled.
7. Critical -> OK
   • CRITICAL alert repeats and escalations cancelled.
   • CRITICAL clear fired.
   • WARNING clear fired.

Example 1: Repeating

While hierarchy is inactive, repeats stop. When hierarchy goes active, repeats start again with a repeat count of zero.

When the hierarchy returns to active, a notification is not fired. This is because the alert for the cell going critical already fired during the previous active period.

Example 2: Escalations

When the cell becomes critical in the inactive period, an alert is not fired. The time when the cell severity changed is recorded and included in the Resume notification.

The alert is queued until active, and the escalation chain begins only in the active period.

A Resume notification is fired because the cell became critical during the inactive period. The notification includes the timestamp of the severity change.

Example 3: Resetting Escalations

The escalation notification fires after the escalation interval, and is cancelled when the alert is suspended on entering the inactive period.

When the hierarchy returns to active, a notification is not fired. This is because the alert for the cell going critical already fired during the previous active period.

When the hierarchy goes active, the escalation chain is restarted. A notification fires if the escalation interval elapses and the cell is still critical.

Example 4: Clears

When the cell is critical on entering the inactive period, a Suspend notification is fired.

When the cell returns to OK during the inactive period, a Clear notification is not fired.

Example 5: Cells Changing State While Alert Is Suspended

When the cell is critical on entering the inactive period, a Suspend notification is fired.

While the alert is suspended, notifications are not fired.

When alert is resumed, cell state is re-evaluated and notifications are restarted. A Resume notification is fired because the cell became critical during the inactive period.

Clear is later fired when cell returns to OK.

Example 6: Intervals That Span Entire Suspended Period

When an alert is suspended, all scheduled repeats and escalations are cancelled. Even if the alert is resumed before they would have fired, a notification does not occur.

When the hierarchy returns to active, a notification is not fired. This is because the alert for the cell going critical already fired during the previous active period.

When the hierarchy goes active, scheduled repeats and escalations are restarted. A notification fires if the escalation interval elapses and the cell is still critical.

Show Alerts

This command shows all the alerts that are applicable to a selected data-item. Each applicable alert's configuration (as configured in the Gateway Setup Editor) is presented in a formatted output to the Active Console Output pane. This command applies only to data view data cells.

Evaluate Alerts

This command shows all the data-items in the system that currently matches the alert criteria in that part of the hierarchy tree. It can be executed by right clicking on an Alert in the Alerting hierarchy in the Gateway Setup Editor. The matching criterion is as specified in the hierarchy tree. Example, if you configure an Alerting hierarchy by managed entity name, and configure an Alert underneath this Alerting hierarchy with a managed entity name to report all Warning and Critical alerts, only this managed entity (or its descendant data-items) that currently have warning or critical severity will be displayed.

For each data-item, it presents the following information:

This command is useful in cases where the first severity change caused the Alert to be raised and notified to users but subsequent severity changes might go un-notified because they only keep the Alert valid at that point in time. One can find all the severity changes in all managed variables that currently match the Alert criteria.

Configuration Settings

Effects are configured in the Effects top-level section of the gateway setup. Configuration consists of a list of effect definitions, each of which contains the minimum required configuration for their type. Each effect is identified by a user-supplied name, which must be unique among all other effect to prevent ambiguity, as effects are referenced by name.