Advanced Features

Auditing Copied

The auditing feature of gateway allows logging of user interactions with the gateway. These logs allow historical tracking of gateway functions and can be used to reconstruct a sequence of events or for security planning.

Auditing works best in conjunction with a configured authentication section. If no authentication is used, then users connecting to gateway will be assigned incrementing usernames “ActiveConsole1”, “ActiveConsole2” and so on. Authentication forces the user to login with their username, making it easier to determine which user has performed which actions.

Audit information Copied

The following events are recorded by the gateway’s auditing feature:

• User log-on and log-off events (this includes connections from Webslinger, Open Access server and Importing Gateways)
• Logon failure due to rejected authentication (not logons which fail due to inability to establish a connection)
• Command execution, where the command has been executed interactively (not as a scheduled command) and is run on the gateway or a probe (run location not set to ‘Client’).
• REST interface requests, whether to execute a command or to list available commands.
• Failed command execution, where the command has been executed interactively via the gateway.
• Gateway setup change events, including failed setup change requests
• Gateway start/stop events

Each record contains the following information:

Audit record ID

A globally unique identifier for the audit record, used to link it to any extended autit information (see next section).

Datetime

Time (to nearest second) when the audited action occurred using the local time on the Gateway, in ISO 8601 basic format.

Audit event

The type of action audited. For example “Execute command”, “Login”.

Username

Username of the user who performed the action.

Workstation IP

IP address the user is connecting from.

Command

Name of the command run (if any). For example “/SNOOZE:manualAllMe”.

Fail reason

If the action failed, a brief reason for the failure. For example “Rest Service not enabled on insecure connections”, “No permission”.

Data item path

Unique path to the data-item the command was run against (if any).

Target Managed Entity

The managed entity associated with the action (if any).

Arguments

Arguments supplied by the user or system when executing a command; the file name of a setup file which is changed; or the comment supplied when setup is applied.

Module

The Gateway component which generated the record. For example, “UserManager”, “CommandManager”, “RestService”.

Gateway name

The name of the gateway which generated the record.

Extended audit information Copied

Certain auditable actions can lead to additional audit details being recorded with reference to that event. Currently this is done for commands executed on dataviews produced by certain plug-ins.

The types of commands that result in extended audit details and the exact information collected would depend on the plug-in. The individual reference guides of plug-ins that return extended audit details explain what details are recorded.

Format Copied

The collected audit details have a generic format in the form of one or more tuples. Each tuple gives the value of a named field for a named “item” related to the particular auditable event. What constitutes an item depends on the component that is providing the audit details.

For example, consider an FKM sampler where a command that affects two triggers is executed. In this case information regarding these two triggers is provided by FKM as audit details. For each trigger, the associated message and input file line are sent as values. In the audit details format, each trigger is represented as a separate “item” and the message and file line for each trigger is represented as a field-value pair associated with that item. E.g.:

Note

The above example may not represent the actual information that FKM returns as extended audit details.

The collection of extended audit details is enabled using the collectExtendedDetails setting. By default, it is not enabled.

Limits Copied

In the case of commands, audit details are not be collected for commands that apply above the dataview level. For example, if a command is executed against a managed entity, audit details are not queried and collected from the various dataview that come under that managed entity.

Linking to Main Audit Record Copied

In each of the audit outputs the main audit records are written independently of audit details records (for example, if the database audit output is being used they are written to separate database tables.)

In order to be able to match an audit details record to its main audit record, each auditable event has a unique id. The same audit record id is written as part of any corresponding audit details records.

Data Volume Copied

Depending on the auditable event in question the volume of information gathered as extended audit details has the potential to be high. For example accepting all files on an FKM dataview that is currently displaying a large number of triggers could lead to a correspondingly large amount of audit details that will be written to the configured audit outputs. This should be considered when deciding whether to enable this feature.

Audit Configuration Copied

Audits logs can be written to three different locations; to the gateway database, to file using a flat format or to file using an XML format (see Audit File Format). Output can be written to several locations simultaneously and also to multiple files using the same format if required.

Audit logging configuration is placed in the auditOutputs top-level section. This section contains a list of auditOutput settings, each of which specifies a single location to place the audit log entry. These locations can be any of the four as listed above.

Note

Since the database used for audit-logging is the gateway database multiple outputs to the database will generate a setup validation warning.

The settings for audit logging are described below:

auditOutputs Copied

The auditOutputs top-level section contains the configuration for the audit logging feature. This section can contain any number of auditOutput configuration sections.

auditOutputs > collectExtendedDetails Copied

Turning this on will result in the gateway collecting extended audit details when certain auditable actions are invoked. See Extended Audit Details.

Mandatory: No

Default: false

auditOutputs > name Copied

Each audit output section should be uniquely named.

auditOutputs > auditOutput Copied

Each audit output section contains the configuration for a single location, to which every audit event in the gateway will be sent. The configuration must contain exactly one of the database, xmlFile or flatFile sections described below.

auditOutputs > auditOutput > database Copied

Specifies that audit information should be written to the gateway database. This section contains no database connection details, since database logging is configured in the database top-level section (see Database Loggin).

auditOutputs > auditOutput > database > maxDescriptionLength Copied

Specifies the maximum length of the description field in the audit_table, where audit data is logged to. The gateway will automatically truncate descriptions which are too long to this length, to ensure that they are logged.

Clients should only need to change this setting if they have extended the description field length above the default length in the database schema provided with gateway.

Mandatory: No

Default: 250

auditOutputs > auditOutput > xmlfile Copied

Writes audit information to the specified file, using XML formatting.

auditOutputs > auditOutput > xmlfile > filename Copied

Specifies the file to log to. This can be an absolute path, or a relative path (from the gateway working directory).

auditOutputs > auditOutput > xmlfile > maxSizeKb Copied

Optional setting to control the maximum size of the audit log file, measured in KB. This should be a non-negative integer value. When the file grows beyond this size, it is renamed to <filename>.old (deleting any previous file with this name) and a new file is created.

auditOutputs > auditOutput > xmlfile > archiveScript Copied

Optional script that is run when the log file rolls over (when it exceeds the maximum size). The script is passed <filename>.old as the first argument, and can then perform any actions on this file as required.

auditOutputs > auditOutput > flatFile Copied

Writes audit information to the specified file, with one line per entry.

auditOutputs > auditOutput > flatFile > filename Copied

The filename of the log file.

auditOutputs > auditOutput > flatFile > maxSizeKb Copied

Optional setting to control the maximum size of the audit log file, measured in KB. This should be a non-negative integer value. When the file grows beyond this size, it is renamed to <filename>.old (deleting any previous file with this name) and a new file is created.

auditOutputs > auditOutput > flatFile > archiveScript Copied

Optional script that is run when the log file rolls over (when it exceeds the maximum size). The script is passed <filename>.old as the first argument, and can then perform any actions on this file as required.

Basic Configuration Copied

The following shows how to configure an audit output in a flat file called “audit.log”, with a rollover size of 102400 kb for the log file and a script to run after the log file has been rolled over.

1. Select the Audit Outputs node in the Navigation tree. This results in the auditOutputs pane being displayed as the main pane.
2. Select the “Add New” button to create a new audit output instance. A row should appear in the main pane, with a default type of “database” as the value of its first field.
3. Change the type from “database” to “flatFile”. This causes a “Flat file…” to appear as the second entry in the row.
4. Select the “Flat file…” entry. This should result in a dialog box being displayed with options to further configure the audit output instance.
5. Specify a name for the flatFile, the maximum size and a suitable archive script. Then select OK. This should result in the same view as depicted for step 4.

Audit File Format Copied

Audit records can be written to file in one of the two formats described below. This section only describes the formats - for the meaning of each of the fields in audit records and extended audit records, please see the sections Audit information and Extended audit information.

XML Format Copied

The root element for the document is and under this each auditable event is written as a single element. For example:

<?xml version="1.0" encoding="iso-8859-1"?>
<auditRecords>
<auditRecord>
<auditRecordId>2af649a6-a162-4e6a-911e-7667b55a5925</auditRecordId>
<time>20161109T101346</time>
<auditEvent>Setup file change</auditEvent>
<username>ActiveConsole1</username>
<workstation>192.168.141.1</workstation>
<arguments>
<argument>
<name>File</name>
<value>gateway.setup.xml</value>
</argument>
</arguments>
<module>SetupManager</module>
<gateway>AuditTest</gateway>
</auditRecord>
</auditRecords>

If the extended audit details mode is enabled, then when any auditable event produces extended details these are written as an element again under the top level element.

The element can be tied back to the element that represents the auditable event using the field which is a unique identifier representing the particular auditable event.

E.g.:

<?xml version="1.0" encoding="ISO-8859-1"?>
<auditRecords>
<auditRecord>
<auditRecordId>0be51672-7d67-4f39-b6ca-6f46befe4ee4</auditRecordId>
<datetime>20161109T135926</datetime>
<auditEvent>Execute command</auditEvent>
<username>ActiveConsole2</username>
<workstation>127.0.0.1</workstation>
<command>/PLUGIN:FKM:acceptFile</command>
<dataItem>/geneos/gateway[(@name=&quot;AuditTest&quot;)]/directory/probe[(@name=&quot;p&quot;)]/managedEntity[(@name=&quot;m&quot;)]/sampler[(@name=&quot;fkm&quot;)][(@type=&quot;&quot;)]/dataview[(@name=&quot;fkm&quot;)]/rows/row[(@name=&quot;fkm.log&quot;)]/cell[(@column=&quot;status&quot;)]</dataItem>
<targetManagedEntity>m</targetManagedEntity>
<arguments>
<argument>
<name>Arg_0</name>
<value>fkm.log</value>
</argument>
</arguments>
<module>CommandManager</module>
<gateway>AuditTest</gateway>
</auditRecord>
<auditDetails>
<auditRecordId>0be51672-7d67-4f39-b6ca-6f46befe4ee4</auditRecordId>
<auditDetail>
<item>fkm.log#warning00000</item>
<field>message</field>
<value>uhoh</value>
</auditDetail>
<auditDetail>
<item>fkm.log#warning00000</item>
<field>line</field>
<value>warning This is a warning.</value>
</auditDetail>
<auditDetail>
<item>fkm.log#warning00001</item>
<field>message</field>
<value>uhoh</value>
</auditDetail>
<auditDetail>
<item>fkm.log#warning00001</item>
<field>line</field>
<value>warning This is a warning too.</value>
</auditDetail>
</auditDetails>
</auditRecords>

Flat Format Copied

In the flat format audit information is written as comma-separated values. Each line in the file represents either an audit record or a single item of audit detail. To distinguish between these two types of lines, the first item in the line is the string RECORD or DETAIL.

Each audit record representing a single auditable event is written as follows:

RECORD,<audit record id>,<datetime>,<username>,<workstation>,<module>,<managed entity name>,<description>,<gateway name>

E.g.:

RECORD,7592a7a4-17d2-44e5-96a8-9b17d78b9af7,Wed Nov  9 14:03:43 2016,ActiveConsole1,192.168.141.1,SetupManager,,Setup file change File:gateway.setup.xml,AuditTest
RECORD,95eb18f1-55a4-4093-a6a7-4e9872499beb,Wed Nov  9 14:03:43 2016,Administrators,192.168.142.1,SetupManager,,New Setup Applied,AuditTest

Note that the ‘description’ field combines information from the ,,, and elements of the XML format.

If the extended audit details mode is enabled then, when any auditable event produces extended details, these are written as one or more audit detail lines. An audit detail line has the following format:

DETAIL,<audit record id>,<item>,<field>,<value>

Audit detail lines can be tied back to the main audit record using the audit record id which is a unique identifier representing the particular auditable event.

E.g.:

RECORD,2b1d5715-8ad4-4b30-b10a-0aad685a8286,Tue Jan 21 11:36:19 2014,ActiveConsole2,127.0.0.1,CommandManager,m,/PLUGIN:FKM:acceptFile DataItem:/geneos/gateway[(@name="AuditTest")]/directory/probe[(@name="p")]/managedEntity[(@name="m")]/sampler[(@name="fkm")][(@type="")]/dataview[(@name="fkm")]/rows/row[(@name="fkm.log")]/cell[(@column="status")] Arg_0:fkm.log,AuditTest
DETAIL,2b1d5715-8ad4-4b30-b10a-0aad685a8286,fkm.log#warning00000,message,uhoh
DETAIL,2b1d5715-8ad4-4b30-b10a-0aad685a8286,fkm.log#warning00000,line,warning This is a warning.
DETAIL,2b1d5715-8ad4-4b30-b10a-0aad685a8286,fkm.log#warning00001,message,uhoh
DETAIL,2b1d5715-8ad4-4b30-b10a-0aad685a8286,fkm.log#warning00001,line,warning This is a warning too.

Gateway Hooks Copied

Overview Copied

Gateway hooks provide a way for clients to fire custom programs or scripts when certain important gateway events occur. We call these programs hooks as they are hooked in to the gateway process. The hooks that are provided interact with the way that the gateway manages its setup. They provide the ability to link to Version Control Systems (VCS) and to provide extra validation of the setup based upon business rules. There are two gateway hooks;

• Validate-setup: This hook is fired during the gateway validation process. The hook can generate extra validation issues, that the gateway will merge with any internally generated issues. These will then either be sent back to the Gateway Setup Editor or printed to the Gateway log, dependent on the reason that the gateway triggered the validation step.
• Post-setup-apply: This hook is fired after the gateway has finished applying a new setup. Information about the setup files used to run the gateway together with information about the change that has occurred are provided to the hook. The expected use of this hook is to pass the files to a VCS. The hook can also pass back to the gateway version numbers for the setup files in use.

Setup Copied

The hooks are all located in a single hooks directory and have pre-defined names. The location of the hooks directory is set at gateway start up by use of the -hooks-dir command line option. The names of the two hook files are “validate-setup” and “post-setup-apply”. If the hook file is not present or the hook directory has not been defined, then the hook will not be fired by the gateway. So if the user runs the gateway with the following command;

gateway2.linux -hooks-dir /opt/geneos/gw/hooks

The gateway will look for the following 2 hook files;

• /opt/geneos/gw/hooks/validate-setup
• /opt/geneos/gw/hooks/post-setup-apply

If a relative path to the hooks directory is provided, that is relative to the working directory of the gateway.

Validate-Setup Copied

If the gateway has a hooks directory defined and the file “validate-setup” is present in the directory then the hook will be fired every time the gateway attempts to validate a setup. On completion this hook provides a list (which can of course be empty) of validation issues to the gateway. These validation issues merged with the issues generated by the gateway. They are then reported to back to the user via the log file or the Setup Editor.

Example Copied

The key use of a validate-setup hook is to add business validation on top of the application validation provided by the gateway. In the resources directory you will find an example validate-setup hook. It is located in resources/hooks/examples/validate-setup. This is a python script that will apply the following extra rules to the setup;

An error issue will be raised for each FKM plugin that is created where the isPCREMode flag has not been set to true. A critical issue will be raised if the httpConnectionRequirements has been set in any way If this script is used then setting httpConnectionRequirements will cause the gateway to add a Critical issue to the Setup Validation Issues list. The gateway will not run a setup that has such an issue, so the setup will not be applied if this xml tag is added using the GSE. If the setup is edited by hand to have this issue and then the gateway started it will refuse to start.

If an FKM plugin is created and no other settings are added then, validating the setup will return an error in the GSE. Saving the setup will create a FKM sampler unless the -max-severity command line parameter set to a none or warn. See the Gateway Command Line options section for more details on this parameter.

Note

The issues created by the hook, when seen in the GSE can be clicked upon to take the setup editor to the errant item, just as any other validation issue.

Triggering the hook Copied

The validate-setup hook will be triggered at the following times;

• Gateway start up
• Validation request sent from the Gateway Setup editor to the gateway
• Setup saved from the Gateway Setup editor to the gateway
• Reload of setup due to defined Gateway Reload intervals
• Reload of setup due to gateway receiving a USR1 signal
• Setup tested using the -validate command line option

Data passed to the hook Copied

The validate-setup hook is passed one piece of information, the location of a temporary file containing the setup of the gateway, using the _SETUP environment variable. This file contains a single piece of XML which is the merged setup that the gateway generates when it combines included setup files into a single XML setup. This is similar to the XML as the user can obtain by running -dump-xml setup command line option. The difference being that the command line option gives the merged setup of the files on disk, while the XML provided to the hook may use XML setup sections from the gateways memory prior to being saved to disk. However, it is not necessary for the hook to understand the origin of the XML. Each setup provided to the hook should be processed in the same way and the results returned back to the gateway.

Note

The merged XML nodes will have additional attribute to specify which setup file this node came from. A node coming from main file will have attribute main="true" whereas a node coming from include file will have attribute include="/path/to/include/file/as/specified/in/the/setup". It is implicit that child nodes not having such attribute inherit it from their ancestor node. This attribute can be used to do pre-setup validation such as assure that all FKM samplers come from a particular file. For illustration, see Merged Setup Example.

The following environment variables have a blank value passed on start up and a non-GSE triggered setup change:

Data passed back by the hook Copied

The hook passes data back to the gateway by printing an XML document to its standard output. This XML document needs to conform to the schema provided in hooks/schema directory (resources/hooks/schema/validate-setup-result.xsd). There is also an example of the output that could be generated in the examples directory (resources/hooks/examples/validate-setup-result.xml) shown below:

<validation>
<issues>
<issue>
<severity>Error</severity>
<path>/gateway/samplers/sampler[2]</path>
<message>Middleware Requirements: FKM must use PCRE</message>
</issue>
<issue>
<severity>Error</severity>
<path>/gateway/samplers/sampler[3]</path>
<message>Middleware Requirements: FKM must use PCRE</message>
</issue>
<issue>
<severity>Critical</severity>
<path>/gateway/operatingEnvironment/httpConnectionRequirements</path>
<message>Middleware Requirements:  Access to ORB pages is forbidden!!!!</message>
</issue>
</issues>
</validation>

As can be seen, the XML is simply a list of validation issues. Each issue should contain the following information:

• A Message : A human readable description of the issue. This will be shown to the user in the GSE and printed in the Gateway setup.
• A Severity : The severity of the issue being reported. This can take one of the following values; - None: This is an information message and does not affect the level of severity of the setup - Warn: This is a warning message. Adding a message of this type will increase the severity of the setup to Warning if it is not currently warning or greater - Error: This is a warning message. Adding a message of this type will increase the severity of the setup to Error if it is not currently warning or greater - Critical: This is a warning message. Adding a message of this type will increase the severity of the setup to Critical if it is not currently warning or greater
• An XPath : This is the XPath to an item in the setup that the issue applies to. This should be an XPath a single data item.

In the above example 3 issues are passed back, 2 samplers have been marked as in error because they are FKM samplers but do not have the isPCREMode set. The third issue is a critical issue cause because an httpConnectionRequirements section has been defined in the setup.

The path to the problem items in the Setup XML is provided.

Note

This path is not the same as the path to the same on the running gateway.

The XPath provided is applied to the merged setup provided by to the hook, so that the gateway can link the issues to the offending items in the setup. This then allows the setup editors using the GSE to select an issue and be taken to the offending item.

The hook does not need to pass data back. If nothing is passed back on the standard output, and the hook terminates with an zero error code. The gateway will treat this as if an XML document had been returned with an empty issues list.

If this hook is present, the gateway requires that is run successfully every time that it is fired. If the hook does not run successfully that the gateway will generate a Critical validation issue and ignore any output from the script. If the failure occurs at start up then the gateway will not start. If the failure occurs when the gateway is attempting to apply a new setup, then the gateway will refuse to apply the new setup and keep running on the existing setup. There are 4 types of failure:

• The hook terminates incorrectly (crash)
• The hook returns a non-zero error code
• The hook fails to terminate in under 2 minutes
• The hook passes back data on the standard output which the gateway cannot parse correctly.

The default timeout of 2 minutes can be changed to a different value using the command line option:

-hooks-timeout

Examples:

gateway2.linux_64... -hooks-timeout 90 #90 second timeout on Gateway validation hooks
gateway2.linux_64... -hooks-timeout 3m #3 minute timeout on Gateway validation hooks

Note

Settings greater than the default of 2 minutes will result in a warning that Gateway performance may be degraded.

Post Setup Apply Copied

If the gateway has a hooks directory defined and the “post-setup-apply” is present then the hook will be run every time the gateway applies a new setup correctly. The script is provided with a list of the setup files used by the gateway. On completion it can pass back a list of version numbers for those files. The version numbers of the currently active files are then printed to the gateway log and updated in the Gateway IncludesData plugin (if configured).

Triggering the hook Copied

The post-setup-apply hook will be triggered every time that the setup is reapplied;

• Gateway start up
• Successful setup save from the Gateway Setup editor
• Successful reload of setup due to defined Gateway Reload intervals
• Successful reload of setup due to gateway receiving a USR1 signal

Data passed to the hook Copied

The gateway passes 4 pieces of information to the post-setup-apply hook. Each stored in an environment variable shown below;

The schema for the _FILES XML block can be located in hooks/schema directory (resources/hooks/schema/post-setup-apply-files.xsd).There is also an example of the output that could be generated in the examples directory (resources/hooks/examples/post-setup-apply-files.xml) shown below;

<setup>
<files>
<file>
<main />
<path>gateway.setup.xml</path>
</file>
<file>
<include>inc1.xml</include>
<path>inc1.xml</path>
<setupPending />
<gseChanged />
</ file>
<file>
<include>http://mwserver/users.xml</include>
<remote />
</file>
</files>
</setup>

The example above shows 3 setup files. The first file is the main file and it is located at gateway.setup.xml (relative to the working directory of the gateway). The lack of an setupPending node indicates that the version of the file on disk is the same as the version being used by the gateway. The second file is an include file that is stored locally on disk. The presence of the setupPending node indicates that the version on disk is being currently ignored by the gateway. Probably because of the use of active times on the include files. The gseChanged node indicates that this hook was fired because the user saved a new setup by saving from the GSE to the gateway and that this file was changed on this occasion. Finally the third file is remote setup file that is downloaded on demand from a web server. No path is provided as it is not saved locally.

Summarizing, for each file the following information is provided;

• name: The include file name is provided in a block. The main file does not have an include block, instead it has a

<setup>
<files>
<file>
<main/>
<path>gateway.setup.xml</path>
</file>
</files>
</setup>

<setup>
<files>
<file>
<path>gateway.setup.xml</path>
<version>35eda74</version>
</file>
<file>
<path>inc1.xml</path>
<version>35eda74</version>
</ file>
<file>
<path>inc2.xml</path>
<version>35eda74</version>
</file>
</files>
</setup>

<http://server.company.com:7039/knowledgeBase/default.html>