Message Tracker Plug-in Technical Reference

adapters > adapter > name

The name of the adapter

Mandatory: Yes

adapters > adapter > checkpoint

The name of the checkpoint that the messages this adapter reads are associated with.

Mandatory: Yes

adapters > adapter > source

The source of the data for this adapter. Currently only files are supported as data sources

Mandatory: Yes

adapters > adapter > source > file > filename

This setting specifies that the file to read message data from. The value of this setting specifies the location of the file. Relative paths will be evaluated from the Netprobe working directory.

If a filename contains wildcard characters, then the adapter will automatically check for creation of newer files matching the wildcard pattern. When a new file is detected (provided the current file has been scanned to the end) the adapter will then switch to monitor the newer file. The file check occurs only so often to stop the system from overusing resources. (See fileReadControl > newFileCheckWaitTime)

There are two types of wildcards specifications which can be used:

1. Globbing using * and ? characters, as typically used in a UNIX shell session. e.g. apache*.log
2. Filename date generation using the <today> dates-stamping within the filenames. e.g. app<today%Y%m%d>.log

Note: <yesterday> and <tomorrow> dates-stamping are not currently supported.

Mandatory: Yes

adapters > adapter > source > file > matchAllFiles

This setting makes the adapter match multiple files instead of the latest file. This is useful e.g. if a fix engine stores conversations per file instead of in one file.

Mandatory: No

adapters > adapter > source > file > multilineMessages

This setting specifies that the messages span multiple file lines. If this is the case a regular expression that can be used to identify lines that start the message must be provided. The end of the message is then detected by additional optional settings.

Mandatory: No

adapters > adapter > source > file > multilineMessages > startPattern

Specifies the regex used to detect the start of a message. The line containing this pattern will be added to the message contents, and all subsequent lines until the end of the message. The end of the message is determined either by the endPattern regex if specified, or by detecting the start of a new message using this start pattern.

Note: The start pattern applies to a single line that starts the message. Character sequences spanning lines cannot be matched.

Mandatory: Yes

adapters > adapter > source > file > multilineMessages > messageEnd

Configures the end-of-message rules for the file. The end of the message is detected either by a file line which matches a configured endPattern, or by finding a new message with the startPattern.

When using only a startPattern, if a new message is not found then the current partial message will be held for the time specified by the maxWaitTimeMs setting for more data to become available, before then being processed. This is to prevent issues when the adapter has read to the end of a log file while the logging process is still writing data.

adapters > adapter > source > file > multilineMessages > messageEnd > maxWaitTimeMs

The maximum wait time specifies how long the adapter will wait (in milliseconds) for more data of a partial message before processing it.

A partial message can occur when the adapter reads to the end of a log file before the logging process has finished writing. This can also happen for the last message in the file, since there will not be a following "start message" line which matches the startPattern.

adapters > adapter > source > file > multilineMessages > messageEnd > endPattern

A regex which indicates the end of a message. The file line matching this regex will also be added to the message.

Note: The end pattern applies to a single line that ends the message. Character sequences spanning lines cannot be matched.

Mandatory: No

adapters > adapter > source > glLogFile > filename

Specifies the location of the GL P10 log file to read messages from. Globbing and wildcards can be used, as in the file source > filename setting.

Mandatory: Yes

adapters > adapter > source > glLogFile > truncatedLines

This setting informs the file reader that the GL log file contains truncated lines (i.e. lines are written out to a fixed line width of 80 characters), which is produced by some versions of GL. An example of this output is shown below.

Note: This setting does not perform auto-detection, therefore only enable it if the file is known to be truncated.

-10 06:31:35:371 (00.TRC.00087) DataReply : 2919
                 DataHead {type: 'X' - [0:0]->[0:0] (0) }
                 <---------------------------------->                 <-------------- REPLY ------------->                 <---------------------------------->                 Fixed field = { key : 0 - chainage : 0 - noUti : 0 - typeMessage : 79 'O'
                                 typeReply : 74 'J' - index : -1 - nbRequestReply : 1 - classOrder : 79 'O'
                                 command : 32 '.' - exchangeName : 'SEHK' - exchangeLabel : ''
                                 GLID : '0008-1-01-01' - mnemo : 'ETS' }
<... message continues ...>

adapters > adapter > filters

The adapter applies a set of filters to the messages . Messages that do not pass all the filters are ignored.

Mandatory: No

adapters > adapter > filters > message

This specifies a single regular expression that is applied to the entire message. The filter can be inverted. (See Message Filtering

Mandatory: No

adapters > adapter > filters > tags > tag

Multiple tag filters can be defined on an adapter. The tag name and the regular expression both have to be supplied. The filter can be inverted. (See Message Filtering)

Mandatory: No

adapters > adapter > formatType

This specifies the type of message to read. ITRS provide a supported set of format types. Each one has its own documentation. It can be specified as data, or as a static variable which can then be shared by more than one adapter.

Mandatory: Yes

adapters > adapter > formatType > messageTrackerFormatType

A reference to a Message Tracker Format Type static variable.

Mandatory: No

See the Gateway 2 Reference Guide for information on configuring Message Tracker Format Type static variables.

adapters > adapter > formatType > data

Specifying a format type without using a static variable.

Mandatory: No

adapters > adapter > formatType > data > timestamp

This specifies the location and format of the message timestamp.

Mandatory: Yes

adapters > adapter > formatType > data > timestamp > regexPattern

This specifies the location of the timestamp with the message. Data extracted by the applying the regular expression to the message and extracting the first group (the data the matches the regular expression within the first bracket) from the message. The system uses perl regular expressions.

More information can be found at http://www.perl.com/doc/manual/html/pod/perlre.html

Mandatory: Yes

adapters > adapter > formatType > data > timestamp > format

This specifies the format of the timestamp. The Time Format codes are available in the Time Formatting Codes section at the end of this document.

Mandatory: Yes

adapters > adapter > tagMapping

This specifies a set of mappings from the tags provided by the formatType to a normalised message. This mapping is composed of two parts: IDs and attributes.

It can be specified as data, or as a static variable which can then be shared by more than one adapter.

Mandatory: Yes

adapters > adapter > tagMapping > messageTrackerTagMapping

A reference to a Message Tracker Tag Mapping static variable.

Mandatory: No

See the Gateway 2 Reference Guide for information on configuring Message Tracker Tag Mapping static variables.

adapters > adapter > tagMapping > data

Specifying a tag mapping without using a static variable.

Mandatory: No

adapters > adapter > tagMapping > data > IDs > item

This creates a named ID from a set of tags. These tags will be user defined if the format type selected was Regex. The tags will be defined by the format type reader for other format types. For example, when using Fix the tags will be the numeric FIX tags. Please see the ITRS provided documentation for the format type being used (see Data Normalization). An ID should be unique for the message. Two messages are considered to be the same message if they share an ID with the same name and the same value.

Mandatory: No

adapters > adapter > tagMapping > data > IDs > item > format

This allows the tags to be combined together in a user specified way. If the format is not specified then the tag values are placed together space separated (see User defined Tag combining

Mandatory: No

adapters > adapter > tagMapping > data > attributes > item

This creates a named attribute from a set of tags. These tags will be user defined if the format type selected was Regex. The tags will be defined by the format type reader for other format types. For example, when using Fix the tags will be the numeric FIX tags. Please see the ITRS provided documentation for the format type being used (see Data Normalization)

Mandatory: No

adapters > adapter > tagMapping > data > attributes > item > format

This allows the tags to be combined together in user specified way. If the format is not specified then the tag values are placed together space separated (see User defined Tag combining)

Mandatory: No

adapters > adapter > realTimeTracking

If this setting is present (even if no sub-settings are configured) then the adapter will pass the message to the real time checkpoint as defined.

adapters > adapter > realTimeTracking > trackingData > description

This setting specifies which attribute or ID to use as a message description. The message description is used in the Lost Messages view. If not set then it will default to the first message ID defined on the current checkpoint.

adapters > adapter > realTimeTracking > trackingData > category

This setting specifies which attribute to use to categorise the Latency messages. The category description is used in the Latency view. If not set then the messages will be uncategorised.

Mandatory: No

adapters > adapter > realTimeTracking > remotePlugin

If this setting is present if the adapter is reading messages but sending them to a remote real time checkpoint. If this is the case the Host, Managed entity and sampler of the plug-in hosting the real time checkpoint must be provided.

Mandatory: No

adapters > adapter > realTimeTracking > remotePlugin > host

Host and listen port of the Netprobe that is providing the real time checkpoint. The form of the input is host:port

Currently the Message Tracker plugin does not support connecting via HTTPS, so cannot send messages to a netprobe that has been started with the -secure flag.

Mandatory: Yes (if remotePlugin exists)

adapters > adapter > realTimeTracking > remotePlugin > managedEntityName

Name of the managed entity that is providing the real time checkpoint.

Mandatory: Yes (if remotePlugin exists)

adapters > adapter > realTimeTracking > remotePlugin > samplerName

Name of the sampler that is providing the real time checkpoint.

Mandatory: Yes (if remotePlugin exists)

adapters > adapter > record

If this is setting is set to true then the message will be logged to the database specified in the database section.

realTimeCheckpoints > checkpoint > name

The name of the real time checkpoint that the plug-in should monitor. This name must be unique for all checkpoints which will be running on a particular Netprobe (i.e. across different samplers). This is so that external adapters (which are configured with the checkpoint name) send data to the correct checkpoint.

realTimeCheckpoints > checkpoint > children

The name of the child checkpoints that this real time checkpoint expects messages to go to.

An optional host can be provided, if the checkpoint resides on a different probe. Messages are passed between probes using Geneos's EMF protocol. If the probe running the plugin is started with the secure flag, then the communications will be sent over a secure channel. It is essential that all probes that are connected together via checkpoints use the same transport. Thus they should all use secure communications (set the -secure flag on start up) or they should all use open communications.

realTimeCheckpoints > checkpoint > parents

The name of the parent checkpoints that this real time checkpoint expects messages to come from.

realTimeCheckpoints > checkpoint > messageDelivery

The message delivery expectations of the message

average > interval

The period of time over which to calculate average latency statistics.

timeDisplayUnits

The units to use when displaying latency calculations.

listSizes > pendingListSize

The maximum number of messages that will be held awaiting confirmation from Child/Parent or the current Checkpoint.

listSizes > reportingListSize

The maximum number of failed messages of a type that will be held.

heartbeatPeriod

The time period between heartbeats.

timeoutPeriods > message

This setting specifies a Lost Message Timeout in seconds. The timeout is measured using the system time (or wall time) between the plug-in first seeing the message and the current time when evaluating lost messages (e.g. the time when a sample occurs). Lost messages are displayed in the Lost Messages view.

Messages older than the configured timeout are considered lost and will be displayed in the Lost Messages view. The reportingListSize setting controls how many lost messages will be retained for display. Lost messages are no longer tracked by the plug-in for further acknowledgements or latency calculations.

The Lost Message timeout is also related to the Unacknowledged and Slow timeouts. The diagram below describes the possible transitions between message views. When a message is seen at a checkpoint, it is tracked until it becomes either Acknowledged (not shown in the diagram) or Lost.

Users may optionally configure an Unacknowledged timeout, which will display messages older than the timeout until that message is either Acknowledged or Lost. The optional Slow timeout will additionally display Acknowledged messages where the latency is greater than specified.

timeoutPeriods > messageUnacknowledged

This setting specifies an Unacknowledged Message Timeout in seconds. The timeout is measured using the system time (or wall time) between the plug-in first seeing the message and the current time (i.e. sample time) when displaying messages as unacknowledged. When this setting is configured the Unacknowledged Messages view is enabled.

The Unacknowledged Message timeout must be less than the Lost Message timeout. Messages seen at a checkpoint will be displayed as Unacknowledged once they are older than the specified timeout. If they remain unacknowledged (no acknowledgements from child checkpoints are sent) then they will be marked as lost when the lost timeout elapses.

This setting only applies to unacknowledged messages; it has no effect on slow messages. See the lost message timeout setting for a diagram describing how these settings are related.

timeoutPeriods > messageLatency

This setting specifies a Slow Message Timeout to millisecond precision. If configured the Slow Message view will be enabled, which will display only acknowledged messages where the latency exceeds the slow message timeout. This latency is computed using the message timestamps between the parent and child checkpoints the message is seen at.

In the case where ALL child checkpoints must acknowledge a message, the message will only be marked as slow if ALL child checkpoints have acknowledged the message and ANY child checkpoint latency is greater than the timeout value. (If not all child checkpoints respond then the message will eventually be marked as lost after the lost message timeout elapses).

See the lost message timeout setting for a diagram describing how these settings are related.

timeoutPeriods > connection

Time that has to elapse with no communication before a connection to a Parent or Child plug-in is considered to have dropped. The time is in seconds.

No communication is represented by the sampler hearing a heartbeat message. If a netprobe process is killed or dies the timeout period will be ignored and the status will be updated immediately.

timeoutPeriods > maxShownUnacknowledMsgsPerQueue

The number of unacknowledged messages that should be shown on each checkpoint's queue. Limiting the number with this setting reduces the size of the dataview should a checkpoint fail completely.

If this is unset the number is unlimited.

database

This selects the database in which to store the Messages that pass through the adapters. Only adapters with the adapters > adapter > record setting set to true will record data. The database must support the ITRS Latency Schema.

Mandatory: No

database > Disable logging

Enable this option to stop collecting data.

Mandatory: No

database > vendor

This selects the database vendor of the database used to store the data. Supported databases are:

• MySQL
• Oracle

Mandatory: Yes (if database has been selected)

database > vendor > mysql > serverName

Name of the machine hosting the database

Mandatory: Yes (if database has been selected)

database > vendor > mysql > databaseName

Name of the database

Mandatory: Yes (if database has been selected)

database > vendor > mysql > port

Port the Database server listens for requests on.

database > vendor > oracle > databaseName

This setting specifies the name of the Oracle database connection to use.

The connection should be specified as part of the Oracle client library installation, defined in the TNSNAMES.ORA configuration file.

Mandatory: Yes

database > userName

Username used when logging into the database

Mandatory: No

database > password

Password used when logging into the database. This can be specified either as plaintext, or encrypted using Geneos standard encryption.

Mandatory: No

initialConnectSamplingDelay

Specifies the number of samples to wait when the plug-in initialises, before connecting to the database. You may wish to use this setting if the plug-in takes several samples to detect and process all log files on startup, to improve database logging performance.

persistenceFile

The name of the file used to store the state of the adapters when the probe stops, so that no data is lost.

fileReadControl > fileCheckWaitTime

The time period a source file (of an adapter) will wait after reaching the end of the file, before trying to read the file again.

Note: A source file may be shared between multiple adapters.

This setting effectively controls how frequently an adapter will poll for new data, when it reaches the end of a log file. A smaller interval means that more CPU will be consumed performing the checks, but new data added to the log file will be detected quicker.

Latency calculations will not be effected by this setting.

fileReadControl > newFileCheckWaitTime

The time period a source file (of an adapter) will wait after reaching the end of the file, before checking for a new file to replace the current one.

Note: A source file may be shared between multiple adapters.

This setting controls how frequently an adapter will poll for new data sources (log files) when it reaches the end of the current file. This is particularly relevant for files specified with wildcard characters. More frequent polling (caused by decreasing the wait time) will cause an increase in CPU usage.

Latency calculations will not be effected by this setting.

fileReadControl > maxLinesPerMinute

This setting controls the maximum number of lines (per minute) that Message Tracker will read from a configured file, and is applied globally for all files read by the plug-in.

This means that the total maximum number of lines read by the plug-in (per minute) will be the value of this setting, multiplied by the number of files that have been configured.

Note: A source file may be shared between multiple adapters, if the adapters are configured with the same file name / pattern.

fileReadControl > consecutiveReads

The consecutiveReads settings control message reading speed from a file source. By limiting the speed users can set avoid peaks in CPU usage during busy times, fine-tune proportions of CPU time given to different adapters, or reduce load caused by database logging.

If set, then the file source will pause reading for delayFor milliseconds, after every delayAfter messages.

Mandatory: No

fileReadControl > consecutiveReads > delayAfter

The file source will pause reading for delayFor milliseconds, after every delayAfter messages.

Note: A file source may be shared between several adapters.

This setting affects messages read from a file consecutively, which assumes that the plug-in does not reach the end of the file (e.g. due to a surge in messages being logged). Decreasing the value of this setting causes less messages to be processed, before the delay is applied. You may want to do this if you have peak times where many messages are logged to a file. Configuring the consecutiveReads setting can then limit the amount of work done to normalise CPU usage.

Latency calculations will not be effected by this setting.

Mandatory: Yes (if consecutiveReads is set)

fileReadControl > consecutiveReads > delayFor

The file source will pause reading for delayFor milliseconds, after every delayAfter messages.

Note: A file source may be shared between several adapters.

This setting is applied when a number of consecutive messages are read from the file. The delay means that no message processing will happen for this file source for the configured time. During the delay other adapters may continue to run, and so the delay can be used to balance CPU load if one adapter is starving the others due to high rates of message logging. It can also be used to limit CPU load overall, or slow down message processing to the speed of your database.

Latency calculations will not be effected by this setting.

fileReadControl > catchupAdapter > overscanTime

An overscan time setting should only be required if non-unique IDs mode has been enabled, due to the alterations to database logging in this mode. For more details, please see the Coping with non-unique IDs section.

This setting controls how many extra messages a catch-up adapter will read. The overscan time is specified in seconds, and is compared against the message timestamps in the file.

Note: This is a minimum value, and an extra message may be read since the message timestamp is not available until the message is processed.

e.g. Suppose that the overscanTime is set to 5 (seconds). This means that when the catch-up adapter finishes scanning the initial portion of the file, it will then read the current timestamp from messages in the file, and continue processing more messages until 5 seconds has elapsed as indicated by the message timestamps.

resetTime

The time that the message counters reset for the Latency View and the Admin Adapter view. All lost messages are removed at the end of each day.

dateOffset

This offset is applied to the read date. It allows data to be read from files for previous days that only have a timestamp. A day offset of 2 will move the day back 2 days.

hideAdminViews

This will hide the Admin Checkpoints and Admin Links views. It will also hide the Admin Adapters view, but only if there is at least one real time check point configured.

permitNonUniqueOrderIds

This setting enables the MessageTracker tracking logic to cope with messages which do not have unique order IDs. Enabling this setting is subject to several pre-requisites as discussed in the Coping with non-unique IDs section, and will also decrease performance slightly. Therefore, the setting should only be enabled if required.