FEED LATENCY MONITOR Plug-in - TECHNICAL REFERENCE

Configuring instruments individually

There are two ways to configure instruments when setting up FLM. The simpler of the two to understand is to configure each instrument individually:

Here the feed-specific codes for each instrument are entered as each instrument is configured. The format of the instrument codes used by each feed type is described in the relevant section of Feed Adapters.

Configuring instrument groups

In the example in the previous section, it is apparent that the instrument codes follow a pattern. Assuming that the ticker symbol for each instrument is acceptable as its display name, it is possible to save some data entry effort by using instrument groups:

For each group, a per-feed pattern should be defined using the placeholder %i which will be replaced with the instrument symbol / code. If a literal % character is required the escape sequence %% should be used.

These patterns will be expanded for every instrument defined in the group, into an individual instrument definition. Different classes of instruments may require different patterns, in which case multiple instrument groups should be configured (one for each pattern).

The “Instruments…” button is used to call up a dialog where the instruments in the group can be specified. A nested dialog allows the pattern to be overridden or suppressed for one or more feeds.

In the example above, we do not want to retrieve the LVMH instrument from the GL feed, so we have set the code for this feed to an empty value. If we had wanted to retrieve the instrument using a code that did not follow the pattern, we would have entered the correct code.

An instrument can only be configured once, either individually or a as a member of a group.

Configuring field codes

Since all feeds of a given type normally use the same field codes, the setup schema for FLM provides a straightforward way to enter these codes in the Gateway Setup Editor, albeit using a very wide table:

Each row of this table begins with the display name of a field. Each standard feed adapter has its own column, and there are three additional columns for custom feed types.

Field codes must be supplied for all fields for the baseline feed type. Other feed types used by an FLM sampler must have at least one field code supplied.

In the example above, the baseline feed type is RFA and field codes are supplied or all fields in that column.

Output

An example latency view is shown below, showing three feed rows and one peak row. Peak rows (if they are enabled, see Configuring the dataviews) are created automatically when the peak latency for a row exceeds a set threshold; they are retained until cleared manually or displaced by a higher peak.

Headline Legend

Table Legend

Note: Average and standard deviation cells will be set to blank if there are no matching ticks contributing to that calculation. Because unmatched ticks can be held over to the next sample, it is possible for the update rate to be shown as zero for a feed, but for the number of matches not to be zero.

Clearing peak rows

Peak rows can be cleared (removed) individually, by selecting a peak row and using the “Clear this peak” right-click menu option.

All of the peak rows displayed for a feed can be removed by selecting a peak row and using the “Clear all peaks” right-click menu option.

Latency display units

FLM can use either millisecond or microsecond resolution in its output. This affects the values displayed and the column headings in the Latency and Details views as well as the format of the tick history file, matched ticks output file and all HTML reports.

N-sample average columns

Up to two sample average columns can be configured in the Latency view; they display moving averages of the averageLagMs value over the specified number of samples. These columns can be used to configure Geneos rules, so that a short-term increase in latency during a single sample does not trigger a rule unnecessarily.

Peak rows

The peak rows feature of the Latency view is enabled or disabled by creating or removing the Peaks section in the sampler setup; the settings within this section control the precise behaviour of the feature.

If peaks are enabled with the default settings, up to five matches with latency greater than 350ms (as configured in the screenshot) will be displayed as peak rows. Rows will be displayed until cleared using an Active Console command or until displaced from the list by a peak with a higher latency. Two of the optional settings control the threshold and maximum number of peaks.

If the last optional setting, ‘Single feed peak per sample”, is enabled, only the worst latency in each sample will be retained, so that the peak rows will show a history of the worst latencies found per sample.

Value threshold

The value threshold controls how close field values must be between feeds to be considered as a possible match when computing latency values. The default is that values differing by more than 0.0000001 do not match. Changing this setting may be necessary if feeds report the same values to differing levels of precision.

For example, the two ticks below have a value difference of 0.0001 (feed value - base value).

• Base: Price/1406.6666
• Feed: Price/1406.6667

Using the default threshold, these ticks would not be considered to match, but with a threshold of 0.001 (as in the example screenshot above), they would match.

Time threshold

This setting, which can be set separately for each non-baseline feed, controls how close together (in time) ticks must be received, to be considered as a possible match when computing latency values.

The setting is denominated in milliseconds (regardless of the latency display units setting); in the example settings above, the value of 0.25 means 250 microseconds. The default is 2000ms; that is, to be considered a match the feed tick must not arrive more than two seconds before or after the baseline tick.

The global time threshold setting sets both the earliest a feed tick can arrive before a matching baseline tick and the latest time it can arrive after. At the feed level the upper and lower thresholds can be set separately:

For example, with the settings shown above, where the RFA feed is the designated baseline, the GL-Feed uses the global threshold (set in this example to +/- 0.25ms) and the Other-Feed uses a minimum of 0ms and a maximum of 0.3ms), suppose that a number of ticks with identical values arrive at the following times:

On the GL-Feed, tick G1 has too great a negative latency to be considered. Ticks G2 and G3 are both in the acceptable time range; since they both match, the first will be used to compute a latency value of -0.169ms (for this tick).

On the Other-Feed, since the ‘Time threshold low’ setting is zero, only ticks with positive latency are considered. L1 has negative latency and is excluded; tick L2 is used to compute a latency value of +0.285ms. This time threshold setting will help the plug-in avoid invalid tick matches where a feed is known to always be slower than the baseline feed (for example it might be derived from baseline data with some additional fields).

Note: This setting interacts with the ‘process duplicate value ticks setting’, described in the next section.

In the example above, with the default value of the setting, ticks G2, G3 and L2 would be ignored, since they are duplicates of earlier ticks. Since ticks G1 and L1 are excluded by the time threshold, no matches would be found for tick R1 in the GL-Feed or Other-Feeds.

Process duplicate value ticks

Market data feeds, as presented to FLM, frequently include ticks all of whose values match the previous tick in the same feed. This may be because a field has been changed that FLM is not subscribing to and the infrastructure has not filtered out the tick.

By default, these duplicated ticks are excluded as possible matches. This setting can be used to include them.

Field match skew

The field match skew setting controls how the FLM matching algorithm selects ticks when there is more than one way to match the same set of ticks.

For example, given these sequences of ticks:

These streams can be matched in two ways { R0 = G0, R1 = G2, R3 = G3 } or { R0 = G0, R2 = G1, R3 = G3 }. The choice concerns the middle ticks. Since the first match yields a latency of 102ms and the second −8ms, FLM will normally prefer the second.

But if we look at the changes between one tick and the next in each stream, we see that in tick R1 two values were updated and the same two values were updated in tick G2; on the other hand although three values changed in tick R2, only one of these changed in tick G1. So it could be argued that FLM should prefer the first match.

The field match skew setting allows FLM to “reward” tick matches with shared field updates by specifying a number of milliseconds to be subtracted from the absolute latency for each field that is updated in common. If the skew value was set to 150, then the “adjusted latency” would be

abs(−8ms) − (1 field * 150ms/field) = (8 − 150)ms = −142ms

for the first match and

abs(102ms) − (2 fields * 150ms/field) = (102 – 300)ms = −198ms

for the second, so the first match would be preferred.

This “adjusted latency” is only used to choose between ways of matching ticks; it is not used in calculating the latency statistics once the tick streams have been matched.

Tick history file

The tick history file is normally written to the Netprobe working directory. It records the ticks received by FLM and shows how they were matched by the matching algorithm. It is used to generate some reports and is also useful for checking that feeds are correctly configured and for observing the effects of changing the settings of the matching algorithm. A new history file is started at midnight each day.

== Sample, 16:25:24 ==
Time:16:25:21.327000  Feed:GL-feed   Inst:ORA
        Field:Ask       Value:1540.50000000  C
        Field:Bid       Value:1538.00000000  U
        Field:Trade     Value:1539.50000000  C*
Time:16:25:24.446000  Feed:RFA-feed  Inst:EDF
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  U*
Time:16:25:24.446000  Feed:RFA-feed  Inst:EDF
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  C*
Time:16:25:24.446000  Feed:Other-feed  Inst:EDF       Match:-2
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  U*

Note: In the example above, the ‘U’ and ‘C’ after each field value indicate whether the field was updated in the current tick or copied from the previous tick. The last line for each tick ends with an asterisk.

Settings are available in the Advanced tab of the FLM sampler settings to control where this file is written, how many previous versions are kept and whether a user-supplied archive script should be run each night.

Matched tick output file

If this output is enabled (via a setting in in the Advanced tab of the FLM sampler settings), it is appended to at every sample, and stores the matches computed by the tick matching algorithm.

Inst,RFA-feed-Timestamp,Ask,Bid,Trade,GL-feed-Timestamp,Ask,Bid,Trade,Latency,Other-feed-Timestamp,Ask,Bid,Trade,Latency
ORA,16:25:21.214,1540.50000000,1539.00000000,1539.50000000,16:25:21.307,1540.50000000,1539.00000000,1539.50000000,93 ms,16:25:21.214,1540.50000000,1539.00000000,1539.50000000,0 ms
EDF,16:25:24.446,644.00000000,643.70000000,643.88276000,,,,,,16:25:24.446,644.00000000,643.70000000,643.88276000,0 ms
LVMH,16:25:24.597, 99.60000000, 99.56000000, 99.57000000,16:25:24.689, 99.60000000, 99.56000000, 99.57000000,92 ms,16:25:24.597, 99.60000000, 99.56000000, 99.57000000,0 ms

Figure 14 – Example of matched tick output file. The EDF tick in the second data row can also be seen in the sample tick history file shown in the previous figure.

Each line of the file displays a matched tick from the baseline feed against matches found from the other configured feeds. Baseline ticks with no matches are not written to the file. The corresponding latency of the feed tick against the base is shown in the Latency columns. This output will be in either milliseconds (ms) or microseconds (us) as determined by the latency display units setting.

Latency report

This command is available as a right-click menu option on a feed row in the Latency view. It produces a HTML report which displays a period of market data up to 30 minutes, matched against data received from the baseline feed. Where a match occurs, the latency for the match is also displayed which can be used to verify the overall feed latency figures. A graph plotting tick values is also provided to give an indication of the changes which occurred during this time period.

When the command is executed, a dialog is displayed requesting the report interval in minutes and an optional start time. The interval value can be any integer value between 1 and 30 (inclusive), the start time has to be a time of day in the format HH:MM. If a start time is specified then the report runs from that point, if no start time is specified then it runs up to the current time.

An example report is shown above. Tick values highlighted with bold text indicate when an update was received – typically the value will change. If the value does not change between updates, then FLM’s behaviour will be determined by the processDuplicateValueTicks configuration parameter.

Coloured lines indicate where a tick match has been found. Different tick matches are highlighted with a different colour to help identify matches when the ticks do not appear directly adjacent to each other.

Latency report (peak)

This command is available as a right-click menu option on a peak row in the Latency view. It produces a similar report to the Latency report command described above. The difference is that rather than the latest X minutes, the resulting report contains (by default) 6 minutes of latency data centred around the time of the tick which produced the peak row (highlighted in red on the report).

This report can be used to investigate a peak in latency values, and determine whether it was a one-off occurrence or if there are further issues which need to be resolved.

Recent-ticks report

This command is available as a right-click menu option on the Latency view. It produces a HTML report detailing the ticks received in the last X minutes. This report contains ticks for all feeds, and therefore only shows tick values. For individual tick latencies, please see the Latency report command described above. This report also includes a graph plotting values over time, allowing comparison of values across multiple feeds.

Show tick history

This command is available as a right-click menu option on the Latency view. It was originally implemented in order to provide an interface similar to FKM’s “view file” command. However, a tick stream updates too rapidly to be observed by a continuously updating display and a snapshot of the tick stream at an arbitrary instant is not very useful.

When the command is executed a dialog is displayed which allows the output can be filtered to view specific feeds or instruments.

The output of the command is a filtered extract of the FLM tick history file.

Go to Feed Adapters to view the complete list of pre-built feed adapters that are package in the Netprobe binaries.

feeds > feed

At least two feeds are required in order for the plug-in to operate.

Each feed must be uniquely named so that they can be referenced in other parts of the plug-in configuration. These names will also be displayed in the dataviews produced by FLM.

Mandatory: At least two feeds must be specified.

feeds > feed > rfa

This section contains the configuration required to obtain data from a TREP platform. Further details about the feed adapter can be found in RFA adapter.

The RFA feed contains 3 settings:

feeds > feed > rfaWombat

The RFA-Wombat feed type supports the same settings as the RFA feed type; it is provided to allow configuration of different field codes, since Wombat data fed through a TREP platform uses different codes than the native data.

The settings for this feed are identical to those of the RFA adapter.

feeds > feed > gl

This section contains the configuration required to obtain data from a GL SLC. Further details about the feed adapter can be found in GL adapter.

feeds > feed > nyxt

This section contains the configuration required to obtain data using a NYSE Openmama connection. Further details about the feed adapter can be found in the NYXT adapter section of this document.

feeds > feed > tt

This section contains the configuration required to obtain data from a TT X_TRADER infrastructure. Further details about the feed adapter can be found in TT adapter.

feeds > feed > custom

This section contains settings for custom field configurations. See the Custom field configuration section of this document for further details.

feeds > feed > testfile

This section contains settings used to allow an FLM tick history file to be replayed as input for testing purposes. Further details about the feed adapter can be found in TestFile adapter.

Feed-specific algorithm settings

Baseline feed setting

Feed

This setting selects a feed configured in the section as the baseline feed for FLM.

When calculating latency figures, all data updates are compared against those of the baseline feed. Positive latency figures therefore indicate that updates are received later than those of the baseline; conversely negative figures indicate updates received earlier.

Mandatory: Yes

Individual instruments

Instrument groups

fields > field

Configures a single field to be used by FLM. Each field must have a unique display name, so that they can be distinguished in the dataviews produced by the plug-in.

Field codes are configured for feed types, rather than specific feeds because – in general – all of the feeds using a given feed adapter will use the same field codes.

Fields for the baseline feed type must always be configured; otherwise no latency comparisons can be performed.

Mandatory: At least one field is required

latencyDisplayUnits

This setting controls the units used to display latency values in FLM plug-in output. There are two possible values, milliseconds or microseconds. Depending on the value of this setting some dataview columns may have a different name; for example a suffix of Ms or Us for millisecond and microsecond units respectively.

valueThreshold

Maximum difference between field values for ticks to be considered to match.

timeThreshold

This specifies the maximum value for the absolute difference (in milliseconds) between the timestamp of a feed tick and the timestamp of a matching baseline tick.

processDuplicateValueTicks

If all values in a tick match the previous tick in the same feed, it is considered a duplicate and, by default, will be excluded from consideration by the matching algorithm.

To force FLM to consider these duplicate tick values, configure this setting to true.

fieldMatchSkew

Number of milliseconds to be subtracted from calculated absolute latency for each field updated in both ticks when choosing between matched ticks.

sampleAverages

Up to two sample average columns can be configured. These columns are displayed in the Latency view, and produce averages of the averageLagMs value over the specified number of samples.

Mandatory: No

useFullTickRange

If this setting is enabled, the matching algorithm will include ticks which have a timestamp later than the end of the current sample interval. Useful only when the test file feed adapter is being used.

dumpAlgoTables

If this setting is enabled and the debug settings for the probe include “FLM:algo”, then in addition to general algorithm debug output, FLM will also dump the data tables that have been generated during execution of the algorithm.

collectInterval

This deprecated setting previously affected the period at which FLM plug-in collected received ticks from the feed threads, and logged them to the history file.

Ticks are now collected by the sampler only on each sample event. Any values specified for this setting will generate a warning in the Netprobe log, but are otherwise ignored by FLM.

Mandatory: No

htmlDir

Report files generated by FLM are placed in the directory specified by this setting. FLM will then serve up these report files in response to HTTP requests.

This directory can optionally be within the path served by a web-server (e.g. Apache) in which case the httpPrefix setting should also be specified.

httpPrefix

This setting controls the prefix of the URLs returned by FLM for a report generation request. If the prefix is empty, FLM will generate an appropriate prefix using the Netprobe hostname and listen port, in order to serve reports in response to HTML requests.

Setting a prefix implies that a different web-server will be taking responsibility for serving the report pages generated by FLM. In this case, report files should be placed directly in a folder served by the web-server, which can be controlled using the htmlDir setting.

peaks

FLM can optionally generate a new table row in the Latency view to indicate a high peak latency value. This feature is enabled by creating a peaks section in the sampler setup. The settings within this section further control the behaviour of peak rows.

Mandatory: No

peaks > minLatencyTime

This setting controls how high the peak latency figure must be to generate a peak row, and is specified as a number of milliseconds. Peak values higher than this setting will cause a row to be created. A microsecond threshold can be specified using floating-point values in milliseconds. E.g. 0.468 milliseconds is equivalent to 468 microseconds.

peaks > maxPeaksPerFeed

This setting controls the maximum number of peak rows that can be created for a single feed. If no peaks section is created then the peaks feature is disabled, otherwise the maximum value is controlled by this setting. A value of 0 will prevent any peak rows from being created.

peaks > singleFeedPeakPerSample

If this setting is enabled, only the worst peak latency in each sample for a particular feed is added as a new peak row.

tickHistoryFile

The tickHistoryFile section contains settings relating to the tick history file. This file is output by the plug-in and contains a record of all ticks received, which is used to generate reports and can also be used for debugging purposes.

Mandatory: No

tickHistoryFile > filePrefix

The filePrefix setting specifies the filename base which will be used for the tick history file. The plug-in will then use this base and add the current date, which is the file that is then logged to. This file is relative to the Netprobe working directory – to specify your own location it is recommended to use an absolute path.

e.g. If the filePrefix is set to:

/home/geneos/flm/history

On 20^th December 2010, the tick history file used by FLM will be:

/home/geneos/flm/history20101220

All files with a file prefix that matches this setting will be considered for the maxFiles limit.

tickHistoryFile > maxFiles

This setting specifies the maximum number of history files to keep, not including the history file for the current day. On checking the limit, the plug-in will enumerate all files matching the pattern <filePrefix >YYYYMMDD* (where YYYYMMDD is a date) and keep the most recent files by date (up to the specified limit), removing all other files.

This limit is checked when the plug-in starts, and whenever a history file is rolled over.

Note: Limit checking is performed before the archiveScript (if any) is run. If your archive script manages history files itself, then this setting should not be set to 0, or else the history file will be removed before the script archives it.

tickHistoryFile > archiveScript

The archive script setting allows users to specify a script, which will be called with the name of the current tick history file when this file is rolled over at the end of the day. This allows users the option to archive the history file somewhere, using a similar mechanism to the Netprobe log file.

e.g.

archiveScript = /bin/gzip