Feed Latency Monitor

EOL

The end of support for Feed Latency Monitor Plugin (FLM) on Solaris platform (x64 and Sparc) is on 31 July, 2019.

Overview Copied

Introduction Copied

The Geneos Feed Latency Monitor plugin (usually abbreviated to FLM) monitors the quality and speed of market data in real-time. The term market data typically refers to numerical quote and trade data reported from trading venues such as stock exchanges, and is usually supplied via third party data streams.

FLM subscribes via direct market data APIs to a set of configured market feeds, from which it requests updates for a list of user-defined market data instruments. The data from these updates are compared against each other, and the time interval between matching updates for different feeds is calculated. Since FLM computes a relative latency between feeds and not an absolute (exchange to client) latency time, it avoids issues of clock synchronisation. Also, since it computes relative latency, the plugin requires at least two data feeds to operate.

The latency figures produced by FLM can be used in Geneos Rules (and other gateway features) alongside monitored metric values. These values can help determine when a feed is lagging behind other feeds, or if a particular instrument receives faster updates on a specific feed.

Market data inputs to FLM are provided by pluggable feed adapter libraries. These are dynamically-linked shared object files (a .so file on UNIX or a .DLL file on Windows) which each connect to a market data feed (using a vendor-specific direct market-data API) and translate data into a normalised format expected by FLM.

Intended Audience Copied

This document is intended for users who wish to install and configure FLM to implement feed latency monitoring as part of their existing Geneos installation.

The purpose of this document is to describe how to install, use and configure FLM. The latter portion of the document also contains a technical reference for more experienced users.

The document assumes that users have some understanding of the configuration requirements of the market data API for the feeds they wish to monitor, as well as a basic knowledge of how to configure Geneos components, such as adding a new Netprobe or sampler to a Gateway configuration.

Architecture Copied

This diagram shows the flow of data (i.e. market data ticks) through the FLM plugin.

Two different types of vendor feed adapter are shown. These are supplied as standard with FLM. Standard adapters also exist for GL and TT.

Adapters are loaded by the Feed Controller component as and when they are required by FLM sampler instances. If a feed in one FLM sampler instance uses the same adapter and feed configuration settings as a feed in another sampler, they will share a single connection to the underlying market data feed.

The FLM engine is the heart of the FLM sampler. It uses a longest common subsequence algorithm to identify groups of ticks for each instrument for each monitored feed that match groups of ticks in the baseline feed. Once matching ticks have been identified, relative latencies are computed and summarised in dataviews (which are published in the same way as for other Geneos samplers.) In addition to latency data, a Values view allows feed configurations to be verified by showing the current values for the configured instruments on each feed.

The FLM engine also can also publish detailed information about the operation of the matching algorithm in the form of CSV reports of matched ticks and HTML reports which combine a graph plotting tick values with a table showing how the ticks from each feed were matched.

Terminology Copied

Deployment Copied

General considerations Copied

FLM is a multi-threaded plugin which uses one thread per data source (typically each feed) to subscribe and request market data. Thus it should work more efficiently on a multi-core machine (producing more accurate tick timestamps) but this is not required to run the plugin.

The standard pluggable feed adapter libraries for FLM are distributed as part of the Netprobe package in the flm sub-folder. The contents of this folder will vary by Netprobe platform, since not all feeds are available on all platforms.

When using the default package layout Netprobe will find and load these libraries automatically as specified by the feed configuration. If using a customised Netprobe folder layout, you should ensure the feed adapter libraries are available on the LD_LIBRARY_PATH for UNIX, or in the PATH environment variable on Windows.

Some FLM features involve generating HTML reports to a web-server; in order to do so the Netprobe must have write permissions to the appropriate server directory.

Graphs for reports are generated using the ChartDirector library which is available for Windows (32-bit), Linux (32 and 64-bit), Solaris Sparc (32-bit) and Solaris x86 (32-bit) Netprobe platforms. On platforms where this library is not available (e.g. AIX) the report will still be generated but no graphs will be output.

When using the standard Netprobe package layout no further ChartDirector configuration is required. For custom layouts, the libraries must be available on the LD_LIBRARY_PATH. Additionally UNIX machines must place the flm/fonts folder adjacent to the libchartdir.so.5.0 library file.

Feed Adapter deployment Copied

FLM is packaged with a number of standard feed adapters. The operating systems that these are supported on vary. In most cases this reflects the availability of the feed platform APIs used.

Go to Feed Adapters to view the complete list of pre-built feed adapters that are package in the Netprobe binaries.

Please refer to the Geneos Compatibility Matrix for the list of supported platforms.

In addition to the adapters listed above, FLM is also packaged with a ‘test file’ adapter, which allows tick history files generated in one FLM environment to be replayed in another.

Because the FLM feed adapters use a direct market data connection to their respective sources, the Netprobe host must have access to these in order for FLM to operate normally. For most feed adapters, this implies access to a TCP/IP resource in the market feed infrastructure; for the TT feed the Netprobe must be running a Windows host with an X_Trader Pro license. For further information about using each of these adapters, please refer to Feed Adapters.

Tick stream processing Copied

Each feed adapter connected to an instance of FLM generates a stream of ticks for each configured instrument. Each tick consists of a value for each configured field, with an indicator that shows whether or not the value has changed from the previous tick.

Latency calculations Copied

FLM computes latency figured by comparing ticks (for a particular instrument) from a given feed against the baseline feed. Only a selected set of fields in the source ticks are actually used for comparison – any other data is ignored. These fields are configured by the user in the sampler setup.

Ticks are collected from data sources continually. During an FLM sample, the ticks that have been collected between samples are then compared – each feed is compared against the baseline feed. Matches between ticks are found using an adjusted Longest Common Subsequence (LCS) algorithm, subject to criteria such as value or time constraints.

Once matches are found, the latency is computed as the time difference between when the feed received the tick, and when the baseline received its matching tick. Averages displayed for instruments are produced by summing the total latency across all fields, and dividing by the number of matches. Similarly averages for feeds are produced from raw values for instruments, which help to prevent loss of precision due to rounding errors.

Monitoring feed values Copied

Although FLM is primarily designed to perform latency calculations, it can also be used to monitor the market data values themselves, for example to verify that a quote feed produced by an in-house system is correctly tracking an external market feed.

If latency figures are not required, a dummy feed can be used as the baseline, to minimise the overhead of the matching algorithm. For instance, the test file feed adapter can be set up with an empty file as its data source. If the baseline feed never receives any ticks, the matching algorithm will not perform any calculations. The Hide Rows sampler setting can be used to remove rows generated for the baseline feed from the Values dataview.

Performance considerations Copied

As a general rule, the number of fields and instruments configured should be kept to a minimum consistent with correctly matching ticks. That is, unless the values themselves are being monitored, you should only configure enough fields to avoid false matches and enough instruments to provide a reasonably steady stream of ticks.

The reason for this is that if the Values view becomes very large, the processing required to update it will become significant. Although this processing can be reduced by increasing the sample interval, doing so can dramatically increase the time and space required to compute the matches. This is because the longest common subsequence algorithm has O(N•M) complexity: that is, to match up two tick streams, it requires time and space proportional to the number of ticks in one stream multiplied by the number of ticks in the other.

In short, other things being equal, doubling any one of the number of instruments, number of fields or number of non-baseline feeds will double the amount of processing required, while increasing the sample interval will reduce the processing needed to post updates to the dataviews but increase the processing required to compute the latency for a given number of seconds of data.

Active times Copied

Do not set an active time on the FLM sampler in order to schedule a period when no changes are posted to its dataviews.

Although latency calculations and updates to the dataviews would indeed be suspended, the feed subscriptions would not be suspended. When the active time ends, the sampler would attempt to perform latency calculations on all the ticks received in the meanwhile. Due to the characteristics of the longest common subsequence algorithm, this could cause the Netprobe to exhaust the memory available to it and abort.

Instead of setting an active time on the FLM sampler, an active time should be set on any Gateway rules used to generate alerts from the FLM dataviews.

User Guide Copied

This chapter describes how to configure and use the most common features of FLM. Users are expected to have some basic experience with using the Gateway Setup Editor to edit Geneos configurations.

Configuring Feeds Copied

For FLM to operate, at least two feeds must be configured. One of the configured feeds must then be designated as the “baseline” feed. Ticks from all other feeds will then be compared against ticks from the baseline feed when running the matching algorithm to compute relative latencies.

Each feed must be uniquely named so that it can be referenced in other parts of the plugin configuration. Its name will also be displayed in the dataviews produced by FLM.

Almost all aspects of configuring a feed depend on the feed adapter used; they are described in the Feed Adapters.

The “Algorithm…” button for each feed allows some aspects of the matching algorithm to be overridden for that feed; it is discussed in the Configuring the matching algorithm section.

Warning

When FLM samplers are using same session ID, the instrument errors encountered in one sampler display in the dataview of the other sampler.

When one of the instruments in a session encounters an error, this error is shown in the dataviews of all samplers that are associated with the session.

This alerts you that one of the instrument subscriptions has a problem.

The workaround for this item is to associate each sampler with its own respective sessions.

This involves modifying the RFA configuration file and the FLM set-up for RFA to associate each sampler with its own sessions.

Configuring Instruments and Fields Copied

For FLM to operate, it must be able to calculate latencies for at least one feed against the baseline feed. To do this, there must be at least one configured instrument in common between the baseline feed and the comparison feed.

Each instrument must be given a unique name which is used for display purposes. A feed-specific stock code or other identifier must also be configured for the instrument, which is used when making a subscription request on the feed. These instrument codes are likely to vary, even between feeds of the same type.

Configuring instruments individually Copied

There are two ways to configure instruments when setting up FLM. The simpler of the two to understand is to configure each instrument individually:

Here the feed-specific codes for each instrument are entered as each instrument is configured. The format of the instrument codes used by each feed type is described in the relevant section of Feed Adapters.

Configuring instrument groups Copied

In the example in the previous section, it is apparent that the instrument codes follow a pattern. Assuming that the ticker symbol for each instrument is acceptable as its display name, it is possible to save some data entry effort by using instrument groups:

For each group, a per-feed pattern should be defined using the placeholder %i which will be replaced with the instrument symbol / code. If a literal % character is required the escape sequence %% should be used.

These patterns will be expanded for every instrument defined in the group, into an individual instrument definition. Different classes of instruments may require different patterns, in which case multiple instrument groups should be configured (one for each pattern).

The “Instruments…” button is used to call up a dialog where the instruments in the group can be specified. A nested dialog allows the pattern to be overridden or suppressed for one or more feeds.

In the example above, we do not want to retrieve the LVMH instrument from the GL feed, so we have set the code for this feed to an empty value. If we had wanted to retrieve the instrument using a code that did not follow the pattern, we would have entered the correct code.

An instrument can only be configured once, either individually or a as a member of a group.

Configuring field codes Copied

Since all feeds of a given type normally use the same field codes, the setup schema for FLM provides a straightforward way to enter these codes in the Gateway Setup Editor, albeit using a very wide table:

Each row of this table begins with the display name of a field. Each standard feed adapter has its own column, and there are three additional columns for custom feed types.

Field codes must be supplied for all fields for the baseline feed type. Other feed types used by an FLM sampler must have at least one field code supplied.

In the example above, the baseline feed type is RFA and field codes are supplied or all fields in that column.

Note

Configuration Tip: The plugin in unable to validate the correctness or existence of either instrument names or field codes. If these are misconfigured, data will not be returned. This can be validated by checking the values view is correctly populated.

Latency View Copied

The latency view shows a summary of calculated latency values for each configured feed.

Output Copied

An example latency view is shown below, showing three feed rows and one peak row. Peak rows (if they are enabled, see Configuring the dataviews) are created automatically when the peak latency for a row exceeds a set threshold; they are retained until cleared manually or displaced by a higher peak.

Headline Legend

Table Legend

Note

The name of several columns in this data view will have the suffix ‘Ms’ or ‘Us’ according to whether the Latency display units setting specifies millisecond or microsecond units.

Note

Average and standard deviation cells will be set to blank if there are no matching ticks contributing to that calculation. Because unmatched ticks can be held over to the next sample, it is possible for the update rate to be shown as zero for a feed, but for the number of matches not to be zero.

Clearing peak rows Copied

Peak rows can be cleared (removed) individually, by selecting a peak row and using the “Clear this peak” right-click menu option.

All of the peak rows displayed for a feed can be removed by selecting a peak row and using the “Clear all peaks” right-click menu option.

Details View Copied

The details view shows the average latency for each instrument and each feed. These figures can be used to determine if a particular instrument has faster updates for a given feed.

Note

The column for the baseline feed will always contain zeroes

Table Legend

Values View Copied

The values view shows the current values of configured instruments and fields for each feed. This view is updated once per sample interval rather than in real-time when an update occurs. It is intended to be used to confirm that the sampler is correctly configured and is not well suited to monitoring the market data values themselves.

Table Legend

Configuring the dataviews Copied

Settings for the Latency and Details view are found on the Advanced tab of the FLM sampler settings:

Latency display units Copied

FLM can use either millisecond or microsecond resolution in its output. This affects the values displayed and the column headings in the Latency and Details views as well as the format of the tick history file, matched ticks output file and all HTML reports.

N-sample average columns Copied

Up to two sample average columns can be configured in the Latency view; they display moving averages of the averageLagMs value over the specified number of samples. These columns can be used to configure Geneos rules, so that a short-term increase in latency during a single sample does not trigger a rule unnecessarily.

Peak rows Copied

The peak rows feature of the Latency view is enabled or disabled by creating or removing the Peaks section in the sampler setup; the settings within this section control the precise behaviour of the feature.

If peaks are enabled with the default settings, up to five matches with latency greater than 350ms (as configured in the screenshot) will be displayed as peak rows. Rows will be displayed until cleared using an Active Console command or until displaced from the list by a peak with a higher latency. Two of the optional settings control the threshold and maximum number of peaks.

If the last optional setting, ‘Single feed peak per sample”, is enabled, only the worst latency in each sample will be retained, so that the peak rows will show a history of the worst latencies found per sample.

Configuring the matching algorithm Copied

Some parameters of the FLM matching algorithm can be configured using settings available in the Advanced tab of the FLM sampler settings. These settings will be applied globally for all feeds. Some setting values can be overridden from the global values defined here, in the feed-specific Algorithm settings dialog.

Value threshold Copied

The value threshold controls how close field values must be between feeds to be considered as a possible match when computing latency values. The default is that values differing by more than 0.0000001 do not match. Changing this setting may be necessary if feeds report the same values to differing levels of precision.

For example, the two ticks below have a value difference of 0.0001 (feed value - base value).

• Base: Price/1406.6666
• Feed: Price/1406.6667

Using the default threshold, these ticks would not be considered to match, but with a threshold of 0.001 (as in the example screenshot above), they would match.

Time threshold Copied

This setting, which can be set separately for each non-baseline feed, controls how close together (in time) ticks must be received, to be considered as a possible match when computing latency values.

The setting is denominated in milliseconds (regardless of the latency display units setting); in the example settings above, the value of 0.25 means 250 microseconds. The default is 2000ms; that is, to be considered a match the feed tick must not arrive more than two seconds before or after the baseline tick.

The global time threshold setting sets both the earliest a feed tick can arrive before a matching baseline tick and the latest time it can arrive after. At the feed level the upper and lower thresholds can be set separately:

For example, with the settings shown above, where the RFA feed is the designated baseline, the GL-Feed uses the global threshold (set in this example to +/- 0.25ms) and the Other-Feed uses a minimum of 0ms and a maximum of 0.3ms), suppose that a number of ticks with identical values arrive at the following times:

On the GL-Feed, tick G1 has too great a negative latency to be considered. Ticks G2 and G3 are both in the acceptable time range; since they both match, the first will be used to compute a latency value of -0.169ms (for this tick).

On the Other-Feed, since the ‘Time threshold low’ setting is zero, only ticks with positive latency are considered. L1 has negative latency and is excluded; tick L2 is used to compute a latency value of +0.285ms. This time threshold setting will help the plugin avoid invalid tick matches where a feed is known to always be slower than the baseline feed (for example it might be derived from baseline data with some additional fields).

Note

This setting interacts with the ‘process duplicate value ticks

In the example above, with the default value of the setting, ticks G2, G3 and L2 would be ignored, since they are duplicates of earlier ticks. Since ticks G1 and L1 are excluded by the time threshold, no matches would be found for tick R1 in the GL-Feed or Other-Feeds.

Process duplicate value ticks Copied

Market data feeds, as presented to FLM, frequently include ticks all of whose values match the previous tick in the same feed. This may be because a field has been changed that FLM is not subscribing to and the infrastructure has not filtered out the tick.

By default, these duplicated ticks are excluded as possible matches. This setting can be used to include them.

Field match skew Copied

The field match skew setting controls how the FLM matching algorithm selects ticks when there is more than one way to match the same set of ticks.

For example, given these sequences of ticks:

These streams can be matched in two ways { R0 = G0, R1 = G2, R3 = G3 } or { R0 = G0, R2 = G1, R3 = G3 }. The choice concerns the middle ticks. Since the first match yields a latency of 102ms and the second −8ms, FLM will normally prefer the second.

But if we look at the changes between one tick and the next in each stream, we see that in tick R1 two values were updated and the same two values were updated in tick G2; on the other hand although three values changed in tick R2, only one of these changed in tick G1. So it could be argued that FLM should prefer the first match.

The field match skew setting allows FLM to “reward” tick matches with shared field updates by specifying a number of milliseconds to be subtracted from the absolute latency for each field that is updated in common. If the skew value was set to 150, then the “adjusted latency” would be

abs(−8ms) − (1 field * 150ms/field) = (8 − 150)ms = −142ms

for the first match and

abs(102ms) − (2 fields * 150ms/field) = (102 – 300)ms = −198ms

for the second, so the first match would be preferred.

This “adjusted latency” is only used to choose between ways of matching ticks; it is not used in calculating the latency statistics once the tick streams have been matched.

Tick output files Copied

Tick history file Copied

The tick history file is normally written to the Netprobe working directory. It records the ticks received by FLM and shows how they were matched by the matching algorithm. It is used to generate some reports and is also useful for checking that feeds are correctly configured and for observing the effects of changing the settings of the matching algorithm. A new history file is started at midnight each day.

== Sample, 16:25:24 ==
Time:16:25:21.327000  Feed:GL-feed   Inst:ORA
        Field:Ask       Value:1540.50000000  C
        Field:Bid       Value:1538.00000000  U
        Field:Trade     Value:1539.50000000  C*
Time:16:25:24.446000  Feed:RFA-feed  Inst:EDF
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  U*
Time:16:25:24.446000  Feed:RFA-feed  Inst:EDF
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  C*
Time:16:25:24.446000  Feed:Other-feed  Inst:EDF       Match:-2
        Field:Ask       Value:644.00000000  C
        Field:Bid       Value:643.70000000  C
        Field:Trade     Value:643.88276000  U*

Note

In the example above, the ‘U’ and ‘C’ after each field value

Settings are available in the Advanced tab of the FLM sampler settings to control where this file is written, how many previous versions are kept and whether a user-supplied archive script should be run each night.

Matched tick output file Copied

If this output is enabled (via a setting in in the Advanced tab of the FLM sampler settings), it is appended to at every sample, and stores the matches computed by the tick matching algorithm.

Inst,RFA-feed-Timestamp,Ask,Bid,Trade,GL-feed-Timestamp,Ask,Bid,Trade,Latency,Other-feed-Timestamp,Ask,Bid,Trade,Latency
ORA,16:25:21.214,1540.50000000,1539.00000000,1539.50000000,16:25:21.307,1540.50000000,1539.00000000,1539.50000000,93 ms,16:25:21.214,1540.50000000,1539.00000000,1539.50000000,0 ms
EDF,16:25:24.446,644.00000000,643.70000000,643.88276000,,,,,,16:25:24.446,644.00000000,643.70000000,643.88276000,0 ms
LVMH,16:25:24.597, 99.60000000, 99.56000000, 99.57000000,16:25:24.689, 99.60000000, 99.56000000, 99.57000000,92 ms,16:25:24.597, 99.60000000, 99.56000000, 99.57000000,0 ms

Figure 14 – Example of matched tick output file. The EDF tick in the second data row can also be seen in the sample tick history file shown in the previous figure.

Each line of the file displays a matched tick from the baseline feed against matches found from the other configured feeds. Baseline ticks with no matches are not written to the file. The corresponding latency of the feed tick against the base is shown in the Latency columns. This output will be in either milliseconds (ms) or microseconds (us) as determined by the latency display units setting.

How to see the timestamps in Excel Copied

The CSV file format is designed for easy import into Microsoft Excel; however to view the timestamp properly it is necessary to use a custom format:

1. Select all the timestamp columns.
2. Right click and select “Format Cells … “.
3. Select “Custom” in the “Category” list.
4. Enter the string “hh:mm:ss.000”.

Web-based reports Copied

A number of HTML reports are available as right-click menu options on rows in the Latency view. The output of each command is an HTML link to the requested report.

Latency report Copied

This command is available as a right-click menu option on a feed row in the Latency view. It produces a HTML report which displays a period of market data up to 30 minutes, matched against data received from the baseline feed. Where a match occurs, the latency for the match is also displayed which can be used to verify the overall feed latency figures. A graph plotting tick values is also provided to give an indication of the changes which occurred during this time period.

When the command is executed, a dialog is displayed requesting the report interval in minutes and an optional start time. The interval value can be any integer value between 1 and 30 (inclusive), the start time has to be a time of day in the format HH:MM. If a start time is specified then the report runs from that point, if no start time is specified then it runs up to the current time.

An example report is shown above. Tick values highlighted with bold text indicate when an update was received – typically the value will change. If the value does not change between updates, then FLM’s behaviour will be determined by the processDuplicateValueTicks configuration parameter.

Coloured lines indicate where a tick match has been found. Different tick matches are highlighted with a different colour to help identify matches when the ticks do not appear directly adjacent to each other.

Latency report (peak) Copied

This command is available as a right-click menu option on a peak row in the Latency view. It produces a similar report to the Latency report command described above. The difference is that rather than the latest X minutes, the resulting report contains (by default) 6 minutes of latency data centred around the time of the tick which produced the peak row (highlighted in red on the report).

This report can be used to investigate a peak in latency values, and determine whether it was a one-off occurrence or if there are further issues which need to be resolved.

Recent-ticks report Copied

This command is available as a right-click menu option on the Latency view. It produces a HTML report detailing the ticks received in the last X minutes. This report contains ticks for all feeds, and therefore only shows tick values. For individual tick latencies, please see the Latency report command described above. This report also includes a graph plotting values over time, allowing comparison of values across multiple feeds.

Deprecated commands Copied

The following command is likely to be withdrawn from a future version of FLM.

Show tick history Copied

This command is available as a right-click menu option on the Latency view. It was originally implemented in order to provide an interface similar to FKM’s “view file” command. However, a tick stream updates too rapidly to be observed by a continuously updating display and a snapshot of the tick stream at an arbitrary instant is not very useful.

When the command is executed a dialog is displayed which allows the output can be filtered to view specific feeds or instruments.

The output of the command is a filtered extract of the FLM tick history file.

Go to Feed Adapters to view the complete list of pre-built feed adapters that are package in the Netprobe binaries.

Configuration settings Copied

This chapter describes the configuration settings and options available for the Feed Latency Monitor plugin. Configuration for the plugin is placed in the Gateway setup file under the FLM (plugin) section of a sampler descriptor.

A Feed Latency Monitor configuration consists of three major parts:

• Feeds - the set of feeds to be monitored.
• Instruments - the set of instruments to be monitored, with their names for each feed.
• Fields - the set of fields to be monitored, with their identifying codes for each type of feed.

Feeds Copied

The feeds section defines the market data feeds which FLM will subscribe to for data and compare to compute relative latency figures.

feeds > feed Copied

At least two feeds are required in order for the plugin to operate.

Each feed must be uniquely named so that they can be referenced in other parts of the plugin configuration. These names will also be displayed in the dataviews produced by FLM.

Mandatory: At least two feeds must be specified.

feeds > feed > rfa Copied

This section contains the configuration required to obtain data from a TREP platform. Further details about the feed adapter can be found in RFA adapter.

The RFA feed contains 3 settings:

feeds > feed > rfaWombat Copied

The RFA-Wombat feed type supports the same settings as the RFA feed type; it is provided to allow configuration of different field codes, since Wombat data fed through a TREP platform uses different codes than the native data.

The settings for this feed are identical to those of the RFA adapter.

feeds > feed > gl Copied

This section contains the configuration required to obtain data from a GL SLC. Further details about the feed adapter can be found in GL adapter.

feeds > feed > nyxt Copied

This section contains the configuration required to obtain data using a NYSE Openmama connection. Further details about the feed adapter can be found in the NYXT adapter section of this document.

feeds > feed > tt Copied

This section contains the configuration required to obtain data from a TT X_TRADER infrastructure. Further details about the feed adapter can be found in TT adapter.

feeds > feed > custom Copied

This section contains settings for custom field configurations. See the Custom field configuration section of this document for further details.

feeds > feed > testfile Copied

This section contains settings used to allow an FLM tick history file to be replayed as input for testing purposes. Further details about the feed adapter can be found in TestFile adapter.

Feed-specific algorithm settings Copied

feeds > feed > algorithm Copied

The feed-specific algorithm settings specified here override the settings defined globally to the FLM plugin.

Baseline feed setting Copied

Feed

This setting selects a feed configured in the section as the baseline feed for FLM.

When calculating latency figures, all data updates are compared against those of the baseline feed. Positive latency figures therefore indicate that updates are received later than those of the baseline; conversely negative figures indicate updates received earlier.

Mandatory: Yes

Instruments Copied

The instruments section contains configuration details for the market data items which will be requested by FLM for use in latency calculations. A minimum configuration must consist of at least one instrument common to the baseline feed and each monitored feed.

Each instrument must be given a unique name. These names will then be used to display instruments in the dataviews produced by FLM.

Instruments can be configured in two ways; individually or as patterns within a group. An instrument configuration includes the specific stock / request / symbol code for interested feeds, which must include the baseline feed. Configuring instruments within a group allow these codes to be generated from the instrument name using a pattern which may be easier to maintain.

Mandatory: Yes

Individual instruments Copied

instruments > instrument Copied

The instrument configuration element defines a single instrument for use by FLM. Configuration of an instrument must define codes two or more feeds, one of which is for the baseline feed.

The feed setting defines the feed(s) from which the instrument will be requested. Feeds are referenced by their name, as defined in the feeds configuration section.

Each referenced feed will need a feed-specific instrument code to request market data updates for the instrument. Please refer to the relevant section of Feed Adapters for the expected format of the instrument code for each feed type.

Instrument groups Copied

instruments > instrumentGroup Copied

An instrument group allows the configuration of instruments where the feed codes are generated from a feed-specific pattern and the instrument name. This can ease management of configuration, and is recommended when instrument codes are similar but differ only by the instrument name.

To configure an instrument group, specify feeds and patterns as if you were configuring a single instrument definition. In the pattern, instead of specifying the instrument code directly you should then use the escape sequence %i which will be replaced with the instrument name. A literal percent character can be defined by using the escape sequence %%.

You can then configure the list of instruments for the instrument group by name. Each name will be configured for each feed by expanding the defined pattern. Occasional special cases can be overridden by defining a feed-specific code for a particular instrument, in the same way as for an individually configured instrument. An instrument may also exclude a feed for which a pattern has been defined, by overriding the code for the feed to be an empty string.

Instrument groups must be configured with a name, but this is currently only for error reporting purposes. If using multiple groups however, it is recommended that these be given descriptive names to aid maintenance of the configuration.

For example, the configuration below contains 2 instruments.

The codes for these instruments will be expanded as:

• For feed RFA:
  • IDN_SELECTFEED.ARM.N
  • IDN_SELECTFEED.SCM.N
• For feed TT:
  • ARM.STOCK.ARM STK
  • SCM.STOCK.SCM STK

Fields Copied

The fields section defines the fields within a market data update for which FLM will compute latency figures. These fields must exist on both the baseline and feed instruments. Typical fields include the trade price, asking price and bid price of an instrument.

fields > field Copied

Configures a single field to be used by FLM. Each field must have a unique display name, so that they can be distinguished in the dataviews produced by the plugin.

Field codes are configured for feed types, rather than specific feeds because – in general – all of the feeds using a given feed adapter will use the same field codes.

Fields for the baseline feed type must always be configured; otherwise no latency comparisons can be performed.

Mandatory: At least one field is required

Additional settings Copied

There are several additional settings which control plugin behaviour globally (i.e. for all feeds, instruments, etc…). These settings are described below.

latencyDisplayUnits Copied

This setting controls the units used to display latency values in FLM plugin output. There are two possible values, milliseconds or microseconds. Depending on the value of this setting some dataview columns may have a different name; for example a suffix of Ms or Us for millisecond and microsecond units respectively.

Mandatory: No Default: milliseconds

valueThreshold Copied

Maximum difference between field values for ticks to be considered to match.

Mandatory: No Default: 0.0000001

timeThreshold Copied

This specifies the maximum value for the absolute difference (in milliseconds) between the timestamp of a feed tick and the timestamp of a matching baseline tick.

Mandatory: No Default: 2000ms

processDuplicateValueTicks Copied

If all values in a tick match the previous tick in the same feed, it is considered a duplicate and, by default, will be excluded from consideration by the matching algorithm.

To force FLM to consider these duplicate tick values, configure this setting to true.

Mandatory: No Default: false

fieldMatchSkew Copied

Number of milliseconds to be subtracted from calculated absolute latency for each field updated in both ticks when choosing between matched ticks.

Mandatory: No Default: 0

sampleAverages Copied

Up to two sample average columns can be configured. These columns are displayed in the Latency view, and produce averages of the averageLagMs value over the specified number of samples.

Mandatory: No

useFullTickRange Copied

If this setting is enabled, the matching algorithm will include ticks which have a timestamp later than the end of the current sample interval. Useful only when the test file feed adapter is being used.

Mandatory: No Default: false

dumpAlgoTables Copied

If this setting is enabled and the debug settings for the probe include “FLM:algo”, then in addition to general algorithm debug output, FLM will also dump the data tables that have been generated during execution of the algorithm.

Mandatory: No Default: false

collectInterval Copied

This deprecated setting previously affected the period at which FLM plugin collected received ticks from the feed threads, and logged them to the history file.

Ticks are now collected by the sampler only on each sample event. Any values specified for this setting will generate a warning in the Netprobe log, but are otherwise ignored by FLM.

Mandatory: No

htmlDir Copied

Report files generated by FLM are placed in the directory specified by this setting. FLM will then serve up these report files in response to HTTP requests.

This directory can optionally be within the path served by a web-server (e.g. Apache) in which case the httpPrefix setting should also be specified.

Mandatory: No Default: . (current working directory of Netprobe)

httpPrefix Copied

This setting controls the prefix of the URLs returned by FLM for a report generation request. If the prefix is empty, FLM will generate an appropriate prefix using the Netprobe hostname and listen port, in order to serve reports in response to HTML requests.

Setting a prefix implies that a different web-server will be taking responsibility for serving the report pages generated by FLM. In this case, report files should be placed directly in a folder served by the web-server, which can be controlled using the htmlDir setting.

Mandatory: No Default: (empty – Netprobe will auto-generate a HTTP prefix)

peaks Copied

FLM can optionally generate a new table row in the Latency view to indicate a high peak latency value. This feature is enabled by creating a peaks section in the sampler setup. The settings within this section further control the behaviour of peak rows.

Mandatory: No

peaks > minLatencyTime Copied

This setting controls how high the peak latency figure must be to generate a peak row, and is specified as a number of milliseconds. Peak values higher than this setting will cause a row to be created. A microsecond threshold can be specified using floating-point values in milliseconds. E.g. 0.468 milliseconds is equivalent to 468 microseconds.

Mandatory: No Default: 350ms

peaks > maxPeaksPerFeed Copied

This setting controls the maximum number of peak rows that can be created for a single feed. If no peaks section is created then the peaks feature is disabled, otherwise the maximum value is controlled by this setting. A value of 0 will prevent any peak rows from being created.

Mandatory: No Default: 5 (if the peaks feature is enabled, 0 if not)

peaks > singleFeedPeakPerSample Copied

If this setting is enabled, only the worst peak latency in each sample for a particular feed is added as a new peak row.

Mandatory: No Default: true

tickHistoryFile Copied

The tickHistoryFile section contains settings relating to the tick history file. This file is output by the plugin and contains a record of all ticks received, which is used to generate reports and can also be used for debugging purposes.

Mandatory: No

tickHistoryFile > filePrefix Copied

The filePrefix setting specifies the filename base which will be used for the tick history file. The plugin will then use this base and add the current date, which is the file that is then logged to. This file is relative to the Netprobe working directory – to specify your own location it is recommended to use an absolute path.

e.g. If the filePrefix is set to:

/home/geneos/flm/history

On 20th December 2010, the tick history file used by FLM will be:

/home/geneos/flm/history20101220

All files with a file prefix that matches this setting will be considered for the maxFiles limit.

Mandatory: No Default: FLM_<samplerName>.hist.

tickHistoryFile > maxFiles Copied

This setting specifies the maximum number of history files to keep, not including the history file for the current day. On checking the limit, the plugin will enumerate all files matching the pattern <filePrefix >YYYYMMDD* (where YYYYMMDD is a date) and keep the most recent files by date (up to the specified limit), removing all other files.

This limit is checked when the plugin starts, and whenever a history file is rolled over.

Note

Limit checking is performed before the archiveScript

Mandatory: No Default: Unlimited

tickHistoryFile > archiveScript Copied

The archive script setting allows users to specify a script, which will be called with the name of the current tick history file when this file is rolled over at the end of the day. This allows users the option to archive the history file somewhere, using a similar mechanism to the Netprobe log file.

Note

If you still wish for the history files to be managed by the

The maxFiles setting should not be set to 0 when using scripts, otherwise the history file will be deleted before the script is run.

e.g.

archiveScript = /bin/gzip

Mandatory: No Default: No archive script

matchedTickOutputFile Copied

This setting allows a file to be specified where the matched ticks will be written to at every sample period. The matched ticks are written as comma separated values format (CSV.) Please see the matched tick output file section for further information.

Mandatory: No Default: None (no file is produced)