Market Data Monitor

General information Copied

Use this document to implement, extend, or maintain monitoring solutions using the Market Data Monitor (MDM) plug-in, as well as for planning the deployment of solutions using MDM.

This provides information on how to install MDM and begin using the APIs to create market data monitoring solutions. This document provides a step-by-step guide on getting a basic monitoring solution running.

Readers of this document are assumed to have some understanding of:

• Configuring Geneos components, in particular how to add a new Netprobe or Sampler to a Gateway configuration.
• The configuration requirements of the market data systems and feeds you wish to monitor.
• How to write programs in at least one scripting or programming language.

A basic understanding of the Lua programming language will also be beneficial, as this document is not intended to be a Lua reference or tutorial. However, the Lua language is small and relatively easy to learn, so anyone with scripting or programming experience should be able to start working with the product very quickly. Links to some useful external resources are provided below for users new to Lua.

To have a better understanding of Market Data Monitor plug-in, read this user guide to know what the other MDM documentation intends to do.

Overview Copied

The Geneos Market Data Monitor (MDM) plug-in provides a scriptable environment which allows the creation of customised market data monitoring solutions.

A sampler script written in this environment can monitor large amounts of market data, reduce this data to a concise set of metrics, publish these metrics to a Geneos Gateway and respond to user actions.

The sampler script is run by an embedded interpreter for version 5.1 of the Lua programming language. On Windows and Linux, the interpreter used, LuaJIT, performs just-in-time compilation of the script. Lua was chosen for its simple but powerful syntax, small runtime footprint and good performance, especially when using the LuaJIT compiler.

MDM makes a number of APIs available to the script as Lua modules, each of which supports a step in the workflow from market data to published metric.

Data Subscription API Copied

Market data (that is, numerical quote and trade data) can be reported either from trading venues such as stock exchanges or published by in-house systems. Access to this data is typically obtained via direct market data APIs provided by third party vendors (such as Thomson Reuters, Bloomberg, TT).

The MDM plug-in is supplied with ready-built adapters for a number of feed vendor APIs. Custom feed adapters can be developed to interface to feeds from other vendors or from in-house systems using the MDM Feed Adapter API, which is documented separately.

The Data Subscription API allows sampler scripts to subscribe to market data from multiple instruments across multiple sources. Whatever the source, the data from each subscription is presented to the script as a normalised stream of ticks.

Note

ITRS and its subsidiaries will not be responsible for any third party charges arising from the use of its subscription APIs that may consume market data, trade orders, market messages or any other data feeds where you may be subject to fees. If you have any concerns with regards to such potential charges, you should contact your market data vendor in the first instance. The text and screenshots where we refer to existing instruments, the prices are entirely fictitious and have been created for demonstration purposes only.

Data Analysis API Copied

MDM sampler scripts can emulate the functionality of the Feed Latency Monitoring (FLM) plug-in, or go beyond this to support a broad range of monitoring and analysis capabilities including (but not limited to):

• Relative feed latency monitoring.
• Alerting on out of character spikes in data values.
• Comparative analysis of data values coming from multiple sources.
• Monitoring of expected vs. actual update rates for given instruments.
• Report Generation.
• Customised latency calculation algorithms and dataviews.

Publishing API Copied

The Publishing API is used to publish the metrics generated using the other APIs to the Geneos Gateway as dataviews. Rules, alerts and other Geneos features can be efficiently applied to these metrics by the Gateway, allowing users to take corrective action.

You can provide user actions back to the sampler through commands published by the script, so that alerts can be acknowledged or ad-hoc reports can be generated.

The MDM plug-in is designed to be highly flexible and open. In addition to the built-in MDM APIs, users may also make use of additional modules written by the Lua community that provide a wide range of functionality.

Recommended reading Copied

The following resources are recommended reading for users wishing to learn the Lua programming language.

Online Copied

Print Copied

Lua script and tutorial Copied

To know more about the Lua scripts and to view the available templates, see MDM Sampler Templates.

System requirement Copied

The Example feed adapter library, flm-feed-example.so (flm-feed-example.dll on Windows), is in the flm subdirectory of the Netprobe’s directory, or in a directory listed in LD\_LIBRARY\_PATH for UNIX, or in the PATH environment variable on Windows.

Install and set up Copied

Ensure that you have read and can follow the system requirements prior to installation and set-up of this plug-in.

Deploy Gateway Copied

To begin using the MDM plug-in with a Geneos Gateway, follow the steps below:

1. Determine which Gateway and Netprobe to use for deployment.

2. Download the tutorial Lua script and copy it to the Netprobe host.

3. In Gateway Setup Editor (GSE), add a new sampler for the “Market Data Monitor” plug-in.

   Configure the sampler with the path to the script you downloaded in the previous step:

   

4. Reference this sampler for use by your target Netprobe. Using Active Console (or another Geneos user interface), you must see a dataview similar to this:

   

Deploy stand-alone test Copied

It is also possible to run an MDM plug-in in a stand-alone mode, without the need for a Gateway. This mode allows an MDM monitoring Lua script to be tested prior to deployment.

To begin using the MDM plug-in in stand-alone mode, follow the steps below:

Download the tutorial Lua script and copy it to the Netprobe host.

Run the Netprobe as follows:

./netprobe.linux_64 -mdmtest code_examples/tutorial.lua

You must see output similar to the below. The script will continue sampling until test mode times out or the process is terminated with Ctrl+C.

<Wed Nov 13 17:48:16>  INFO: mdmTest Publish view: spreads
<Wed Nov 13 17:48:16>  INFO: mdmTest #####################
<Wed Nov 13 17:48:16>  INFO: mdmTest <!>samplingStatus  OK
<Wed Nov 13 17:48:16>  INFO: mdmTest <!>totalTicks      0
<Wed Nov 13 17:48:16>  INFO: mdmTest instrument  minSpread  maxSpread  ticksPerSample  maxInterval  tradePrice
<Wed Nov 13 17:48:16>  INFO: mdmTest ----------  ---------  ---------  --------------  -----------  ----------
<Wed Nov 13 17:48:16>  INFO: mdmTest #####################
...
<Wed Nov 13 17:48:17>  INFO: mdmTest Publish view: spreads
<Wed Nov 13 17:48:17>  INFO: mdmTest #####################
<Wed Nov 13 17:48:17>  INFO: mdmTest <!>samplingStatus  OK
<Wed Nov 13 17:48:17>  INFO: mdmTest <!>totalTicks      6
<Wed Nov 13 17:48:17>  INFO: mdmTest instrument  minSpread  maxSpread  ticksPerSample  maxInterval  tradePrice
<Wed Nov 13 17:48:17>  INFO: mdmTest ----------  ---------  ---------  --------------  -----------  ----------
<Wed Nov 13 17:48:17>  INFO: mdmTest GOOG
0.01       0.01       3               0.2          55.28
<Wed Nov 13 17:48:17>  INFO: mdmTest IBM
0.01       0.01       3               0.2          55.32
<Wed Nov 13 17:48:17>  INFO: mdmTest #####################

Deployment Copied

The Market Data Monitor (MDM) plug-in is a multi-threaded plug-in which uses one thread per data source (typically each feed) to subscribe and request market data. It will work efficiently on a multi-core machine, but this is not a requirement to run the plug-in.

Feed adapter Copied

The Netprobe package contains a number of standard feed adapters. Not all adapters are available on all the operating systems where the MDM plug-in is available; in most cases this reflects the availability of the feed platform APIs used.

You may also refer to the Geneos Compatibility Matrix for the list of supported platforms.

In addition to the adapters listed above, the Netprobe package also contains Example and TestFile adapters on all the above platforms.

The Example adapter is a simple feed adapter which accepts all instrument subscriptions and returns random data, and is used for several examples in this guide.

While, the TestFile adapter allows a tick history file generated by one environment to be replayed in another, typically for testing purposes.

Feed adapters are implemented as shared libraries. In the default package layout, the standard feed adapters are located in a Netprobe sub-folder named (for legacy reasons) flm. When a standard feed adapter is specified via the Data Subscription API, Netprobe will load it from this directory if possible. Alternatively, if you use a customised Netprobe folder layout, you should ensure that the feed adapter libraries are available on the LD\_LIBRARY\_PATH for UNIX, or in the PATH environment variable on Windows.

Most feed adapters require access to a TCP/IP resource in the market feed infrastructure; for the TT feed the Netprobe must be running on a Windows host with an X_Trader Pro license. Additional information about deploying each of the supported feed adapters can be found in the Feed Adapters section of this documentation.

The example below shows the configuration for two named feeds, one of which is the base feed. You should edit the configuration file to configure feeds for your own environment. At least two feeds are required to perform relative latency calculations, one of which must be nominated at the base feed. Additional comparison feeds can be added as required.

The feed names (baseFeed and compFeed in the example) should be adjusted to something more meaningful. These names will be used as display names in the dataview output, and are also used to reference instrument and field configurations.

The unedited configuration file contains the following feed definitions. These feeds use the Bloomberg adapter. See Feed Adapters for the applicable settings for your feed type.

-- Feed configuration
-- Base feed
configureFeed("baseFeed", true,
{ feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"}, bloomberg  = { serverHost = "bpipe.ldn.itrs" }
})
-- Comparison feed 1
configureFeed("compFeed", false,
{ feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"}, bloomberg  = { serverHost = "bpipe1.ldn.itrs" }
})
-- Add additional comparison feeds here, if required

Lua scripting environment Copied

The Lua runtime environment and Geneos Lua modules are embedded within the Netprobe binary; they have no special configuration requirements. In particular, the Geneos Lua modules do not depend on any special setting of the LUA\_PATH and LUA\_CPATH environment variables.

However, if your MDM script makes use of external Lua modules, you will probably need to set one or both of these variables for the Netprobe.

In addition to the default locations (and any locations defined by the LUA\_PATH environment), the Netprobe will append the debug/lua folder in the Netprobe installation directory to the package.path. Similarly, the folder debug/lib will be appended (after any LUA\_CPATH environment additions) to the package.cpath.

On the platforms for which it is available, the LuaJIT interpreter is used.

Gateway Setup Editor (GSE) settings Copied

Script Copied

The MDM plug-in executes the Lua sampler script specified on the Script field. The script can be specified as an absolute path, or as a path relative to the Netprobe working directory.

Note

Lua scripts are read during the sampler start-up, and are stored in the cache afterwards. If you update the script for an MDM sampler, then you must restart the Netprobe to use the updated script.

Parameters Copied

An optional setting, Parameters, allows configuration parameters specific to the script to be set up in the GSE. The script can access them as described in MDM Writing Samplers and Feed Adapters.

Parameters should be entered in the multi-line text field provided, in the format name=value with one parameter per line. Values (but not names) will be trimmed of whitespace. A line starting with a # character is treated as a comment and ignored.

Alternatively, the parameter list may be configured using variables, with the following limitations:

• If using a name-value-list variable or string-list variable, the Parameters field must contain only a reference to that variable and nothing else.
• Variables cannot contain structured data. Variable types which are safe to use include Boolean, double, integer, and string.

Inline Files Copied

Inline Files are defined as an optional setting in the Advanced tab of the plug-in configuration. This setting allows files to be defined inline in the Gateway setup file (i.e. contained within it). When the sampler configuration is transmitted to Netprobe, the file content is written out to disk before the MDM plug-in script is executed.

Each inline file definition must have a filename specified, which is the location on the Netprobe host where the file content will be written. Relative paths will be interpreted relative to the Netprobe working directory. Absolute paths are also permitted.

For an inline file to be successfully written by Netprobe, the target directory must exist and be writeable by the Netprobe user. If the inline filename matches an existing file, the existing file will be overwritten by the Netprobe on setup change (assuming the file is writable by Netprobe). Any errors that occur when writing inline files to disk will be output to the Netprobe log file.

As inline files are processed before running the MDM plug-in script, it is possible to define the script as an inline file. To do this, the script setting and inline filename must match. Other configuration files such as the RFA config file or the NYXT MAMA properties file may also be transferred using this method.

Note

the GSE does not provide any syntax highlighting / checking for inline files. It is suggested that Lua scripts (and other files) be created in an IDE or editor, before being copied & pasted into the GSE. Also note, there are known GSE rendering issues that make editing scripts inside the GSE problematic. As a workaround, the view can be reset by changing the display mode (in the top-right corner) to Summary mode, then back to Normal.

Debug Logs Copied

The MDM debug logs contain quite a number of information useful for diagnosing problems in the plugin. These include information on feeds, instruments, subscriptions, and function calls among others.

There are two ways to enable these logs. First is by setting the environment variable DEBUG as “MARKET-DATA-MONITOR”. For example:

setenv DEBUG "MARKET-DATA-MONITOR"

Second method is to enable the debug logs through the GSE probe settings.

Stand-alone test mode Copied

An MDM Lua script may be executed from the command-line using the Netprobe for testing purposes.

This stand-alone mode has several differences from production execution by a Gateway:

• The script will ‘sample’ every second, instead of allowing a configurable sample interval. “Real” sampling intervals (for example every N seconds) may be approximated by only performing sampling logic every N calls.
• Published dataviews will be output in a human-readable format to the Netprobe log.
• User-defined commands will not be executed (as there is no GUI with which to execute them). User commands may be tested by writing code to run the callback function during sampling.
• Only a single test script may be run at a time. Netprobes connected to a Gateway may run multiple MDM samplers concurrently.
• Test mode can be exited tidily by returning false from the doSample function. This has no effect if executed in a sampler started by the Gateway.
• The test mode will end after running for 30 minutes.

The syntax for running a script in this mode is as follows:

./netprobe.<platform> -mdmtest <luaScript> [paramName1=paramValue1 [paramName2=paramValue2 ...]]

Any parameters passed are available to the script as if they had been set up in the Gateway Setup Editor. Note that parameter arguments that contain spaces or shell special characters need to be quoted or escaped as usual for the shell you are using.