MDM Writing Samplers and Feed Adapters - User guide

Publishing API

geneos.sampler

This module provides the mechanisms that allow an MDM script to act as a Geneos sampler, publishing content to the Gateway. Dataviews can be created and updated directly from Lua code and commands can be published to create points of user interaction and workflows within the monitoring.

geneos.sampler.commands

This sub module provides a fluent interface for constructing commands, allowing complex definitions to be created using legible and maintainable code.

Publish a Dataview

The usual way to get output from a Geneos sampler is for it to create and update a dataview. An MDM sampler script can produce zero or more views each containing a number of headlines, rows and columns. Dataviews may be created at any time, but it is recommended that the view(s) be created during script initialisation, and subsequently updated and published each sample.

We will now add code to the skeleton sampler script above to create and publish a dataview. The changes made to this script are discussed below. Since the connection to market data process is yet to be discussed, you cannot put any market data into the view. This will simply publish some static table data, with a headline samples that counts the number of times doSample is called (and so updates each sample).

-- Create data view(s)
-- ===================

-- Define columns and create view
local cols = { "instrument", "minSpread", "maxSpread", "ticksPerSample", "maxInterval", "tradePrice" }
local view = assert(gs.createView("spreads", cols))

-- Publish initial view data
view.headline.samples = 0
view.row['sample'] = { "our", "data", "will", "appear", "here"}
view.row['dummy'] = { ['ticksPerSample'] = 0 }
assert(view:publish())

Views are created by the createView function, which expects a view name and a list of column names. If column names are valid Lua identifiers code that updates views can use a more concise / readable syntax. In any case, it is strongly recommended that column names should use only the alphanumeric characters, underscore, dash and space [A-Za-z0-9_-]. View names must be unique within each sampler, and column names must be non-empty and unique within the view. Notice the use of the Lua assert function to check for any error returned by createView.

The view object returned by createView is a Lua table with member functions to allow it to be published and refreshed. It also has members called headline and row which are themselves tables representing the content of the view. A newly created view has an empty row table and a single headline variable named samplingStatus with a value of OK.

A headline can be created or updated simply by assigning a value to it; it can be deleted by assigning nil. In the script above, we add the samples headline just after creating the view and we update it in the doSample method. Notice that because samples is a valid identifier, we can use the short syntax view.headline.samples. If we had instead called the headline #samples, we would have had to write view.headline['#samples'].

Rows are added, updated and removed in a very similar way. However, a row has a value for each column, so the value associated with a row is also a table. When you assign a table value to a row, you can use either an array (as for row['sample'] above) or a mapping (as for row['dummy']). If using an array, then the order of values in the array should match the order of the column names specified when the view was created (minus the first column which represents row names). If using a mapping, the mapping keys should be strings matching the column names. Notice that, because ticksPerSample is a valid identifier, the mapping could have been given using the short syntax: { ticksPerSample=0}.

-- Define the doSample function which will be called every sample
-- ==============================================================

local numSamples = 0
gs.doSample = function()
    numSamples = numSamples + 1
    view.headline.samples = numSamples
    assert(view:publish())
end

The code fragment above will increment a count variable every sample. This count is displayed in the output dataview as the value of the samples headline, which is updated simply by overwriting the entry with a new value. Updating table cells can be done either by replacing the entire row with a new table, or updating specific cell values within the table as required.

View content changes will not immediately be seen by the Gateway; you must call the publish member function to publish your changes. It is recommended that the publish function only be called once per view per sample.

Finally, to make it easier to check the progress of our sampler, we will also add a couple of logMessages at the start and end of our script:

-- Load the API modules used by the script
local gs = require 'geneos.sampler'
gs.logMessage('INFO', "Starting")

gs.logMessage('INFO', "Started OK")
-- End of script

The script now displays and updates (part of) a dataview:

Subscribe to Market Data

To fulfil the primary purpose of a Market Data Monitor plug-in, we need to subscribe to some data. For this tutorial we will use the Example Adapter to provide (random) data, since it is very simple to configure and requires no external data source.

The following code examples have been applied to the publishing dataviews example, to form a new updated script.

Subscribing to market data involves:

1. Identifying the feed adapter to be used.
2. Providing configuration information specific to that adapter.
3. Identifying the instruments to be subscribed to.
4. Identifying the fields to be subscribed to for each instrument.

All this information is collected together into a Lua table of 'type' FeedConfig, and is a required parameter when creating a feed. The first part of the configuration tells the Feed Controller which adapter (library) is to be used:

-- Configure and connect to market data sources
-- ============================================

-- Feed configuration is stored in a Lua table
local feedConfiguration = {}

-- Feed Controller configuration
feedConfiguration.feed = {
    type = "example",
    ["library.filename"] = "flm-feed-example.so",
}

The next part sets parameters specific to the feed adapter. The FeedConfig field name should match the type setting in the previous part, in this case example. The Example adapter only has one parameter, the number of milliseconds it pauses (waits for) between publishing tick updates.

-- Feed adapter configuration
feedConfiguration.example = {
    publishingPeriod = 200,
}

The next two parts specify the instruments and fields for the initial set of subscriptions. If no initial subscriptions are required, these fields may be omitted. Each of them maps a set of names used for display by the sampler to a set of codes understood by the feed. This allows verbose codes used by the feed to be mapped to concise names that are understood by the end users; it also allows inconsistent codes used by different feeds to be mapped to a common set of names.

In general, you will need to refer to the documentation for the feed adapter and that of market data vendor to determine the correct instrument and field codes. In the case of the Example feed, the instrument codes used are unimportant; any string value given creates a new instrument subscription. Field codes are again mostly unimportant; there are a few special field codes, but any other code will simply generate values where each is 0.01 greater than the previous value. To make the data slightly more realistic, we order the field codes so that Bid is generated before Trade and Ask after.

-- Define the set of instruments
-- Instrument subscriptions are defined as a mapping from display name to stock code
feedConfiguration.instruments = {
    GOOG = "DATA_SERVICE.GOOG",
    IBM  = "DATA_SERVICE.IBM",
}

-- Define the set of fields
-- Fields are defined as a mapping from display name to field code
feedConfiguration.fields = {
    Bid   = "01_BID",
    Trade = "02_TRADE_PRICE",
    Ask   = "03_ASK",
}

Having set up the configuration we can now create and start the feed. This is done using the addFeed() function in the geneos.marketdata module, so we must add a reference to this module in our script.

-- Load the API modules used by the script
local gs = require 'geneos.sampler'
local  md  =  require  'geneos.marketdata'

-- Create and start a feed using the above config
local tutorialFeed = assert(md.addFeed("Tutorial-Feed", feedConfiguration))
tutorialFeed:start()

The addFeed() function returns a Feed object. This a Lua userdata object with methods to allow the feed to be started and stopped, subscriptions to instruments to be added or removed and data to be collected. The name passed when the feed is defined (Tutorial-Feed in this tutorial) is used to identify the feed in log messages issued by the market data API and by the Feed Controller. It should be unique among feeds created by the sampler. Note again the use of assert to check for any errors when adding the feed.

We can now omit the rows sample and dummy when publishing initial view data, which were added for the "Publishing a Dataview" part of this tutorial. Instead we create / update a dataview row per instrument during a doSample function call, populating the rows with tick data from our subscriptions.

-- Define the doSample function which will be called every sample
-- ==============================================================

local numSamples = 0
gs.doSample = function()
numSamples = numSamples + 1
view.headline.samples = numSamples

    local firstTick, lastTick = tutorialFeed:getTicks('GOOG')
    local lastTrade = lastTick and lastTick.field['Trade']
    view.row['GOOG'] = { tradePrice = lastTrade }

    firstTick, lastTick = tutorialFeed:getTicks('IBM')
    lastTrade = lastTick and lastTick.field['Trade']
    view.row['IBM'] = { tradePrice = lastTrade }

    assert(view:publish())
end

Once a feed has been started, a stream of ticks starts to build up for each instrument subscription. These ticks are fetched by calling the getTicks() method of the feed object. This returns two values: the first and the last tick received since the last time the subscription was queried. If there were no ticks, both these return values will be nil - this will generally be the case on the first sample.

The ticks are provided as a linked list, meaning all the ticks can be accessed by following the links from the first. We will show this in more detail in the next section of this tutorial. For now we will just display the 'Trade' field from the last (i.e. latest) tick of each instrument. Note that each tick is a Lua table which contains a table called field which in turn contains the field values. Note also the use of the Lua and operator to guard against attempting to access lastTick.field if the variable is nil.

The script now populates one of the columns identified when the dataview is defined:

Analyse the Tick Data

Although it is possible to copy all the values of each tick into a dataview row and use Gateway rules to compute statistics, this would be a poor design choice. The work the Netprobe would do to copy the data would be more than the work needed to analyse it. Similarly, the Gateway would need to process large volumes of row updates before it could begin to run rules on the data. Instead, an MDM sampler should compute the required metrics from the tick data and publish only these metrics to the Gateway.

Because the Example Adapter a simplistic simulator which does not attempt to reproduce live market data, there are limits to the analysis we can usefully do in this tutorial. This section demonstrates the principles involved, which can be extended as required to perform more complex analysis on real input data.

The metrics we will calculate for each instrument are:

• Minimum and maximum spread per sample (calculated as difference between bid and ask).
• Number of ticks received per sample.
• Maximum time interval between ticks.
• Last trade price.

We will also replace our count of samples with a count of the total number of ticks received.

To analyse each tick for an instrument, we will need to iterate over all the ticks received. Ticks are provided as a singly-linked-list with a next field, so to iterate we just need to follow these links until the next tick is nil. As an alternative, we could instead call geneos.marketdata.ticksToArray() and iterate over the ticks in an array.

local tick = tutorialFeed:getTicks('IBM')
while tick do-- TODO use tick data in calculations
    tick = tick.next
end

For "per-sample" metrics this loop is sufficient, and we can do all of our calculations in the "TODO" section of the loop above. However, to calculate our example metric "maximum time interval between ticks" we need to store some data between samples. In this case we store the last tick received in each sample, so that we can compare its arrival time with the time on the first new tick, thus computing the time interval.

The loop to iterate over the ticks from the current sample then becomes the following:

local prevTick = nil

function processTicks()
    local tick = prevTick or { field = {} }
    tick.next = tutorialFeed:getTicks('IBM')

    -- TODO Extract required values from 'tick'
    -- This will be either the last tick from the previous sample, or an empty sentinel value.

    -- Iterate over the ticks
    while tick.next do
        tick = tick.next
        -- TODO Use tick data to calculate metrics
    end

    -- Store last tick of this sample for next time
    prevTick = tick
end

Continuing to update our tutorial script developed in the previous sections, use this script that includes the changes described below.

Define a helper function for rounding computed values to two decimal places for display purposes:

-- Define functions to analyse the data
-- ====================================

-- Utility function to round non-nil (positive) values to two decimal places
local function round2dp(value)
    return value and math.floor(value * 100 + 0.5) / 100
end

Next, define a function to perform the analysis of the ticks received in the current sample, for a single instrument.

-- A table to map instrument names to the last tick seen for each instrument
local prevTick = {}

-- A function to process the sample data for a specific instrument.
-- Takes a single string argument; the instrument name.
local function processTicks(instName)
    local tick = prevTick[instName] or { field = {} }
    tick.next = tutorialFeed:getTicks(instName)

    -- Extract required values from 'tick'
    local prevTime = tick.timeLast

    -- Metrics we will compute from the tick data
    local tickCount, maxInterval, minSpread, maxSpread
    tickCount = 0

    -- Iterate over the ticks
    while tick.next do
        tick = tick.next

        -- Use tick data to calculate metrics
        tickCount = tickCount + 1

        local interval = prevTime and tick.timeFirst - prevTime
        if not maxInterval or interval > maxInterval then maxInterval = interval end
        prevTime = tick.timeLast

        local spread = tick.field.Ask - tick.field.Bid
        if not minSpread or spread < minSpread then minSpread = spread end
        if not maxSpread or spread > maxSpread then maxSpread = spread end
    end
    prevTick[instName] = tick

    -- Create a table of row values.
    -- We will place this result directly into a view, so the field names must match the column names.
    local rowValues = {
        ticksPerSample = tickCount,
        maxInterval = round2dp(maxInterval),
        minSpread = round2dp(minSpread),
        maxSpread = round2dp(maxSpread),
        tradePrice = tick.field.Trade -- Trade price is taken from the last tick seen
    }

    -- Return two values; (1) a table of row values and (2) the count of new ticks
    return rowValues, tickCount
end

This function follows the tick iteration loop which keeps the last tick from the previous sample. The example is extended slightly to take an argument specifying which instrument to process. This means we must also change the tick storage to store the last tick on a per-instrument basis, and so prevTick becomes a map instead of a single value.

The calculations in the body of the while loop generate the metrics which will be published to the view for that instrument. After iterating over all ticks, the row for the view is generated and returned to the caller along with the number of ticks processed.

The functions defined so far take care of computing metrics at the level of each instrument subscription. We also want to show how summary statistics can be calculated and displayed, so we replace the samples headline with one to show totalTicks:

-- Publish initial view data
view.headline.totalTicks = 0
assert(view:publish())

Finally, we modify the doSample function to call the analysis function processTicks for each subscribed instrument. The first result returned by processTicks is applied to the dataview row; the second is used to compute the value for the totalTicks headline.

-- Define the doSample function which will be called every sample
-- ==============================================================

local totalTicks = 0
gs.doSample = function()
for name, _ in pairs(feedConfiguration.instruments) do  
-- Loop over each subscribed instrumentlocal rowResult, tickCount = processTicks(name)     
-- processTicks() returns 2 values
view.row[name] = rowResult                          
-- update view row for this instrument with new metrics
totalTicks = totalTicks + tickCount                 
-- update tally of total ticks observed
    end
    view.headline.totalTicks = totalTicks
    assert(view:publish())
end

The script now populates all the columns in the dataview:

This documentation describes the Feed Adapter (data input) API that a shared-object must expose to operate with MDM plug-ins. Use of this API will enable clients to create their own adapter libraries to interface with their own proprietary systems. The document also explains the packaged C++ wrapper API for C++ development, and the Lua Feed API for development of feeds via Lua scripting.

The Geneos Market Data Monitor (MDM) plug-in provides a programmable environment allowing users to create customised market data monitoring solutions.

Input data to MDM plug-in instances are provided by feed adapter library shared-objects (.so files) on UNIX platforms, or dynamic-link libraries (.dll files) on Windows platforms. These software modules translate market data obtained via the use of a direct market data APIs into a normalised format expected by the plug-in.

Audience

This document is intended for users who wish to write a custom feed adapter shared-object (or DLL) to provide an additional data input for the Market Data Monitor plug-in.

Users developing against the C API are expected to understand the C programming language.

Users developing against the C++ API are expected to understand the C++ programming language. They will also need an understanding of the C API that the C++ API is built upon. The packaged C++ example feed code requires the GCC C++ compiler 3.4.6 (or better) and GNU Make 3.80 (or better) to compile. A Visual Studio 2010 solution is also provided for compilation on Windows platforms.

Users developing using the Lua Feed API are expected to understand the Lua programming language.

Package Contents

The API package file contains a single folder mdmFeedAdapterApiKit. The contents of this folder are as follows:

API Interface

Connectivity to each type of market data system is provided by a separate feed adapter library (.so or .dll file). Communication with each library (including loading the library) is managed by an internal component of Netprobe called the Feed Controller. This component also handles combining requests and distributing ticks for multiple plug-in instances.

Each feed adapter library must export the functions defined by the Feed Adapter C API. These functions must be present for the Feed Controller to load the library, and are used to control and communicate with the adapter library. The functions fall into 5 main categories:

• Library management - initialisation and un-initialisation.
• Feed management - creation and destruction.
• Instrument management - subscription and un-subscription.
• Reporting - errors and status updates.
• Publishing - tick data.

Architecture

Tick data flow through feed adapter libraries to Netprobe plug-in instances.

The diagram above displays the flow of tick data through the system, provided via 3 different feed implementations. The blue boxes (with white text) indicate modules to be written by the developer. These modules will communicate with a market data system using the vendor-provided API, and transform the data as required to a format suitable for input to Netprobe.

C Feed Adapter

A C feed adapter is a shared-object written in C that exports the functions defined by the Feed Adapter C API.

C++ Feed Adapter

A C++ feed adapter is a shared-object that still exposes the functions from the Feed Adapter C API, but where the code internally is written in C++.

The API Kit provides source code for an optional C++ "api" library, which wraps the C API in an object-orientated interface, and provides some common utility methods useful for feed adapter development. This package also provides source code for an example C++ feed implementation, which is discussed later in this document.

Lua Feed Adapter

The Lua feed adapter library geneos-feed-lua.so is a standard library provided with the Netprobe package. This adapter provides a Lua version of the Feed Adapter C API, allowing execution of a Lua script to provide tick data. Lua scripting allows rapid development of new feed adapters without the need to compile code.

In the diagram, the script file inputFeed.lua is being executed to provide feed data. This example script is provided as part of the API Kit package and is discussed later in this document.

Feed Adapter interaction with Netprobe

Each feed adapter library is loaded once only, and used to host all feeds configured for that adapter type. For example, multiple feeds connecting to different TREP components may be configured, but the flm-feed-rfa.so adapter library will only be loaded by Netprobe once. If the configuration is changed to remove all feeds for a given adapter type, the feed adapter library will be unloaded from the Netprobe process.

Changes in feed configuration from plug-in instances are processed by the Feed Controller and presented to each affected feed adapter library as a series of API calls. The exact calls will vary according to the setup changes made, but typically consist of instruments being added to (or removed from) a feed instance.

Feed adapters should push tick data to the Netprobe on arrival, via the provided function calls. In particular, the feed adapters should not wait until an API call is made from the Netprobe to provide data. As such it is likely that feed adapter library code (or the direct market data API) will need to execute in a separate thread. Input data will be buffered by Netprobe (up to a limit) until the plug-in performs a sample, at which point it is processed and consumed.

If multiple MDM plug-ins are configured, plug-in instances will share the loaded feed adapter libraries; each library will still be loaded once only. Moreover if feed definitions in different plug-in instances are identical, then the MDM plug-ins will share the feed between them. This behaviour ensures that the Netprobe presents the minimum number of connections to each market data system.

By implementing the API interface, your feed adapter library will support these behaviours by default. No special handling for these conditions is required.

Warning:

The feed adapter library is a shared-library, which means it shares the same memory address-space within the Netprobe. As such, system calls such as exit, abort, terminate will end the Netprobe process and thus should be avoided. Additionally, coding errors such as dereferencing invalid pointers, memory leaks or file handle leaks can all negatively affect the Netprobe.

It is recommended that the custom library be thoroughly tested before production usage.

Compiling the C template file

As discussed in the Concepts section, a feed adapter library must export the methods defined by the Feed Adapter C API.

A template C source file containing a stub feed adapter library implementation can be found in the C template section. This template can be used as a starting point when developing your own feed adapter library.

To compile on Linux, copy or save the file to your machine and run one of the commands shown below. The APIKIT_DIR variable should refer to the directory that the API Kit was extracted to (this directory should contain an "api" folder). You can either set APIKIT_DIR as an environment variable prior to running the command, or replace the variable reference with the path directly.

$ gcc -Wall -fPIC -shared -I${APIKIT_DIR} template.c -o template.so
or
$ gcc -Wall -fPIC -shared -I/path/to/kit  template.c -o template.so

Reporting errors

Feed Adapter C API calls into the feed adapter library are all made by the main Netprobe thread. If any function returns a failure result (i.e. the value 0), the Netprobe will attempt to obtain a description of the error via the get_error_text() function. This error text will then be output in the Netprobe log.

The get_error_text() function is expected to behave in a similar manner to the C library function snprintf. It should print a NUL-terminated string to the input buffer describing the latest error resulting from an API call, truncating the message as necessary. If the input buffer is not large enough to contain the entire message then the length of the required buffer should be returned, otherwise 0 should be returned.

The Netprobe will attempt to obtain the error message with an initial buffer size of 2048 (2kB). If this is not sufficient, the call will be repeated with a buffer increased to the size returned by the function.

A reference implementation of this function is provided as part of the C template example code.

Library management

The initialise_library() function is called immediately after the feed adapter library is loaded by the Netprobe. This call provides the feed adapter library with a set of function pointers used for publishing data to the Netprobe. These pointers will be valid for the lifetime of the feed adapter library.

The initialise function can also be used to initialise any dependent libraries of global state required by all feeds, perhaps as an alternative to platform specific shared-object initialisation (e.g. the GCC __attribute__((constructor))syntax for Unix, or the DllMain function on Windows).

The finalise_library() function is called immediately before the feed adapter library is unloaded by the Netprobe. This function may be used to perform a tidy shutdown where necessary.

Feed management

The create_feed() function is called to create a new feed instance in the feed adapter library. This function will be called once per unique feed configuration present on the Netprobe. Multiple plug-in instances with the same feed configuration will result in only a single call to create_feed(), meaning access to the feed will be shared between plug-in instances.

A feed identifier is generated by the Netprobe and passed to the feed adapter library. This id will be used to reference the new feed for the remainder of its lifetime, and must be provided when publishing data. The feed is also passed configuration information, as a list of key-value pairs. The num_params argument gives the number of entries present in the keys and values arrays. The code example below shows how to print all feed parameters when the feed is created:

int create_feed(feed_id_t id, size_t num_params, const char** keys, const char** values)
{
    for (size_t i = 0; i < num_params; ++i)
        printf("key '%s' has value '%s'\n", keys[i], values[i]);
    return 1;
}

The destroy_feed() function is called to destroy a previously created feed when it is no longer required. Upon completion of the function call, no further data should be published for the feed. If the feed is running in a thread, it may be necessary to block in the destroy_feed() call until the thread has been terminated.

Both functions should return non-zero on success. For failure return zero and ensure the next call to get_error_text() returns a populated error message.

During sampling, the MDM plug-in will attempt to obtain the latest feed status by calling the get_feed_status() function. This function should return a short summary of the feed status that is suitable for display in a dataview. The full error text for any problems with a feed should be reported via a log message, or via the get_error_text() function for API calls failures.

The get_feed_status() implementation should behave as get_error_text() does; copying the feed status into the buffer provided. One added complication with this function is that the feed status will be requested by the main Netprobe thread, but (typically) populated by a feed thread. Therefore you must ensure adequate locking (e.g. by using a mutex) around access to this data. Alternatively, for feeds with a limited number of states it may be possible to use a C sig_atomic_t type (or C++11 atomic types) to indicate the current feed state without locking.

Instrument management

Instrument subscriptions for a given feed are requested via the subscribe_instrument() function. The function is passed a Netprobe-generated identifier for the subscription (the inst_sub_t type) along with the instrument stock code (or other string-based identifier). The subscription identifier is an opaque pointer to a subscription record maintained by the Netprobe. This identifier (along with the feed id) is required information when publishing data.

The instrument fields that MDM plug-in is configured with are passed as an array of string values. The order of field names in this array is important, because when a tick is published back to the Netprobe the field values are expected to match this same order. Example C code for printing a subscription request is shown below:

int subscribe_instrument(feed_id_t feed, inst_sub_t sub, const char* instrument,
                         size_t num_fields, const char** field_names)
{
    printf("new subscription request for feed(%d), subscription(%p)\n",
    feed.id, sub.data);
    for (size_t i = 0; i < num_fields; ++i)
        printf("  index %02d, field name '%s'\n", i, field_names[i]);
    return 1;
}

An unsubscribe_instrument() API call will be made by Netprobe when an instrument is removed from the configuration (or in some cases when another plug-in instance is added that uses the same feed and instrument). Once the function call has completed, the subscription id (inst_sub_t) becomes invalid and must not be used to publish any further data. As for the destroy_feed() function, for threaded feeds it may be necessary to block in the unsubscribe_instrument() call until the feed confirms that the instrument is no longer in use.

Quick Start

To get started with an example Lua feed implementation please follow the steps below:

1. Download the API Kit package, and a suitable Netprobe for your operating system.
2. Unpack the contents of both packages into the same directory.
3. Obtain an MDM application key from your ITRS support contact (showing that you have accepted the Geneos API agreement) and place it in your Netprobe directory.
4. Run the Netprobe with the -mdmtest command-line option, to execute the example in stand-alone mode (see example below).
5. Alter the mdmFeedAdapterApiKit/scripts/inputFeed.lua script as required to implement your own feed adapter.

An example terminal session showing these steps on Linux is shown below. For other platforms, you will need to change the Netprobe suffix in the commands below (e.g. use netprobe.sunx86 instead of netprobe.linux). Users on Windows machines may also need to alter the path separators to use \ (backslash) instead of / (slash).

$ ls
mdm_appkey.txt mdmFeedAdapterApiKit.tar.gz netprobe.linux.tar.gz
$ mkdir unpack
$ cd unpack
$ tar xf ../netprobe.linux.tar.gz
$ tar xf ../mdmFeedAdapterApiKit.tar.gz
$ cp ../mdm_appkey.txt .
$ ./netprobe.linux -mdmtest mdmFeedAdapterApiKit/scripts/mdmTest_geneos-feed-lua.so.lua
... <Netprobe test output>

Example Netprobe test output is shown below.

<Fri Nov 15 17:04:56>  INFO: mdmTest Reading key file 'mdm_appkey.txt', contents listed below.
<Fri Nov 15 17:04:56>  INFO: mdmTest Client      : ITRS Group
<Fri Nov 15 17:04:56>  INFO: mdmTest Contact     : Doc-Generator
<Fri Nov 15 17:04:56>  INFO: mdmTest Test mode   : Enabled
<Fri Nov 15 17:04:56>  INFO: mdmTest Gateway mode: Enabled
<Fri Nov 15 17:04:56>  INFO: FeedController Create feed request for 'LuaFeed' from plugin 'mdmTest'.
<Fri Nov 15 17:04:56>  INFO: FeedController First-time load of shared-library geneos-feed-lua.so
<Fri Nov 15 17:04:56>  INFO: FeedController:AdapterLibrary Successfully loaded feed library 'flm/geneos-feed-lua.so'. Reported version: 'RA-131115'.
<Fri Nov 15 17:04:56>  INFO: FeedController:AdapterFeed Calling create_feed() with parameters:
  feed.library.filename => geneos-feed-lua.so
  feed.name => LuaFeed
  feed.type => lua
  feed.verbose => true
  lua.script => mdmFeedAdapterApiKit/scripts/inputFeed.lua
  lua.value.max => 1000
  lua.value.min => 1
<Fri Nov 15 17:04:56>  INFO: FeedController:AdapterFeed Feed creation successful, feed_id=0.
<Fri Nov 15 17:04:56>  INFO: Computing subscription differences for feed LuaFeed (mdmTest)
<Fri Nov 15 17:04:56>  INFO: Existing subscription data: (none)
<Fri Nov 15 17:04:56>  INFO: Unaltered subscriptions: (none)
<Fri Nov 15 17:04:56>  INFO: New subscriptions:
  LuaFeed: CODE.INST.1 (Inst1) [Ask, Bid, Trade]
  LuaFeed: CODE.INST.2 (Inst2) [Ask, Bid, Trade]
<Fri Nov 15 17:04:56>  INFO: FeedController:AdapterFeed Subscribing to instrument 'CODE.INST.1' for feed 'LuaFeed' (sub=0xb823c70). 3 fields; [Ask, Bid, Trade]
<Fri Nov 15 17:04:56>  INFO: FeedController:AdapterFeed Subscribing to instrument 'CODE.INST.2' for feed 'LuaFeed' (sub=0xb823870). 3 fields; [Ask, Bid, Trade]
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameters:
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'feed.library.filename' has value 'geneos-feed-lua.so'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'lua.value.min' has value '1'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'feed.verbose' has value 'true'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'lua.value.max' has value '1000'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'feed.name' has value 'LuaFeed'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'feed.type' has value 'lua'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Parameter 'lua.script' has value 'mdmFeedAdapterApiKit/scripts/inputFeed.lua'
<Fri Nov 15 17:04:56>  INFO: LuaFeed 
<Fri Nov 15 17:04:56>  INFO: LuaFeed Using minimum = 1
<Fri Nov 15 17:04:56>  INFO: LuaFeed Using maximum = 1000
<Fri Nov 15 17:04:56>  INFO: LuaFeed Added subscription for instrument 'CODE.INST.1'
<Fri Nov 15 17:04:56>  INFO: LuaFeed Added subscription for instrument 'CODE.INST.2'
Sample 1  Status OK
LuaFeed Inst1 F3=591, F2=795, F1=699
LuaFeed Inst2 F3=662, F2=754, F1=81
Sample 2  Status OK
LuaFeed Inst1 F3=427, F2=340, F1=736
LuaFeed Inst2 F3=55, F2=569, F1=115
Sample 3  Status OK
LuaFeed Inst1 F3=176, F2=975, F1=706
LuaFeed Inst2 F3=355, F2=975, F1=277
Sample 4  Status OK
LuaFeed Inst1 F3=576, F2=473, F1=680
LuaFeed Inst2 F3=183, F2=797, F1=452
Sample 5  Status OK
LuaFeed Inst1 F3=597, F2=660, F1=212
LuaFeed Inst2 F3=584, F2=145, F1=830
<Fri Nov 15 17:05:01>  INFO: Computing subscription differences for feed LuaFeed (mdmTest)
<Fri Nov 15 17:05:01>  INFO: Existing subscription data:
CODE.INST.1 [Ask, Bid, Trade]
CODE.INST.2 [Ask, Bid, Trade]
<Fri Nov 15 17:05:01>  INFO: Unaltered subscriptions:
  LuaFeed: CODE.INST.1 (Inst1) [Ask, Bid, Trade]
<Fri Nov 15 17:05:01>  INFO: New subscriptions: (none)
<Fri Nov 15 17:05:01>  INFO: FeedController:AdapterFeed Unsubscribing from instrument 'CODE.INST.2' for feed 'LuaFeed' (sub=0xb823870).
<Fri Nov 15 17:05:01>  INFO: LuaFeed Removed subscription for instrument 'CODE.INST.2'
Sample 6  Status OK
LuaFeed Inst1 F3=50, F2=25, F1=837
Sample 7  Status OK
LuaFeed Inst1 F3=827, F2=830, F1=627
Sample 8  Status OK
LuaFeed Inst1 F3=922, F2=181, F1=256
<Fri Nov 15 17:05:04>  INFO: Computing subscription differences for feed LuaFeed (mdmTest)
<Fri Nov 15 17:05:04>  INFO: Existing subscription data:
CODE.INST.1 [Ask, Bid, Trade]
<Fri Nov 15 17:05:04>  INFO: Unaltered subscriptions:
  LuaFeed: CODE.INST.1 (Inst1) [Ask, Bid, Trade]
<Fri Nov 15 17:05:04>  INFO: New subscriptions:
  LuaFeed: NEW.CODE.INST.2 (Inst2) [F1, F3]
<Fri Nov 15 17:05:04>  INFO: FeedController:AdapterFeed Subscribing to instrument 'NEW.CODE.INST.2' for feed 'LuaFeed' (sub=0xb823940). 2 fields; [F1, F3]
<Fri Nov 15 17:05:04>  INFO: LuaFeed Added subscription for instrument 'NEW.CODE.INST.2'
Sample 9  Status OK
LuaFeed Inst1 F3=355, F2=317, F1=79
LuaFeed Inst2 F3=922, F1=26
Sample 10 Status OK
LuaFeed Inst1 F3=554, F2=677, F1=745
LuaFeed Inst2 F3=118, F1=134
<Fri Nov 15 17:05:06>  INFO: mdmTest Script doSample returned false, exiting...
<Fri Nov 15 17:05:06>  INFO: FeedController:AdapterFeed Calling destroy_feed() for feed 'LuaFeed', feed_id=0.
<Fri Nov 15 17:05:06>  INFO: LuaFeed Terminate event received, ending feed script
<Fri Nov 15 17:05:07>  INFO: LuaFeed Finished executing script 'mdmFeedAdapterApiKit/scripts/inputFeed.lua' for feed 'LuaFeed'.
<Fri Nov 15 17:05:07>  INFO: FeedController:AdapterLibrary Unloading feed library 'flm/geneos-feed-lua.so'.

This command executes a Lua feed adapter input script mdmFeedAdapterApiKit/scripts/inputFeed.lua. By running the Netprobe with the -mdmtest command-line option the script can be run stand-alone, without requiring a Geneos gateway process and associated configuration. The argument of the -mdmtest option actually specifies an MDM sampler script mdmTest_geneos-feed-lua.so.lua, the content of which is described more fully in the Market Data Monitor (MDM) plug-in documentation.

Note: the two Lua scripts serve different purposes and are not the same:

• inputFeed.lua is an feed adapter written in Lua, using the API discussed in this section.
• mdmTest_geneos-feed-lua.so.lua is a sampler script used to exercise the feed adapter.

Introduction

Lua feed adapter scripts are executed by the standard Lua feed adapter provided by ITRS. Multiple feed adapter scripts can be executed simultaneously, each one providing access to a different market data feed. Each script is executed in a separate thread so that the feeds can publish data to the Netprobe while other monitoring tasks are performed. No data is shared between different Lua scripts unless they explicitly communicate with each other in some way (e.g. using files or via a TCP/IP connection).

Lua feed adapter input scripts should be written using version 5.1 of the Lua language. Some Netprobe platforms support Just-In-Time compilation of Lua code via the LuaJIT implementation of Lua, which provides improved execution speed of the Lua script. These platforms also allow use of the Lua FFI library, which allows calling of C functions from Lua.

Getting a script running

The Lua script is expected to run for the lifetime of the feed, maintaining a connection to the market data system (or other source). Incoming instrument data from the source system should be translated to a format which can be published to the Netprobe. If the script exits unexpectedly, the feed status changes to an error state which can be detected in the MDM sampler script.

For example, when running the Quick Start with an empty inputFeed.lua script the following output is displayed.

$ ./netprobe.linux -mdmtest mdmFeedAdapterApiKit/scripts/mdmTest_geneos-feed-lua.so.lua
... <some output omitted>
Sample 8        Status ERROR: Script exited
Sample 9        Status ERROR: Script exited
Sample 10       Status ERROR: Script exited
Lua test script sampler.doSample returned false, exiting...
<Fri Sep 20 08:36:32>  INFO: FeedController:AdapterFeed Calling destroy_feed() for feed 'LuaFeed', feed_id=0.
<Fri Sep 20 08:36:32>  INFO: FeedController:AdapterLibrary Unloading feed library 'flm/geneos-feed-lua.so'.

If we ensure that inputFeed.lua never exits by adding an infinite loop, then we get the following:

-- infinite loop, sleeps for 1 second each iteration
while true do
    feed.sleep(1.0)
end

$ ./netprobe.linux -mdmtest mdmFeedAdapterApiKit/scripts/mdmTest_geneos-feed-lua.so.lua
... <some output omitted>
Sample 8        Status OK
Sample 9        Status OK
Sample 10       Status OK
Lua test script sampler.doSample returned false, exiting...
<Thu Sep 19 16:14:39>  INFO: FeedController:AdapterFeed Calling destroy_feed() for feed 'LuaFeed', feed_id=0.
... <test never terminates>

The test does not terminate because the feed adapter script is still running due to the infinite loop, even though the sampler script has now finished. To end the test you will need to Ctrl+C in your terminal window or kill the Netprobe process.

Responding to events

The "failure to terminate" problem we just encountered occurs because the Lua script is executing in a thread. This means the script can run asynchronously from the Netprobe, allowing tick data to be published even while the Netprobe is busy with other monitoring tasks. However since it is the main Netprobe thread which manages the feeds, requests to the feed adapter (such as asking it to terminate) are sent via an event queue.

Checking for events is done using the getEvent() function, which is provided as part of the Lua "feed" module. This function checks the event queue and returns the first event on the queue, or nil if there are no events. getEvent() returns several types of event, but to fix our immediate problem we are only interested in the terminate event.

Updating our code to respond to the terminate event now gives us the following:

local run = true
while run do
    local event = feed.getEvent()
    if event and event.type == "terminate" then
        -- respond to terminate event by ending the loop and exiting the script
        print("terminating feed script")
        run = false
    end
    feed.sleep(1.0)
end

$ ./netprobe.linux -mdmtest mdmFeedAdapterApiKit/scripts/mdmTest_geneos-feed-lua.so.lua
... <some output omitted>
Sample 9        Status OK
Sample 10       Status OK
Lua test script sampler.doSample returned false, exiting...
<Fri Sep 20 08:44:11>  INFO: FeedController:AdapterFeed Calling destroy_feed() for feed 'LuaFeed', feed_id=0.
terminating feed script
<Fri Sep 20 08:44:12>  INFO: LuaFeed Finished executing script 'scripts/luaFeed_Input.lua' for feed 'LuaFeed'.
<Fri Sep 20 08:44:12>  INFO: FeedController:AdapterLibrary Unloading feed library 'flm/geneos-feed-lua.so'.

Other events that we may receive are the subscribe and unsubscribe events. These events indicate requests from the Netprobe that we subscribe for (or unsubscribe from) a particular instrument. These instruments will be configured by the MDM sampler script. To identify which event we have received, check the type field of the event object.

local run = true
while run do
    local event = feed.getEvent()
    if event then
        if event.type == "terminate" then
            run = false
        elseif event.type == "subscribe" then
            -- TODO subscribe to the instrument on the market data system
        elseif event.type == "unsubscribe" then
            -- TODO unsubscribe from the instrument on the market data system
        end
    end
    feed.sleep(1.0)
end

We can tidy this code up somewhat by writing a separate function to handle each type of event. If we store these functions in a map using the event type as a key, we can then use the type name to lookup the appropriate handler function for the event and call it directly.

local run = true

local  handler_map  =  {
    terminate   = function(event) run = false end,
    subscribe   = function(event) print("TODO subscribe") end,
    unsubscribe = function(event) print("TODO unsubscribe") end
}

while run do
    local event = feed.getEvent()
    if event then
        local  handler_func  =  handler_map[event.type]
        if handler_func then
            -- call the handler, passing the event as the argument
            handler_func(event)
        end
    end
    feed.sleep(1.0)
end

Finally, one remaining problem we have is that the script will currently sleep for 1 second between processing events from the Netprobe. This is fine for terminate events since you will only receive one such event near the end of the script lifetime, but you may receive multiple subscribe and unsubscribe events which should be processed promptly. We fix this last issue by checking for more events after processing.

The code below now forms a simple event loop for responding to Netprobe events. You will see parts of this code in the inputFeed.lua example file. When writing your own feed adapter script implementation, the feed.sleep(1.0) call should be replaced with a check (or wait) for data from the market data system. If waiting for data, this should ideally be a timed wait so that the script still checks for Netprobe events even when data is not being received.

local run = true

local handler_map = {
    terminate   = function(event) run = false end,
    subscribe   = function(event) print("TODO subscribe") end,
    unsubscribe = function(event) print("TODO unsubscribe") end
}

local function checkEvents()
    local event = feed.getEvent()
    while  event  do
        local handler_func = handler_map[event.type]
        if handler_func then
            handler_func(event)
        end
        event  =  feed.getEvent()
    end
end

while run do
    checkEvents()
    feed.sleep(1.0) -- TODO replace with check for data/events from the market data system
end

Subscribing and publishing

Instruments to subscribe to are sent to the feed script from Netprobe as a subscribe event. The event contains the stock-code or RIC of the instrument, and the names of the fields for this instrument which should be published. An example subscribe event is shown below. To remove an instrument, Netprobe sents an unsubscribe event.

local subscribeEvent = {
    type = "subscribe",
    instrument = "MY.CODE.N",
    fields = { ASK="", BID="", COMMENT="" }
}

This table structure is similar to that of a tick which is used for publishing:

local tick = {
    time = 1379409456.763708,
    instrument = "MY.CODE.N",
    fields = { ASK=128.75, BID=128.50, COMMENT="A string value" }
}
feed.publish(tick)

Because the two types are so similar, you can create a tick from a subscribe event by simply removing the type field and adding a time field (which indicates when the tick was received). The time field should then be updated every time the tick field values are updated, before calling feed.publish(). An example of this can be found in the inputFeed.lua script.

Feed Parameters

Parameters to the Lua feed script are passed in the params table. Both keys and values in this table are strings. A number of default parameters will be passed, including:

• feed.name - The name of the feed.
• feed.library.filename - The filename of the ITRS standard Lua feed adapter shared-object.
• lua.script - The Lua script being executed.

Other user-defined configuration parameters (as defined in the MDM sampler script when the feed is added) will also appear in this table. When using the test script to exercise the feed, you can add additional parameters by adding new fields to the lua table (see below) in the call to addFeed. Nested tables in this structure will be flattened to a dotted-name parameter key.

local lf = md.addFeed(
    "LuaFeed",
    {
        feed = {
            type = "lua",
            verbose = "true",
            library = { filename = "geneos-feed-lua.so" }
        },
        lua  =  {
            script = "inputFeed.lua",
            value = { min = 1, max = 1000 }
        },
        instruments = { Inst1 = "CODE.INST.1", Inst2 = "CODE.INST.2" },
        fields = { F1 = "Trade", F2 = "Ask", F3 = "Bid" }
    }
)

This configuration results in the following parameters being passed to the inputFeed.lua script:

feed.params = {
    ["feed.name"]             = "LuaFeed",
    ["feed.type"]             = "lua",
    ["feed.verbose"]          = "true",
    ["feed.library.filename"] = "geneos-feed-lua.so",

    ["lua.script"]    = "inputFeed.lua",
    ["lua.value.min"] = "1",
    ["lua.value.max"] = "1000"
}

Configuration using a Geneos Gateway

To use your feed in a live MDM plug-in (as opposed to the Netprobe stand-alone mode) you should:

1. Deploy your feed script to a location accessible by the Netprobe.
2. Configure an MDM plug-in on the Gateway as described in the plug-in documentation.
3. Adjust the MDM sampler script (if necessary) to reference your feed script.

Aside from the script location settings, from the point-of-view of the MDM sampler script the Lua feed adapter is as a regular feed.