MDM Sampler and Feed Adapter Technical Reference

Functions

geneos.helpers.gettimeofday()

Gets the current time.

This function is similar to os.time, but returns the time to microsecond accuracy using the C function 'gettimeofday'.

The output is a single number representing the time as seconds since the UNIX Epoch, with its fractional component indicating the microseconds. This number may be split using the functions in the math module.

Returns:

The current time at the point the call was made.

geneos.helpers.formatTime([timefmt, ]time)

Formats a time-of-day with a fractional seconds component.

This function splits the time using the math.modf function, and formats this as a local time value (not UTC) using os.date.

Note: This function does not perform any type checking on its arguments for performance reasons.

Parameters:

• timefmt: (optional string)
• time: (number)

Returns:

(string) Formatted time in "HH:MM:SS.sss" format (or as specified by timefmt) The fractional part will be rounded to the specified number of decimal places.

geneos.helpers.parseTimeInterval(interval)

Parses a time interval from a string in ISO8601 syntax.

This function interprets time intervals in the format [PT][nH][nM][nS], for example PT27H15M37,5S (27 hours 15 minutes 37.5 seconds), 1.5H, 90M or 5400S (three different ways of saying 90 minutes). The input string must contain at least one of 'H', 'M' or 'S'. Intervals expressed as a number of years, months or days are not supported.

Parameters:

• interval: (string)

Returns:

• (optional number) The number of seconds in the interval, nil on failure
• (optional string) An error message on failure

geneos.helpers.zip(keys, values)

Combine two sequences (arrays) into a single key-value map.

Parameters:

• keys: (table)
• values: (table)

Returns:

(table) A map whose keys are the values from the first sequence, and whose values are the corresponding values from the second sequence.

Usage:

local gh = require "geneos.helpers"
local keys = { "A", "B", "C" }
local vals = { 10, 20, 30 }
local map  = gh.zip(keys, vals)
-- map == { A=10, B=20, C=30 }

geneos.helpers.createtable(narr, nrec)

Creates a pre-sized table.

This is a direct wrapper for the lua_createtable function in the Lua C API.

Parameters:

• narr: (number)
• nrec: (number)

Returns:

A new empty table pre-allocated with space for narr array elements and nrec non-array elements.

geneos.helpers.examine(obj)

Outputs an object to io.output (which is usually stdout).

The purpose of this function is to display the content of an object for debugging. This function will only cope with simple types; functions, threads or userdata types are not described in any detail. Cycles of tables are handled, by omitting the contents of the table when repetitions are encountered.

For a more complete debugging helper which handles more conditions, please see https://github.com/kikito/inspect.lua

Parameters:

obj:

Functions

geneos.latency.files.newTickHistoryFile([filename[, timefmt[, fieldNames]]])

Constructs a new tick history file logger.

A TickHistory logger logs full details of each tick received, in a format that can be read by the TestFile adapter.

This type of logger cn be used independently of the latency algorithm. However, if the set of fields subscribed to will change after a feed is started, the fieldNames parameter should be used and should be set to cover all the field names that will need to be logged. This is because, for performance reasons, the sequence of field names used for each feed is fixed after the first tick.

Parameters:

• filename: (optional string)
• timefmt: (optional string)
• fieldNames: (optional table)

Returns:

(TickHistory) The newly constructed tick history logger.

geneos.latency.files.newMatchedTicksFile(configNames[, filename[, microsec]])

Constructs a new matched tick file writer.

A MatchedTicks logger writes a file in CSV (comma separated values) format, showing the matches computed by the tick matching algorithm.

To correctly write the CSV file, this class needs information about the names of the feeds and of the fields used for matching. Since this information is supplied when configuring a latency context, the Context class provides a convenience method that can be used when calling this method (see the usage example below.)

Parameters:

• configNames: (table)
• filename: (optional string)
• microsec: (optional boolean)

Returns:

(MatchedTicks) The newly constructed matched ticks logger.

Usage:

local lt = require "geneos.latency"
local lf = require "geneos.latency.files"
local ctx = lt.newContext{}
--[[ calls to ctx:setBase() and ctx:addfeed() go here ]]
local configNames = ctx:getConfiguredNames()
local matchedTicksFile = lf.newMatchedTicksFile(configNames, 'matches.csv')
ctx:addWriter(matchedTicksFile:getWriter()):start(

geneos.latency.files.newTickLogger(properties)

Used only when creating a custom logger.

This method creates a prototype for a custom logger. To construct a ready-to-use tick logger, use newTickHistoryFile() or newMatchedTicksFile().

The object returned by this method is intended to act as a prototype for a customised logger class. Instances of such a class are created by invoking the new() method of the prototype. These instances inherit the methods of TickLogger and any methods replaced in or added to the prototype.

An instance of a TickLogger subclass should open a file as soon as it is created (or after it has checked that all required fields are present). However, creating the prototype in order to customise it should not cause a file to be opened. To resolve this conflict of requirements, the TickLogger constructor calls the init() method. TickLogger:init() does nothing, but subclasses should override it to call TickLogger:open(), after performing any required checks. To use the logger with a latency context, a method to create a writer function is also needed.

Parameters:

• properties: (table)

Returns:

(TickLogger) The newly constructed prototype logger.

Class TickLogger

classgeneos.latency.files.TickLogger

Base class for tick loggers.

This class provides methods for opening a file with a name that includes a formatted date, and for rolling over logging to a new file at the end of a specified interval.

This class is not normally used directly; it is subclassed by TickHistory and MatchedTicks.

TickLogger.filename

Template to use for the name of the logging file.

Can include format codes as understood by geneos.helpers.formatTime(), which will be interpolated into the file name when the file is opened (or when it is rolled over.) Defaults to history.%Y%m%d.

TickLogger.timefmt

Template for tick timestamps.

Defaults to %H:%M:%S.%3f, that is, millisecond precision. If overridden, this should be set to a format acceptable to geneos.helpers.formatTime().

TickLogger.file

The TickLogger:open() method sets this to the file handle returned by io.open.

TickLogger:new(properties)

Constructor method inherited by derived classes.

Derived classes should not override this method.

Parameters:

• properties: (table)

Returns:

• (TickLogger) The newly constructed prototype logger.

TickLogger:init()

Placeholder method, called when a tick logger is constructed.

This base class implementation does nothing. Derived classes should override it to check or set fields representing configurable propertties and then call TickLogger:open().

TickLogger:open()

Opens a file, expanding any time format codes in the name.

The file is opened for append, so that if the sampler is restarted, an existing file will not be lost.

TickLogger:getActualFilename()

Returns the name of the current log file.

If the file is in the course of being rolled over, returns the name of the last file used. Also returns any error message encountered if the file could not be opened. This function may be used to populate a headline in an 'admin' dataview showing the name of the file.

Returns:

• (optional string) The file name, including the expansion of any time format codes; nil if the file was not opened.
• (optional string) The error message returned if the file could not be opened; nil if the file was opened successfully.

TickLogger:setFileRoll([rollInterval[, firstRollTime[, rollCallback]]])

Enables timed rollover on the logging file.

After this method is called, each time the specified roll interval expires, the file will be closed and reopened. If filename contains time format codes understood by geneos.helpers.formatTime(), these will be reinterpreted, so that a new file can be opened. Optionally, a callback function will be invoked each time the file is rolled over; this function could, for example, run a script to archive the file that has just been closed.

Parameters:

• rollInterval: (number)
• firstRollTime: (number)
• rollCallback: (function)

Returns:

• (TickLogger) The tick logger (i.e. self)

Usage:

local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"

-- Tick history with hourly rollover, starting 5 minutes from now
local oneHour = gh.parseTimeInterval("1H")
local fiveMinutesFromNow = gh.gettimeofday() + gh.parseTimeInterval("5M")
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d%H%M"):setFileRoll(oneHour, fiveMinutesFromNow)

-- Matched ticks CSV with daily rollover, starting at midnight
local matchedTickCsv = lf.newMatchedTicksFile("matches.%y%m%d.csv"):setFileRoll()

TickLogger:checkFileRoll(sampleTime)

Tests whether file should be rolled over.

Derived classes should call this method before logging each sample, passing the sample time. If a roll time has been set and has elapsed, this method will do the following, in this order:

• Calculate the next roll time by adding the roll interval to the roll time that has just expired.
• Close the existing logging file by calling TickLogger:close().
• Invoke the roll callback function, if any.
• Open the new logging file by calling TickLogger:open().

Parameters:

• sampleTime: (number)

Returns:

• (boolean) True if the file was rolled over.

TickLogger:sampleEnd()

Flushes the logging file.

TickLogger:close()

Closes the logging file.

Class TickHistory

classgeneos.latency.files.TickHistory

This class Logs full details of each tick received, in a format that can be read by the TestFile adapter.

Ticks from all subscriptions are logged in order of their timestamps, with information showing how they were matched. Note that this file can become very large quite quickly; the TickLogger:setFileRoll() method can be used to mitigate this.

See also these methods inherited from from TickLogger: TickLogger:setFileRoll(), TickLogger:getActualFilename()

Instances of this class are constructed by the geneos.latency.files.newTickHistoryFile() function.

TickHistory:getWriter()

Provides a tick history writer function.

This method returns a function which can be passed to geneos.latency.Context:addWriter().

The writer function calls TickLogger:checkFileRoll at the start of each sample.

Returns:

• (geneos.latency.historyWriter()) A history writer function

Usage:

local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"

-- Create a tick and enable daily rollover
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d"):setFileRoll()

--[[ code to create ctx as a latency context omitted here ]]
ctx:addWriter(tickHist:getWriter())

Class MatchedTicks

classgeneos.latency.files.MatchedTicks

This class writes a file in CSV (comma separated values) format, showing the matches computed by the tick matching algorithm.

Each line of the file displays a matched tick from the baseline feed against matches found from the other configured feeds. Baseline ticks with no matches are not written to the file.

This class overrides overrides TickLogger.timefmt to %H:%M:%S.%6f if microsecond precision is specified when an instance is created.

See also these methods inherited from from TickLogger: TickLogger:setFileRoll(), TickLogger:getActualFilename()

Instances of this class are constructed by the geneos.latency.files.newMatchedTicksFile() function.

MatchedTicks:getWriter()

Provides a tick history writer function.

This method returns a function which can be passed to geneos.latency.Context:addWriter().

The writer function calls TickLogger:checkFileRoll at the start of each sample.

Returns:

• (geneos.latency.historyWriter()) A history writer function

Functions

geneos.latency.newContext(options)

Constructs a latency context.

Optional settings for the context can be specified by setting any of the following fields in the options parameter.

tolerances, timeDiffMax, timeDiffMin: used when constructing a tick comparison function. See getTickComparator() below. If no value is provided for these fields in options, a default of two seconds is used for timeDiffMax and the other fields are left as nil.

Parameters:

• options: (table)

Returns:

• (Context) The new latency context

geneos.latency.getTickComparator(tolerances[, timeDiffMax[, timeDiffMin]])

Constructs a tick comparison function.

This factory function is used by Context:setBase() and Context:addFeed() to construct tick preprocessing and matching functions if the corresponding parameters are omitted in the function call.

It can be called directly to create a customised matching function.

Parameters:

• tolerances: (table)
• timeDiffMax: (optional number)
• timeDiffMin: (optional number)

Returns:

• a function that meets the requirements both of a tick matching function LcsComparator() and of a tick pre-processing function TickPreprocessor()

Usage:

local lat = require "geneos.latency"
-- Create a matcher with following customisations:
--    Allow Mid field to match with tolerance of .005
--    Require exact match (and allow string matching) for Venue field
--    Do not accept matches with latency > 1.5 sec or less than zero
local tolerances = { Mid = 0.005, Venue = 0 }
local match_fn = lat.getTickComparator(tolerances, 1.5, 0.0)

geneos.latency.historyWriter(event, data)

Specification of a tick history writer function.

You can define one or more functions that match this specification and add them to a latency context.

If any history writer is present, all ticks received in a sample by the latency context will be:

• Sorted into sequence in order of arrival time.
• A field named seq will be added to each tick giving its position in the sequence of all ticks received since the context was constructed.

At the end of each sample, each history writer will be called for the ticks that arrived in the previous sample. History writing is deferred in this way because a the remark field of a tick may be updated if a match is found for that tick against data in the next sample.

Parameters:

• event: (string)
• data: (optional number, Tick)

Usage:

local gh  = require "geneos.helpers"
local lat = require "geneos.latency"
-- Define a function to output the time of each sample and the arrival time
-- and sequence number of each tick
local function printLogger(event, data)
    if event == 'sampleTime' then
        io.write(gh.formatTime("== Sample, %H:%M:%S ==", data))
    elseif event == 'tick' then
        io.write(string.format("%s %s %d %s %s", data.feed, data.inst, data.seq,
            gh.formatTime("%H:%M:%S.%3f", data.timeFirst), data.remark or '-'))
    end
end
-- Register the function  (ctx is a latency context created elsewhere)
ctx:addWriter(printLogger)

geneos.latency.TickPreprocessor(prev, current)

Specification for tick pre-processing functions used by the Context class.

The Context class uses a function of this type to detect and remove duplicate ticks from each feed, including the baseline feed. An alternative tick pre-processing function can be passed to Context:setBase() and Context:addFeed() to inspect, amend or discard ticks before they are passed to the LCS algorithm.

Each time the function is called, it is passed two tick values: one is the previous accepted tick in the subscription (that is, the stream of ticks for a given instrument from a given feed), the other is the current tick. The 'previous' tick will have nil values for feed and instrument if no ticks have yet been accepted in the subscription.

Parameters:

• prev: (geneos.marketdata.Tick)
• current: (geneos.marketdata.Tick)

Returns:

• (boolean) nil or false if the current tick is to be accepted, true (or any non-nil value other than false) if it is to be discarded.

geneos.latency.LcsComparator(base, feed)

Specification for comparison functions used by the LCS algorithm.

A comparison function (comparator) is used by the LCS algorithm (as implemented by the Lcs class) to compare values from different sequences against each other for equality.

When used in the Latency API, the function will be called to compare each tick from the baseline feed with each tick from a comparison feed, to identify possible matches. An alternative comparison function can be passed to Context:addFeed() to change either the criteria for deciding whether ticks match or the weighting assigned to a match.

Parameters:

• base:
• feed:

Returns:

• (optional number) nil or false for no match, or if matched a weighting value. The weighting is used by the LCS algorithm to select between multiple match candidates, with larger values indicating a lower ranking meaning the match is less likely to be chosen.
• An optional value to associate with the match. If the match is selected by the LCS algorithm, this value will be passed to the specified LcsMatchListener() (see Lcs:compute()).

geneos.latency.LcsMatchListener(value, index, base, feed)

Specification for callback functions used by the LCS algorithm.

A match listener function is used receive results from the LCS algorithm (as implemented by the Lcs class).

The listener will be called-back repeatedly during execution of the algorithm (during Lcs:compute()). Each call will identify a single pair of ticks matched between the "base" and "feed" sequences.

Parameters:

• value:
• index:
• base:
• feed:

Class Context

classgeneos.latency.Context

The geneos.latency.Context class performs relative feed latency calculations.

Instances of this class are constructed by the geneos.latency.newContext() function.

Context:setBase(name, baseConfig[, preprocCmp])

Specifies name and configuration of the baseline feed.

This method should be called before Context:addFeed()

Parameters:

• name: (string)
• baseConfig: (geneos.marketdata.FeedConfig)
• preprocCmp: (optional TickPreprocessor())

Returns:

• (Context) The context object (i.e. self)

Context:addFeed(name, feedConfig[, preprocCmp[, matchCmp[, metrics]]])

Specifies name and configuration of a monitored (not baseline) feed.

Note: The configuration is pruned so that those instruments and fields that are not present in the configuration of the baseline feed are removed from the configuration of the monitored feed

Parameters:

• name: (string)
• feedConfig: (geneos.marketdata.FeedConfig)
• preprocCmp: (optional TickPreprocessor())
• matchCmp: (optional LcsComparator())
• metrics: (optional Metrics)

Returns:

• (Context) The context object (i.e. self)

Context:addWriter(writer)

Adds a tick history writer.

This method allows a function to be supplied which will be called once per sample for all the input ticks in the previous sample. See also Context:writeHistory() If used, this method should be called before calling Context:start()

Parameters:

• writer: (historyWriter())

Returns:

• (Context) The context object (i.e. self)

Context:getConfiguredNames()

Provides a table that can be used to configure a tick history writer.

This function is useful when configuring a matchedTicks history writer.

See geneos.latency.files.newMatchedTicksFile() for a usage example.

Returns:

• (table) A table with the following fields: fieldNames: the field names configured for the base feed, as a sorted sequence; baseName: the name of the base feed; feedNames: the names of the other feeds, as a sequence.

Context:start()

Establishes subscriptions for the feeds that have been specified by calling setBase and addFeed.

This method should not be called until after calling Context:setBase() and Context:addFeed() (and Context:addWriter(), if required)

Returns:

• (Context) The context object (i.e. self)

Context:sample([sampleTime])

Process one sample.

Collects data from all subscriptions, runs LCS algorithm and invokes history writer(s) to log the previous sample

Parameters:

• sampleTime: (optional number)

Context:getMetrics(feedId[, inst])

Get metrics for given feed or subscription.

Parameters:

• feedId:
• inst: (optional string)

• (optional Metrics) The requested metrics object, or nil if feedId or inst is not valid.

Context:getLastTick(feedId, inst)

Get last non-duplicate tick received on given subscription.

Parameters:

feedId:

inst: (string)

Returns:

• (optional geneos.marketdata.Tick) The requested tick, if available. if the subscription does not exist, the return value is nil. If no ticks have been received, the return value is a tick with times set to zero and and empty table of field values.

Context:getFeedStatus(feedId)

Get status of given feed.

Parameters:

• feedId:

Returns:

• (optional string) The status of the specified feed, or nil if feedId is not valid.

Context:writeHistory()

Invokes history writer(s) to log tick history for previous sample.

Note: history writing is delayed by one sample so that matches can be annotated across a sample boundary.

This method is called by Context:sample(); it can also be called directly if required, to log the last sample if the sampler is being stopped. This can be achieved by defining geneos.sampler.finalise(), as in the example below.

Usage:

local gs = require "geneos.sampler"
--[[ ctx is a latency context with a history writer ]]
gs.finalise = function()
    ctx:writeHistory()
end

Class Lcs

classgeneos.latency.Lcs

An implementation of the Longest Common Subsequence (LCS) algorithm.

This class can be used to run the LCS algorithm on any pair of sequences of comparable items.

In the Latency API, it is used to compare a series of ticks from the baseline feed against a series of ticks from the comparison (or monitored) feed. The default implementation uses a comparison function which matches applying constraints to time and field values.

Output from the algorithm is a series consisting of pairs of matched ticks (a baseline and feed tick) which are then used to compute latency statistics.

To use this class, callers should:

1. Create a new instance with Lcs:new(),
2. Fill the matrix using Lcs:populate() or Lcs:populateFromArray() passing an LcsComparator() function.
3. Run the LCS algorithm with Lcs:compute() and process the matches via an LcsMatchListener() function.

Usage:

local md  = require "geneos.marketdata"
local lat = require "geneos.latency"
local comparator -- A comparator function, e.g. lat.getTickComparator

-- Get ticks for processing
local ticks1 = feed_1:getTicks("myInstrument")
local ticks2 = feed_2:getTicks("myInstrument")

-- Run LCS algorithm and place results into metrics object
local metrics = lat.Metrics:new()
metrics:onSampleBegin(md.countTicks(ticks2))
lat.Lcs:new():populate(ticks1, ticks2, comparator):compute(metrics)
metrics:onSampleEnd()

Lcs:new([proto])

Constructs an Lcs matrix.

The functionality of the new matrix may be altered by providing a table argument proto, populated with new (or overriding) function and field definitions.

Parameters:

Returns:

• (Lcs) A new LCS matrix.

Lcs:populate(baseTicks, feedTicks, cmp[, baseLength[, feedLength]])

Populates the LCS matrix from a list of ticks.

This method populates the LCS matrix with a linked-list series of ticks, such as those returned from a call to geneos.marketdata.Feed:getTicks().

Parameters:

• baseTicks: (geneos.marketdata.Tick)
• feedTicks: (geneos.marketdata.Tick)
• cmp: (LcsComparator())
• baseLength: (optional number)
• feedLength: (optional number)

Returns:

• (Lcs) This LCS matrix object (i.e. self).

Lcs:populateFromArray(baseSeq, feedSeq, cmp[, baseLength[, feedLength]])

Populates the LCS matrix from an array of values.

This method populates the LCS matrix with values from two source arrays.

Parameters:

• baseSeq: (table)
• feedSeq: (table)
• cmp: (LcsComparator())
• baseLength: (optional number)
• feedLength: (optional number)

Returns:

• (Lcs) This LCS matrix object (i.e. self).

Lcs:compute(listener)

Computes the LCS for a populated matrix.

This method should not be called until the LCS matrix has been populated. When called it will compute the longest common subsequence from the input sequences, and call the specified listener function for each match in the output sequence.

The output sequence is selected by minimising the total weight of the sequence using the weighting values returned by the LcsComparator() function, while maximising the number of matches that make up the sequence.

The listener function should follow the LcsMatchListener() function specification. This function will be called once for each match found, and matches will be returned in reverse order (matches found later in the input sequences will be returned first).

Alternatively, the listener may be a table with an onMatch field that is a function following the LcsMatchListener() function specification.

Parameters:

• listener: (LcsMatchListener(), table)

Returns:

• lastMatchedBase Index of base item selected for last match (0 if there were no matches)
• lastMatchedFeed Index of feed item selected for last match (0 if there were no matches)

Class Metrics

classgeneos.latency.Metrics

The Metrics class stores and computes latency statistics.

Instances of the Metrics class are constructed by geneos.latency.Context for each feed, and for each instrument subscription. Each sample, metrics data is gathered at the instrument subscription level as the LCS algorithm is run. Data for instruments of a feed are then rolled up to the feed level at the end of processing.

A Metrics class instance is a Lua table containing the methods described below, and the following fields. Note that because latencies are calculated for monitored (not baseline) feeds, metrics objects for the baseline feed have only the numTicks statistics field.

• feed The name of the feed for which this object collects metrics.
• inst The name of the instrument, or nil for feed-level metrics.
• numTicks The number of ticks in the current sample.
• matches The number of matched ticks found in the current sample.
• latMax The maximum latency value for the current sample.
• latMin The minimum latency value for the current sample.
• latSum The sum of latency values for the current sample.
• latSumSq The sum of of the squares of the latency values for the current sample.

You can use the Metrics:new() method to create a specialisation of this class and pass it to geneos.latency.Context:addFeed() to be used as a prototype for metrics objects of subscriptions of that feed.

Usage:

-- Statistics calculation follows the template method (design) pattern.
-- The algorithm runs according to the following pseudocode:
for f in feeds() do
    f.metrics:onSampleBegin(0)
    for i in f.instruments() do
        i.metrics:onSampleBegin(tickCount)
        -- series of i.metrics:onMatch calls as LCS algorithm is executed
        i.metrics:onSampleEnd()
        f.metrics:aggregate(i.metrics) -- roll up stats
    end
f.metrics:onSampleEnd()
end

Metrics:new([proto])

Constructor.

Use this method to create a new instance, or derive a new class from the Metrics object. A derived class instance may be passed as an argument to geneos.latency.Context:addFeed(), where it will be used a prototype for creating new metrics objects.

Parameters:

• proto: (optional table)

Returns:

• (Metrics) An object which can be passed to Context:addFeed() to be used as the metrics object for the feed. The object will also and as a prototype to create metrics objects for instrument subscriptions.

Usage:

local lat = require "geneos.latency"

-- Derive a metrics object with an onMatch function that outputs match information
local fmt = "Saw match on %s.%s at %f"
local displayMatchMetrics = lat.Metrics:new {
    onMatch = function(self, latency, nm, bt, ft)
        lat.Metrics.onMatch(self, latency, nm, bt, ft) -- must call inherited method
        io.write(fmt:format(ft.feed, ft.inst, ft.timeFirst))
    end
}

-- Use the specialised metrics object.
-- ctx is an instance of geneos.latency.Context created and initialised elsewhere.
ctx:addFeed(feedName, feedConfig, nil, nil, displayMatchMetrics)

Metrics:onSampleBegin(numTicks)

Called at start of a sample to reset the statistics.

This method will:

• Save the number of ticks in the numTicks field
• Set the matches, latSum and latSumSq fields to 0.
• Set the latMin and latMax fields to nil.

You should override this method if you add any fields to the class that need to be initialised each sample. If you override this method, you must ensure the inherited method is called from within your overriding method.

Parameters:

• numTicks: (number)

Returns:

• (Metrics) This metrics object (i.e. self).

Metrics:onMatch(latency, nm, bt, ft)

Called to update the statistics for each tick match detected by the LCS algorithm.

This method updates the contained statistics to include the new match. In addition it will add/update the remark field of the base- and feed-tick to register the matches (see geneos.marketdata.Tick.remark).

You can override this method to do additional processing on each match. Note that the LCS matching algorithm reports matches in reverse chronological order (i.e. matches for more recent ticks are reported first). If you override this method, you must ensure the inherited method is called from within your overriding method.

Note: this method does not validate its input arguments for performance reasons.

Parameters:

• latency: (number)
• nm: (number)
• bt: (geneos.marketdata.Tick)
• ft: (geneos.marketdata.Tick)

Metrics:aggregate(other)

Called to update feed metrics with a set of computed subscription metrics.

This method updates the statistics fields of the object, namely:

• numTicks
• matches
• latMax
• latMin
• latSum
• latSumSq

Note: latMax and latMin values will be nil unless the metrics object has a non-zero match count.

Override this method if you add any fields that need to be aggregated at the feed level. If you override this method, you must ensure the inherited method is called from within your overriding method.

Parameters:

• other: (Metrics)

Returns:

• (Metrics) This metrics object (i.e. self).

Metrics:onSampleEnd()

Called at the end of processing for each sample.

This method does nothing; it is provided as a hook to be overridden if required.

This method is called for subscription-level metrics before aggregation into the feed. It is called for feed-level metrics after all subscriptions have been aggregated.

Returns:

• (Metrics) This metrics object (i.e. self).

Metrics:getAverage()

Calculates the mean of match latency for the sample.

Returns:

1. (optional number) Total latency divided by number of matches, or nil if there were no matches.

Metrics:getStdDev()

Calculates standard deviation of match latency for the sample.

Returns:

• (optional number) Root mean squared deviation of observed latency values from the mean, or nil if there were no matches.

Functions

geneos.marketdata.addFeed(name, feedConfig)

Configures a connection to a data feed.

Parameters:

• name: (string)
• feedConfig: (FeedConfig)

Returns:

• (optional Feed) A feed object representing the new connection, or nil on failure.
• (optional string) An error message on failure.

Usage:

local md = require "geneos.marketdata"
local config = {
  feed = { type = "example", ["library.filename"] = "flm-feed-example", },
  example = { publishingPeriod = "2000", }
  instruments = { ["GBP/USD"] = "some.code.for.GBP", ["USD/EUR"] = "the.code.for.EUR", },
  fields = { Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK", }
}
local exampleFeed = assert(md.addFeed("Example", config))

geneos.marketdata.countTicks(first[, last])

Counts the number of ticks in a tick list, as returned by Feed:getTicks().

This function counts the number of ticks in the inclusive range [first,last]. If last is nil, then all ticks in the list will be counted.

Parameters:

Returns:

• The count of ticks in the list.

geneos.marketdata.ticksToArray(first[, last[, count]])

Converts a list of ticks (as returned by Feed:getTicks()) to an array.

This function converts the ticks in the inclusive range [first,last] from a list to an array. The resulting output has the same order, so the tick at index 1 will be first. If last is nil then all ticks in the list will be converted. If count is not nil, it will be used to optimise the allocation of the array

Note: the next field in each tick will remain untouched. This may be of particular importance if you reorder the ticks in the array.

Parameters:

• first: (Tick)
• last: (optional Tick)
• count: (optional number)

Returns:

• (table) An array containing the ticks.
• (number) The number of ticks in the array.

Class FeedConfig

classgeneos.marketdata.FeedConfig

The feed configuration 'class' is a Lua table containing the configuration for a data feed connection.

A table conforming to this type is a required parameter for the geneos.marketdata.addFeed() function.

Settings that control how the feed adapter library is loaded.

The following fields can appear in this nested table:

• type (Required.) The feed type. If this is "custom", the feed adapter library file name is interpreted relative to the Netprobe working directory; otherwise it is interpreted relative to the flm subdirectory of the Netprobe directory.
• library.filename (Required.) The file name of the feed adapter library.
• library.skipVersionCheck (Optional.) Set to "true" if the version of the feed adapter library does not need to match the version of the Netprobe.
• library.debug.load (Optional.) Set to "true" to debug the process of loading the feed adapter library
• verbose (Optional.) Interpreted by the feed adapter library. In general, setting it to True enables verbose output from the feed adapter. It is a boolean and writes to the Netprobe logs.

Note: Where the name of a field includes dots, it can be specified either using Lua's syntax for field names that are not valid identifiers ["key.subkey"]="value" or using a nested table key={subkey="value"}

Usage:

-- The following examples show a configuration for an RFA feed,
-- and are equivalent due to the default values for the parameters.

-- verbose syntax, with redundant optional fields
local config = {
  feed = {
    type = "rfa",                                -- The feed type. See note above.
    ["library.filename"] = "flm-feed-rfa.so", }, -- Required: File name of the feed adapter library.
    ["library.skipVersionCheck"] = "false"       -- Do the version check (this is the default)
    ["library.debug.load"] = "false"             -- Don't debug loading the library (also default)
    verbose = "false",                           -- No verbose output from adapter (also default)
  },
  rfa = {} -- see FeedConfig.type
}

-- More concise syntax (but still with one redundant optional field)
local config = {
  feed = { type = "rfa", library = { filename = "flm-feed-rfa.so", skipVersionCheck = "false" } },
  rfa = {} -- see FeedConfig.type
}

<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7fc5040035b0)
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7fc5040035b0': SubscriptionStarted = { exceptions[] = { } }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 BID = 148.785000 ASK = 148.800000 HIGH = 148.965000 LOW = 148.749000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { ASK = 148.800000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }

<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7f3764000bd0)
<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7f3764000bd0': SubscriptionStarted = { exceptions[] = { } }
26SEP2018_10:36:45.991 24706:139876365346560 ERROR blpapi_platformtransporttcp.cpp:553 blpapi.session.transporttcp.{1}.<127.0.0.1:13523> Connection failed 

FeedConfig._type_

Feed adapter library settings.

The name of this nested table must match the value of the feed.type parameter. If the feed type is rfa for example, the table must be named rfa (see example 1 below).

The contents of this table depends on the feed parameters accepted by the feed adapter library. For non-"custom" feed types, parameters passed to the feed adapter will be prefixed with the feed type. If the feed type is "custom", the key names will be passed verbatim (i.e. without the "custom" prefix).

Usage:

-- Example 1: rfa feed configuration
local config = {
    feed = { type = "rfa", library = { filename = "flm-feed-rfa.so" } },
    rfa = { configFile = "RFA_DLL.cfg", session = "Session2" }
}
-- The feed adapter receives the following parameters:
--   feed.type = "rfa"
--   feed.library.filename = "flm-feed-rfa.so"
--   rfa.configFile = "RFA_DLL.cfg"
--   rfa.session = "Session2"

-- Example 2: "custom" feed configuration
local config = {
  feed = { type = "custom", ["library.filename"] = "feed_x.dll", },
  custom = { feed_x = { config = "local.ini" }, volume = "11" }
}
-- The feed adapter receives the following parameters:
--   feed.type = "custom"
--   feed.library.filename = "feed_x.dll"
--   feed_x.config = "local.ini"
--   volume = "11"

FeedConfig.instruments

Instruments to be subscribed to when feed is started.

This nested table is optional. If it is present, it must be a mapping from the instrument names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the instruments.

If this nested table is present, the fields nested table is required and will be used to specify the fields to subscribe to for each instrument.

Usage:

local config = {
  feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
  instruments = {
      ABC = "DIRECT_FEED.ABC.N",
      DEF = "DIRECT_FEED.DEF.N",
  },
  fields = { } -- see FeedConfig.fields
}

FeedConfig.fields

Default field name mapping.

This nested table is optional, unless the instruments table is present, in which case it is required. If it is present, it must be a mapping from the field names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the fields.

This table is used to specify the fields to subscribe to for each instrument, unless a different field mapping is specified in a call to Feed:subscribe()

Usage:

local config = {
  feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
  instruments = { },     -- see FeedConfig.instruments
  fields = {
    Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK"
  }
}

Class Tick

classgeneos.marketdata.Tick

The tick class is a Lua table representing a feed tick.

Tick objects are returned by the getTicks() method of feed objects. Ticks are generally members of linked lists representing part of the stream of ticks received for a single instrument on a feed.

Because a tick is represented by a Lua table, Lua functions (whether in your script or in another API module) can join or split these lists as required by setting or clearing the next field of the relevant ticks. Lua's garbage collector will dispose of ticks when and only when they are no longer part of a list whose head is still referenced by some variable that is still in scope.

Normally a Tick object represents a single tick from the feed adapter and the timeFirst and timeLast fields will be the same. However, if a number of ticks have been conflated (for instance, if the buffer between the feed adapter and the sampler has been overrun), then the timeFirst field will contain the arrival time of the first conflated tick and the timeLast field will contain the arrival time of the last conflated tick.

Tick.feed

The name of the feed, as supplied to geneos.marketdata.addFeed().

Tick.inst

The display name of the instrument.

Tick.next

The next tick in the list.

If there is no next tick, this key will be absent.

Tick.timeFirst

The time the tick (or first conflated tick) arrived.

The value of this field is a Lua number representing a number of seconds since the Unix epoch (Midnight UTC on 1st January 1970) That is, it has the same interpretation as a C language time_t, but it also has a fractional part.

Tick.timeLast

The time the tick (or last conflated tick) arrived.

Tick.field

The market data fields of the tick.

This is a table mapping the display name of each of the fields configured for the instrument to its value, as reported by this tick or copied from the previous tick. Field values may be numbers or strings, depending on the implementation and configuration of the feed adapter.

Tick.seq

The arrival sequence number of the tick.

This field is added to each tick by the Latency API if a tick history writer is configured.

Tick.image

Flag that indicates this tick is an image.

The boolean flag will be present and set to true if the tick was created as an image from the underling data source. If the flag is not present then the tick represents and update to an exising image.

Tick.remark

Annotation by latency API.

The Latency API adds up to one of the following remarks to certain ticks as it processes each sample:

Duplicate indicates that the tick was excluded by the pre-processing function defined for the feed;

MatchforN indicates that this feed tick matches the baseline tick with sequence number N;

MatchedbyN indicates that this baseline tick was matched by the feed tick with sequence number N (and may also have been matched by ticks from other feeds);

MatchwithlatencyL indicates that this feed tick matches the baseline tick that arrived L seconds previously;

MatchedwithlatencyL indicates that this baseline tick was matched by the feed tick that arrived L seconds later (and may also have been matched by ticks from other feeds).

Functions

geneos.sampler.commands.newDefinition()

Creates a new command definition.

Returns:

• (Definition) The new command definition.

Class Definition

classgeneos.sampler.commands.Definition

A command definition specifies the properties and arguments of a command.

Definition:setMenuPath(path)

Sets the menu path of the command.

The menu path defines the hierarchy where the command is displayed, when performing a right-click context-menu action in a Geneos GUI.

Parameters:

• path: (table)

Returns:

• (Definition) The definition object (i.e. self).

Definition:setDescription(description)

Sets the command description.

The description will appear at the top of the arguments dialog (if the command requires arguments).

Parameters:

• description: (string)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addStringArgument([description[, default]])

Adds a string-valued argument to the command.

String arguments will be rendered as a text-box in a Geneos GUI.

Parameters:

• description: (optional string)
• default: (optional string)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addIntegerArgument([description[, default]])

Adds an integer-valued argument to the command.

Integer arguments will be rendered as number spinner in a Geneos GUI.

Parameters:

• description: (optional string)
• default: (optional integer)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addNumericArgument([description[, default]])

Adds a numeric-valued (floating-point) argument to the command.

Floating-point arguments will be rendered as a formatted text-box in a Geneos GUI.

Parameters:

• description: (optional string)
• default: (optional number)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addBooleanArgument([description[, default]])

Adds a Boolean-valued argument to the command.

Boolean arguments will be rendered as a check-box in a Geneos GUI.

Parameters:

Returns:

• (Definition) The definition object (i.e. self).

Definition:addChoiceArgument(description, options)

Adds a choice of string-valued arguments to the command.

Choice arguments will be rendered as a drop down list in a Geneos GUI.

Parameters:

• description: (nil, string)
• options: (table)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addRowTarget(view[, pattern])

Adds a target to the command, for a row (or rows) of the specified view.

Parameters:

• view: (string, View)
• pattern: (optional string)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addColumnTarget(view, column)

Adds a target to the command, for a column of the specified view.

Parameters:

• view: (string, View)
• column: (string)

Returns:

• (Definition) The definition object (i.e. self).

Definition:addHeadlineTarget(view, headline)

Adds a target to the command, for a headline of the specified view.

Parameters:

• view: (string, View)
• headline: (string)

Returns:

• (Definition) The definition object (i.e. self).

Functions

geneos.sampler.doSample()

User-supplied function to perform sampling.

This function must be defined by the sampler script. It is called at the start of each sampling interval and whenever a user invokes the "Sample Now" command.

In standalone test mode, if this function returns a boolean false value (literally false, not nil), the sampler (and Netprobe) will stop. When running under the control of a Gateway, the return value is ignored.

Returns:

• (optional boolean) See above.

geneos.sampler.finalise()

User-supplied function called when sampler is stopped.

This function may optionally be defined by the sampler script. It is called just before the sampler is stopped, whether because it is being removed from the configuration or because the Netprobe itself is stopping.

Usage:

local gs = require "geneos.sampler"
gs.finalise = function() gs.logMessage("INFO", "Stopping") end

geneos.sampler.logMessage(level, ...)

Writes a message to the Netprobe log.

Specify a logging level followed by a series of arguments to be concatenated. Concatenation will occur directly, without any added separator characters.

Parameters:

• level:
• ...:

Usage:

-- The following are equivalent:
local gs = require "geneos.sampler"
gs.logMessage(40, "Hello ", "World!")
gs.logMessage(gs.logLevel.INFO, "Hello ", "World!")
gs.logMessage("INFO", "Hello ", "World!")

geneos.sampler.createView(name, columns)

Creates a new dataview published to the Geneos gateway.

This function creates a new dataview for the sampler which can then be populated and published. The newly created view will contain no table rows, and a single headline variable samplingStatus with the (string) value OK.

View names must be unique within a given sampler. Both view and column names should use only alphanumeric characters, underscore, dash or space (A-Za-z0-9_- ).

Parameters:

• name: (string)
• columns: (table)

Returns:

1. (optional View) A view object representing the new view, or nil on failure.

2. (optional string) An error message on failure. One of the following:

   • "view name must be a non-empty string"
   • "view '$name' already exists"
   • "view defines empty column name at index $ind"
   • "view defines duplicate column name '$col'"

Usage:

local gs = require 'geneos.sampler'
local view = assert(gs.createView("newView", { "rowName", "dataColumn1", "dataColumn2" }))

geneos.sampler.publishCommand(name, execute[, cmdDef])

Publishes a command to Gateway for this sampler.

This function publishes a command associated with this sampler to the Gateway. From Gateway, the command will then become available to clients such as Active Console 2 or Open Access. The precise rendering of the command and its input fields for arguments will vary depending upon the client.

When publishing, users should specify a short descriptive name for their command. This name must be non-empty string and should ideally contain only the characters a-z, A-Z, 0-9, _ (underscore). - (dash) and space. The short name must be unique for each command, and will produce an error if not.

Short names identify the different commands within a particular sampler. For use in the rest of Geneos a "full command name" is constructed, by prefixing the short name with the entity and sampler names (and also the sampler type if specified). The full command name will be returned by the function if the command is successfully published. Permissions or executions (e.g. from a rule or action) of commands can be applied on the Gateway as normal, using the full command name.

When a command is executed the execute function will be called. This function is passed two arguments, a target table and an args array. The target identifies which data item (e.g. a dataview cell) the user clicked on to issue the command. Args is an array containing the command arguments supplied by the user (if the command specified any arguments).

The cmdDef argument takes a command definition which specifies additional properties for the command, including arguments and targets. This definition can be provided either directly as a table, or via the fluent API available in the geneos.sampler.commands module. Arguments allow the command to prompt for users for input when the command is executed. Targets specify which data items the command will be available for (i.e. which items the command will show up for on a right-click). If no command definition is provided, the command will take no arguments and be available by right-clicking on any dataview published by the sampler.

For a more complete discussion on these topics, please see the user guide section.

Parameters:

• name: (string)
• execute: (function)
• cmdDef: (optional geneos.sampler.commands.Definition)

Returns:

1. (nil, string) On success returns the full command name as a string. Returns nil on error.
2. (string) On success returns nil. On error returns a message describing the problem.

Usage:

local sampler = require 'geneos.sampler'
local cmd     = require 'geneos.sampler.commands'
assert(sampler.publishCommand(
  'aNewCommand',
  function(target, args)
    if target.headline then
      print('command executed on ' .. target.view .. ':' .. target.headline)
    elseif target.row and target.column then
      print('command executed on ' .. target.view .. ':' .. target.row .. '.' .. target.column)
    elseif target.view then
      print('command executed on ' .. target.view)
    end
    print('Message was ' .. args[1])
  end,
  cmd.newDefinition()
    :setDescription('A new command')
    :addStringArgument('Message', 'A default message value')
    :addRowTarget('viewName', 'rowPattern*')
    :addColumnTarget('otherView', 'aColumnName')
    :addHeadlineTarget('otherView', 'headlineName')
))
-- or equivalently
assert(sampler.publishCommand(
  'another command',
  function(target, args) --[[ function body here ]] end,
  {
    description = 'A new command',
    args = {
      { type='string', description='Message', default='A default message value' }
    },
    targets = {
      { type='row',      view='myView',    name='rowPattern*'  },
      { type='column',   view='otherView', name='aColumnName'  },
      { type='headline', view='otherView', name='headlineName' }
    }
  }
))

Tables

geneos.sampler.logLevel

A table of constants for use with the logMessage() function.

The name of the constant corresponds with the severity description output by the Netprobe.

Fields:

• FATAL: (num)
• ERROR: (num)
• WARN: (num)
• INFO: (num)
• DEBUG: (num)

geneos.sampler.params

A table of parameters defined in the sampler configuration.

The parameters are specified as key-value pairs, and are stored in the table as strings.

Fields

geneos.sampler.name

The sampler name, as configured in the gateway setup file.

geneos.sampler.type

The sampler type.

The name of the type the sampler was referenced from. This will be an empty string if the sampler was not referenced from a type.

geneos.sampler.entityName

The name of the managed entity the sampler is attached to.

geneos.sampler.sampleInterval

The sample interval for the sampler.

This field may have a nil value if the sampler is configured to sample manually or at specific times.