FIX Analyser 2

Intended Audience Copied

This document is intended for users who wish to install and configure the FIX Analyser 2 plugin to implement FIX message monitoring as part of their existing Geneos installation.

The purpose of this document is to describe how to install, use and configure the FIX Analyser 2 plugin. The latter portion of the document also contains a technical reference for more experienced users.

The document assumes that users have some understanding of the FIX Protocol (see http://www.fixprotocol.org), as well as a basic knowledge of how to configure Geneos components such as adding a new Netprobe or sampler to a Gateway configuration.

Architecture Copied

This diagram shows the recommended deployment of Fix Analyser 2.

FIX logs produced by application processes (such as a FIX engine) are hosted on an Application server. These log files are read by a lightweight Geneos File Agent process and forwarded to FIX Analyser 2 which is hosted on a Monitoring server. This deployment ensures that parsing and analysis of the FIX messages do not impact on the performance of any applications on the Application server.

FIX Analyser 2 uses a SQLite in-process database to store order information for the FIX messages it is processing. The data contained in this database is stored as a file on the local disk and is used for tracking order states and latency data.

Message data can optionally be stored by FIX Analyser 2 in a MySQL, MariaDB, or PostgreSQL database. This data is used to augment HTML-based order reports by displaying drill-down message-level data for each order and enabling the message querying feature. Where possible, we recommend installing this database to take full advantage of the enhanced plug-in functionality.

As with previous versions of FIX Analyser, files available locally to the FIX Analyser 2 process can be read directly, without the use of a Geneos File Agent. However, since this involves deploying the analyser process on an application server, we recommend this configuration only for lower volume message rates (depending upon the host hardware capabilities).

Installation Copied

Component Prerequisites Copied

The components of the FIX Analyser 2 solution include:

• FIX Analyser 2 Netprobe
• File Agent
• MySQL/MariaDB or PostgreSQL

Refer to the Geneos Compatibility Matrix for the list of supported platforms.

Notes

• FIX Analyser 2 Netprobe is supported only on Linux 64-bit OS versions. If files are hosted on other platforms, a File Agent process is required to monitor them.
• With a configuration consisting of five dataviews, the FIX Analyser 2 Netprobe requires approximately 500MB of disk space for the SQLite database when processing 1,000,000 orders per day.
• If you are upgrading to FIX Analyser 2 version 4.13 from a lower version, you must delete the SQLite local cache. For more information, see localCache > database > sqlite > file.
• The FIX Analyser 2 Netprobe host must have the MySQL/MariaDB/PostgreSQL 64-bit client drivers installed if using the optional message database.
• The MySQL/MariaDB/PostgreSQL message database requires an average of 10GB of disk space to store three days’ worth of persisted data containing around 9 million messages (approximately 1 million orders, assuming an average of 8 acknowledgment messages per order).
• Netprobe, File Agent, and MySQL/MariaDB/PostgreSQL processes all require a TCP/IP listen port for communication. If any of these processes are behind a firewall, you must add a rule to allow access to these ports.
• When Security-Enhanced Linux (SELinux) is running in Enforcing mode, it may deny certain Geneos functions depending on the implemented configurations and policies.
  • To see which functions SELinux denies, check the audit log, typically located at /var/log/audit.log. Look for log entries of type AVC, which provide details of any denied access (for example, denied connections to TCP ports).
  • If you experience issues due to SELinux in Enforcing mode, you can either disable SELinux or create policy modules to grant the required access. Please consult your administrator or security team for assistance.

FIX Analyser 2 Netprobe binary Copied

The FIX Analyser 2 plug-in is hosted within a specialised Netprobe binary. Unlike a standard Netprobe this binary contains only the FIX Analyser 2 plug-in. CPU and memory usage of this probe is expected to be higher than for normal monitoring tasks due to the analytics being performed (dependent upon the message rate being processed). This probe allows plug-ins to run with a higher maximum memory threshold to cater for this.

Because of this change, other plug-ins such as CPU or FKM (for example) are not available in the FIX Analyser 2 binary, therefore other monitoring tasks will still require a standard Netprobe. If you are using the suggested architecture (see diagram above) you should consider running a standard Netprobe on the monitoring server for Geneos self-monitoring.

The default listen port for a FIX Analyser 2 Netprobe is now 7030. This can be changed as normal via the -port command-line argument or the NET_PORT environment variable.

Other Netprobe features such as executing remote commands, running action scripts, or floating / self-announcing run modes are unchanged. Command-line configuration options not relating to plug-ins are also unchanged. Please refer to the Netprobe User Guide for further details.

There are cases when the Fix Analyser machine and the File Agents or FIX logs may be on different time zones. Therefore, the time zone of FIX Analyser Netprobe is adjusted with the TZ environment variable in the start-up script.

Warning

Time zone differences between the FIX log timestamps and FIX Analyser machine can affect the displayed value of secondsSinceLastActivity and the actual calculation of time-out for the SessionStatus view.

If the FIX logs are in UTC and the FIX Analyser Netprobe is on a different time zone:

export TZ=UTC

Packaging Copied

The FIX Analyser 2 Netprobe is packaged as a tarball (a GZip’d TAR file). This package contains the following files:

The minimum installation of the FIX Analyser 2 Netprobe binary consists of the Netprobe executable and the libsqlite3.so dynamic library.

Users must use the provided libsqlite3.so dynamic library (by setting the LD_LIBRARY_PATH environment variable if the file is not in the Netprobe working directory). This version (3.7.11) is typically more up-to-date than the versions installed at the system level. The database file may become corrupted if Netprobe is run against the same database file using different SQLite versions.

The scripts in resources/db-schema are related to the installation of the MySQL/MariaDB/PostgreSQL message database. Once installation has been completed (or if you do not intend to use this feature), the files can be removed.

File Agent binary Copied

To monitor FIX log files hosted on supported platforms other than Linux 64-bit RHEL 5+, a File Agent must be installed on the machine.

For installation details please see the Geneos File Agent User Guide documentation.

MySQL/MariaDB/PostgreSQL Message Database Copied

FIX Analyser 2 can optionally log FIX message data to a MySQL, MariaDB, or PostgreSQL database. This data is used to enhance HTML-based order reports by displaying drill-down message-level data for each order and enabling parameterised message queries. To install a MySQL, MariaDB, or PostgreSQL database for use by FIX Analyser 2, please follow the steps below.

Server installation Copied

The free MySQL/MariaDB/PostgreSQL database can be installed using the package manager for your Linux distribution or downloaded from their respective websites. Due to licensing reasons, this software cannot be distributed by ITRS as part of any Geneos product.

Installing the database Copied

Please refer to the MySQL, MariaDB or PostgreSQL installation instructions for your platform.

FIX Analyser 2 requires the event scheduler feature of MySQL/MariaDB, or equivalent functionality in PostgreSQL, to automate data housekeeping tasks for low-maintenance operation of the message database.

To view the list of supported database, refer to Database support in Compatibility Matrix.

Securing the database Copied

The newly installed database must be secured.

MySQL/MariaDB Copied

For MySQL/MariaDB, this can be achieved using the mysql_secure_installation script:

/usr/bin/mysql_secure_installation

Recommended Security Options:

PostgreSQL Copied

For PostgreSQL, you can secure the installation by:

• Setting a strong password for the default postgres user.
• Modifying pg_hba.conf to limit remote connections and ensure proper authentication and access control.
• Using SSL to secure database connections.

Persisting the event scheduler Copied

• For MySQL/MariaDB, the FIX Analyser 2 schema install script will enable the event scheduler. However, to ensure this setting persists across server restarts, add the following line to your my.cnf configuration file:

  event-scheduler=ON
  

• For PostgreSQL, implement a similar data housekeeping mechanism using pg_cron or pgAgent to automate the deletion of old data.

Database schema installation Copied

FIX Analyser 2 comes packaged with a database schema file that can be used to create a database instance on MySQL, MariaDB, or PostgreSQL servers.

The current schema version is 1.3, which introduces the SellSideInst column in the data table. This allows FIX Analyser 2 to uniquely distinguish orders that have the same ClOrdID and BuySideInst.

There are several methods for installing this schema as described below.

Attended scripted installation Copied

A schema installation / upgrade script is provided in the resources/db-schema folder of the FIX Analyser 2 package. This script should be executed with database administrator (DBA) access.

To install a new FIX Analyser 2 database on MySQL/MariaDB/PostgreSQL, run:

cd resources/db-schema
./fix_analyser_schema_install.sh <mysql|postgresql> -n

During the installation process, you will be prompted to provide:

• DBA username and password
• Hostname and port of the MySQL/MariaDB/PostgreSQL server
• Name of the database to create
• FIX Analyser user account details (username/password)
• Time for daily data housekeeping (default: 00:15)
• Number of days to retain message data (default: 3)

Unattended scripted installation Copied

The installation script can also be executed in unattended mode by providing the required values as environment variables. For example:

Manual installation Copied

If you are unable or do not wish to use the installation script, you can manually install the database schema using the following steps. The SQL statements should be executed as a database administrator (DBA) on the respective database server.

Manual Installation for MySQL/MariaDB Copied

1. Connect to the MySQL/MariaDB Database.

   Use the MySQL/MariaDB command-line client to connect to your database:

   mysql -u dba_username -p dba_password [ -h db_host -P db_port ]
   

   For full details, see the MySQL documentation here.

2. Enable the MySQL/MariaDB event scheduler.

   To enable the event scheduler, run the following SQL commands:

   SET GLOBAL event_scheduler = ON;
   SHOW PROCESSLIST;
   

   The output of the SHOW PROCESSLIST command should return a row with the text event_scheduler if successful.

   Note

   You should also enable the scheduler permanently as detailed in the server installation section of this document.

3. Create a new database for FIX Analyser 2.

   CREATE DATABASE FIX_DB;
   

4. Create a user for FIX Analyser 2.

   Create a user with appropriate permissions to connect to the new database:

   CREATE USER fix_db@'%' IDENTIFIED BY 'fix_pw';
   

5. Grant user permissions.

   Grant the necessary permissions for the FIX Analyser user:

   GRANT event, execute, select, insert, update, delete ON FIX_DB.* TO fix_db@'%';
   

6. Load the database schema.

   Load the schema into the newly created database. You can either:

   Run the source command from within MySQL; or

   source mysql-fix-analyser-v1.3-db-schema.sql;
   

   From the shell, pipe the schema file into the MySQL client:

   mysql -u dba_username -p FIX_DB < mysql-fix-analyser-v1.3-db-schema.sql
   

7. Schedule housekeeping.

   Schedule a daily housekeeping event to remove expired message data. Set HH:MM to the daily execution time and DAYS to the number of days to retain message data:

   CREATE EVENT FIX_DB_housekeeping
   ON SCHEDULE EVERY 1 DAY
   STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE)
   DO CALL FIX_DB.removeExpiredFiles(DAYS);
   

   Recommendations:

   • Schedule the event at a time when no FIX sessions are operating or when no messages are sent. The default time is at 00:15.
   • Set the housekeeping schedule on the same day as the resetTime setting.

Manual Installation for PostgreSQL Copied

1. Connect to the PostgreSQL Database.

   Use the psql command-line tool to connect to your database:

   psql -U dba_username -h db_host -p db_port
   

2. Create a new database for FIX Analyser 2.

   CREATE DATABASE FIX_DB;
   

3. Create a user for FIX Analyser 2.

   Create a user with appropriate permissions for the FIX Analyser database:

   CREATE USER fix_db WITH PASSWORD 'fix_pw';
   

4. Grant user permissions.

   Grant the necessary permissions to the new user:

   GRANT ALL PRIVILEGES ON DATABASE FIX_DB TO fix_db;
   

5. Load the database schema.

   Load the schema into the database using the psql command:

   psql -U dba_username -d FIX_DB -f postgresql-fix-analyser-v1.3-db-schema.sql
   

6. Schedule housekeeping.

   PostgreSQL does not have an internal event scheduler like MySQL, so you must use pg_cron or an external task scheduler to periodically execute the housekeeping function:

   DO $$
   BEGIN
       PERFORM FIX_DB.removeExpiredFiles(DAYS);
   END $$;
   

Message Queue Limits Copied

FIX Analyser 2 maintains an internal queue of messages to be logged into the database. If the connection to the MySQL/MariaDB/PostgreSQL server becomes slow or unavailable, up to 1.5 million inserts will be cached before new data begins to be discarded.

Message Processing Copied

This chapter discusses how FIX Analyser 2 processes incoming FIX messages and generates data views based on the user configuration. This information is useful background knowledge when discussing column configurations and filtering in the plug-in setup.

Data flow Copied

This diagram shows the internal data flow of the FIX Analyser 2 plug-in.

The main stages in this data flow are as follows:

1. FIX log files are read as raw textual data.
2. Input text is optionally filtered before being further processed.
3. The data is parsed as a FIX message allowing access to FIX field values.
4. Order tracking creates a new order record for the message (if required) or updates the state of an existing order record. The FIX message is augmented with additional order state information before further processing. See section Message Processing.
5. Messages seen in sequence (see the out-of-order messages section below) are processed immediately. For each configured view in the FIX Analyser 2 setup, a table of analytics data is maintained. All tables are updated as appropriate by the message.
6. When the plug-in samples, the latest data from the analytics tables is published as dataviews, which are then processed by Gateway and displayed in Active Console.
7. Unsequenced messages (see the out-of-order messages section below) are processed by the out-of-order message table, in the same way that analytics tables are normally updated. When the “missing” message is seen (or a timeout elapses) this data is re-integrated back into the analytics tables making it available for sampling.
8. FIX messages are optionally logged to the MySQL/MariaDB/PostgreSQL message database.

   Note

   In the packaged MySQL database schema, the FixMessage is initially configured to a limit of 2048 characters. If you require FIX Messages with a length exceeding 2048 characters, you can modify the MySQL database schema by either increasing the VARCHAR or changing the data type to MEDIUMTEXT.

Order tracking Copied

The FIX Analyser 2 plug-in tracks the state of FIX (protocol version 4.2) orders, for use by configured order columns in a user view definition. By tracking orders information crossing multiple FIX messages can be captured, including latency information (the time for acknowledgement and fills) and the latest value of FIX fields referenced by order columns, for example the LeavesQty (FIX field 151).

Orders are inferred by examining all messages which are processed by the plug-in, using the following logic.

• All FIX messages of type Single Order (35=D), Cancel Request (35=F) or Cancel/Replace Request (35=G) are considered to be an “order”.
• The first reply seen for an order acknowledges the order, and is used to calculate the ackLatency time. The reply can be an Execution Report (35=8) or Order Cancel Reject (35=9) coming from the opposite message direction to the order request.
• Order acknowledgements for which the plug-in has not seen a corresponding request are handled by the out-of-order message processing. If no request message is seen then the acknowledgement latency is 0ms.
• An order may skip states depending on the message received. For example, a Single Order (35=D) message followed by an Execution Report (35=8) message where the order is filled immediately will move straight from the unacknowledged to filled states. In this situation the ackLatency and fillLatency will be identical.

There are several possible states an order can be in, which can be used for filtering using the order-column match states setting. These states are also used internally and are displayed in the diagram below. The diagram also shows, for each order type, the conditions (i.e. FIX message contents) under which the order would transition to this state. See Technical Reference guide for more details.

The FIX Analyser 2 plug-in also uses an internal tracking state for each order, which can (if desired) be configured for display by a value-type order-column or for output in a HTML report. These tracking states map to the order states above in the following way:

• New - an order not seen previously by FIX Analyser 2, corresponding to the order state unacknowledged.
• Acknowledged - an acknowledged order, corresponding to order states New and Partially filled.
• Finished - a completed order for which we expect no further messages. Corresponds to the remaining order states Filled, Replaced, Cancelled and Rejected

Out-of-order messages Copied

FIX Analyser 2 can handle a specific case of out-of-order FIX messages when tracking orders, where the initial message in an order is seen after some acknowledgement messages for that order. This case assumes that the acknowledgement messages are still ordered with respect to each other, and is intended to handle processing of FIX engine log files where the sent and received messages are logged to separate files.

Messages for sequenced orders, where the FIX messages are seen in normal ordered sequence, are processed immediately by configured views. For more information on sequenced orders, see Data flow.

Out-of-order messages are processed by creating an order-specific row in the out-of-order message table. Data stored in this table is retained and not published to dataviews when the plugin samples, and is considered ‘deferred’. An order will cease to be deferred if:

• The initial order creation message is seen.
• 15 seconds has elapsed since the order was deferred. The elapsed time is measured by absolute system time, not using message timestamps.
• There are more than 1000 order entries in the out-of-order message table. In this case, the oldest entries will cease to be deferred.

When an order stops being deferred, the stored data is reintegrated back into the views and will be visible in a dataview in the next sample. Once reintegrated, any further FIX messages for the order are processed directly by the views.

Filters configured for user view order columns may work slightly differently when messages are seen out-of-order. Orders reintegrated because the initial order creation message was seen will evaluate filters on order columns against this initial message. The remaining reintegration cases will evaluate the filters against the first acknowledgement message seen.

Debug deferred orders Copied

Beginning Geneos 5.3.0, deferred orders are no longer displayed in the Netprobe log.

To enable the logging of the received messages for deferred orders, do the following:

1. In the Gateway Setup Editor, select the probe running the FIX Analyser 2 sampler.
2. In that probe, select the Debug tab.
3. Under the Debug field, click Add new and enter FA2 under Module.
4. Click Setting. The Setting window appears.
5. In the Setting window, add deferred_orders to the Setting field and click Close.
6. Click Save current document to apply your changes.

Success

The Netprobe now logs deferred order messages to the log file.

Session state tracking Copied

If a session analysis view or session status columns of a user view has been configured, the view will display the current state of the FIX session for each unique pair of SenderCompId / TargetCompId (session parties) FIX field values. The session state is inferred based on the content of the FIX messages processed by the plug-in.

For best results when displaying session states, it is recommended that FIX administrative messages are made available to FIX Analyser 2, including logon, logoff and heartbeat messages.

The logic used to determine the connection state is described below:

• A session’s status is Up if the plug-in has seen a logon message (35=A) from an initiating party and the corresponding reply from the accepting party (an explicit logon). This means that the session is in the connected state.
• A session will also be considered connected if the plug-in detects a conversation already in progress (an implicit logon). This is detected by observing messages sent and received by both parties (that are not rejected).
• A session is considered to disconnected (displayed as Down) if a logout message and acknowledgement is seen. This indicates that the connection has been terminated normally.
• A connection is also considered disconnected if no messages are sent by either party for longer than the heartbeat interval defined for that session. In this case, the session status will still be displayed as Down, and the logoff column will additionally display TimedOut.
• Sessions for which the plug-in has seen no FIX messages at all, or only an initial logon message but not the acknowledgement, displays the status Pending.
• A session’s status is Down if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5). Sample FIX messages:

35=A
2016-06-01 05:34:38,039 INFO [Fix message assembler]
in.PARTY01_PARTY02 - <29 Logon
(8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21:
34:38108=3098=010=081)
35=5
2016-06-01 05:34:38,043 INFO [Protocol]
out.PARTY01_PARTY02 - >2774 Logout
(8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
21:34:3858=MsgSeqNum too low, expecting 2760 but received
29 Logon10=077)

• A session’s status is Down if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logoff (MessageType: 35=5), but in the opposite direction. Sample FIX messages:

35=A
2016-06-01 05:34:38,039 INFO [Fix message assembler]
in.PARTY01_PARTY02 - <29 Logon (8=FIX.4.29=6235=A49=
PARTY0156=PARTY0234=2952=20160531-21:34:38108=3098=010=
081)
35=5
2016-06-01 05:34:38,043 INFO [Protocol]
out.PARTY01_PARTY02 - >2774 Logout (
8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
21:34:3858=MsgSeqNum too low,
expecting 2760 but received 29 Logon10=077)
35=5 (Opposite direction as symbolized by "<")
2016-06-01 05:34:38,050 INFO [Fix message assembler]
in.PARTY01_PARTY02 - <2774 Logout (8=FIX.4.29=009935=534=
277449=PARTY0156=PARTY0252=20160531-21:34:3858=00100000001
o8YD_4204- Logout accepted10=021)

• A session’s status is Down if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logon, and an Explicit Logoff. Sample FIX messages:

35=A
2016-06-01 05:34:38,039 INFO [Fix message assembler]
in.PARTY01_PARTY02 - <29 Logon
(8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21:
34:38108=3098=010=081)
35=5
2016-06-01 05:34:38,043 INFO [Protocol]
out.PARTY01_PARTY02 - >2774 Logout
(8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531-
21:34:3858=MsgSeqNum too low,
expecting 2760 but received 29 Logon10=077)
35=A
2016-06-01 05:34:40,039 INFO [Fix message assembler]
in.PARTY01_PARTY02 - <30 Logon
(8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=3052=20160531-21:
34:40108=3098=010=081)
35=5
2016-06-01 05:34:40,043 INFO [Protocol]
out.PARTY01_PARTY02 - >2775 Logout
(8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277552=20160531-
21:34:4058=MsgSeqNum too low,
expecting 2760 but received 29 Logon10=077)

Data persistence Copied

During operation each FIX Analyser 2 sampler creates and maintains a SQLite database stored locally as a file on disk. This database is used to persist information about the current state of the sampler including:

• Order data - data for all FIX orders seen used by the order tracking functionality.
• File offsets - The current read position of input files FIX Analyser 2 is processing.
• View data - computed values as shown in the published dataviews.

This persistent storage allows FIX Analyser 2 to resume processing from where it left off, if the sampler is stopped and restarted (as opposed to re-processing FIX log files from the start). Offsets of remote files read via a File Agent process are also persisted.

Creation and maintenance of this database is intended to be (almost) invisible to the user. During normal operation no user involvement should be required.

Persistence of data for setup changes is maintained on a best effort basis. Data for unaltered areas of configuration will be retained, whereas other data may be cleared. For example, if a new dataview column is added the existing column values will be retained. Any new data will then update both the old values in existing columns and create new values for new columns. Changing a dataview column definition will result in stored values for the column to be lost - values are not recomputed against the new configuration.

FIX Analyser 2 can be made to forget its current state by manually moving or deleting the SQLite database. This is done by removing the database file which by default is named managedEntityName.samplerName.db and is located in the Netprobe working directory. This file should not be removed while FIX Analyser 2 is still running - either stop the Netprobe or disable / unconfigure the sampler prior to file deletion.

The order and view content of the database file are cleared daily at the specified reset time (by default at 00:00 - midnight).

User Guide Copied

This chapter describes how to configure and use the most common features of the FIX Analyser 2 plug-in. Users are expected to have some basic experience with using the Gateway Setup Editor to edit Geneos configurations.

Configuring Files Copied

Inputs to the FIX Analyser 2 plug-in are configured in the form of paths to log files containing FIX messages. These log files can either be available locally on the FIX Analyser 2 Netprobe host, or accessed from a remote host using TCP/IP via a Geneos File Agent process. More information on these settings can be found in the Files configuration section.

Message direction Copied

At a minimum, files must be configured with a file path and a message direction. The messageDirection setting describes to the plug-in whether the file contains FIX messages in two directions (the sendAndReceive option) or a single direction only.

Single direction only (the send or receive settings) should be used in the case where the FIX engine (or other logging process) splits the two sides of a FIX conversation into different log files. In this case it is not important which file is configured with which direction, but the files must have different directions (see the example below). It is recommended the direction is chosen to provide the least confusion to users maintaining the configuration.

Files containing FIX messages in both directions should be configured with the sendAndReceive message direction. This configuration also requires a regular expression (or regex for short) that specifies which messages in the file are in the receive direction - messages not matching this regex are assumed to be in the send direction.

As per single-direction send or receive files, it is not important which messages are chosen to be in the receive direction and which in the send: the regex should be chosen to be simple to maintain.

File paths Copied

Paths used in a file definition in FIX Analyser 2 can be configured to be either absolute, or relative to the working directory of the FIX Analyser 2 Netprobe (or File Agent if this file is hosted remotely). Path configurations may also contain wildcard characters (\* and ?) which by default will monitor all matching files found.

The file properties settings can be used to monitor only the latest modified file, which makes the behaviour more like other Geneos plug-ins such as FKM (see image below). These properties also control features such as reading FIX messages spanning multiple file lines (multi-line messages).

FIX Analyser 2 also supports date patterns (today, yesterday and tomorrow) for configuring paths, again in common with the Geneos FKM plug-in. See the filename setting and the time formatting codes appendix for more details.

Timestamp parsing Copied

When a new file definition is created in the FIX Analyser 2 plug-in configuration, timestamp parsing for FIX messages from this file is automatically set to sendingTime, which parses the content of the SendingTime field (52).

Depending upon the FIX engine or process sending the FIX messages, the precision of this field may only be to second resolution. FIX Analyser allows custom parsing of timestamp information using other information available in the same file line to obtain more precise timestamps.

To do this, users should examine the monitored log file and locate the timestamps for FIX message lines. A custom regex to capture the timestamp portion of the line should then be configured, where the first parenthesised expression will be the timestamp text. Many log files place the timestamp information at the start of a line, so a regex of the form ^(.{N}) where N is the length of the time text is generally a good starting point.

Once the text is extracted, it must then be interpreted using a format string. This information tells FIX Analyser 2 which parts of the timestamp text correspond with years, hours or seconds etc… using time formatting codes.

The configuration for example 2 above appears in the GSE as follows:

Reading remote files using Geneos File Agent Copied

FIX Analyser 2 supports reading a log file remotely via a Geneos File Agent. The configuration for this mode is very simple. Assuming the File Agent has been installed on the target machine, simply configure a file definition as you could for a normal file, then set the File Agent settings on the file to point to the File Agent process hostname (and port number, if not using the default port).

Additionally, the Fix Analyser 2 Netprobe has the ability to connect securely to the File Agent. To do this, the File Agent must be running in secure mode, and the Fix Analyser 2 Netprobe must enable the secure connection via the Active Console.

Admin View Copied

The administration (or Admin) view of the FIX Analyser 2 plug-in displays information regarding the FIX log file inputs to the plug-in.

Monitored files are represented as rows in the dataview output. If these files are located via a wildcard pattern in the default (‘match all’) mode, then they will appear as sub-rows of the pattern row. A non-wildcarded or ‘match latest’ path will be displayed as a single row. An example view is shown below.

Headline Legend

*Headlines only published when message database logging is enabled.

Table Legend

Session Analysis View Copied

The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day.

Configuration Copied

Configuration of the view consists simply of enabling the session analysis section in the views configuration list (see below). For additional information relating to session timeouts and expected status, the session analysis view can optionally be configured with a list of named conversations either directly or via an external INI-file.

/PLUGIN:FIX-analyseR2:sessionInfo

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='SessionAnalysis']/rows/row/cell

Arguments

Output Copied

Table Legend

Session administration events Copied

The adminMsg column for a session analysis view displays information about administrative events detected by the plug-in. Output displayed will be in the format “time: event” where time is the timestamp of the FIX message and event may be one of the following (see below).

The adminMsg column for each session row will be cleared (set to empty) when two minutes have passed since the FIX message timestamp relating to the event, except for session reject or logout messages which remain until replaced.

Message Detector Views Copied

The message detector view checks all incoming FIX messages against a filter configured for the view. Messages which pass the filter are processed and a new row representing this message is added to the view. Users have control over the filter used, how the messages are grouped and what is extracted from the message for display (user-configured columns).

Configuration Copied

The configuration example below shows the “Cancels” view from the example template configuration. The view uses the filter “cancel” which is defined in the template to match FIX messages where 35=8 (MsgType=ExecutionReport) and 150=4 (ExecType=Cancelled). Therefore, every row added to the dataview will represent an order being cancelled.

The messages in the output dataview are categorised by the uniqueId configuration. In the example configuration the fields chosen are FIX fields 49 and 56 (SenderCompId and TargetCompId). Therefore, each group of messages in the dataview represent the cancelled orders for each FIX conversation. Multiple messages seen for a connection will be displayed as sub-rows under a main group row, up to the configured messageCount limit.

The contents of each row are defined by the columns section. In this example, the values for each column (except Time) are extracted using the specified FIX field ids from the cancel message (fields 11, 55, … 38, 1). The Time column itself uses the named field TimeStamp which is one of an additional set of message properties provided by FIX Analyser 2.

Output Copied

Headline Legend

Table Legend

Clearing message rows Copied

Rows in the message detector view will be cleared automatically once the messageTimeout period (if configured) has elapsed. Messages can also be removed manually using the “Fix Analyser > Accept Message” right-click menu option. A whole message group can be removed by using the same option on a group row instead of a message sub-row.

/PLUGIN:FIX-analyseR2:acceptMessage

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param(\"PluginName\")='FIX-analyseR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell

Arguments

User Views Copied

The user view allows users to define their own views by applying select statistical operations to process messages monitored by the plug-in.

User views are configured in the views section of the plug-in setup. Multiple views can be configured each of which will process the same input data as read by the file configuration. Views can be defined directly in the plug-in setup, or referenced from a common section (using the Gateway Setup Editor’s static variables feature) to promote configuration sharing.

Categorisation (row configuration) Copied

Messages are categorised into 1- or 2-level groups using the row identifier options by specifying primary (and optionally secondary) FIX fields to categorise on. These categorised are displayed as rows (and sub-rows for 2-level groups) in the output dataview. A field chosen for categorisation should be present in the initial message of a FIX order and its value should not change over the life of an order (although it may be omitted from subsequent messages in the order).

Below is an example of the SessionStatisitics view with a primary and secondary row identifiers.

Statistical Calculations (column configuration) Copied

Statistical calculations on the input data are defined by configuring columns. There are two different types of column; message columns which take input from individual FIX messages and order columns which use values from order data computed by the plug-in.

Both column types are configured in the columns table in the User View configuration (see screenshot below).

The simplest type of column to configure is a message count column. To configure this, simply give the column a short understandable name, then click the “Message…” button and select count from the combo-box. This configuration will display a message count for each category row in the output dataview.

Other types of message column available are sum (which sums the values of the specified FIX field over all input messages) or value (which takes the latest value seen of the specified field). See Technical Reference guide for more details.

Message and order column data can be refined by adding either a filter or a rolling period or both. A filter attached to a column definition means the column will only process messages passing the filter. In the example above, a count message column “businessMsgCount” is configured with a filter BusinessMessage, which only passes business messages (ignoring FIX administration messages). The result is that the column only counts this type of message, rather than all FIX messages seen like the “msgCount” column.

A rolling period will cause the column to output only values seen within the defined period. The “msgCount_5Mins” column above will display a count of all messages seen in the preceding 5 minutes of each sample. If the BusinessMessage filter were additionally applied to this definition, the result would be a count of all business messages seen in the last 5 minutes.

Order columns behave much like message columns, but take data from order records rather than individual messages. The order tracking feature of FIX Analyser 2 tracks the latest value of FIX fields referenced by order columns per order, in addition to computing derived properties such as the ackLatency. This allows metrics such as the total LeavesQty (FIX field 151) to be calculated for each client.

Calculating this value using message columns would not be possible, as each successive ExecutionReport message would give decreasing values for the LeavesQty, which if summed would be incorrect. Using a value message column would give the correct value for a single order, but fail when messages for multiple orders are summarised by a single row in the view.

Order columns also have an additional built-in filter defined by the matchStates setting. This filter determines whether the FIX field value from the specified order will contribute to the output value or not, based on the current order state.

The screenshot above shows an order column configuration using match states. This definition produces a count of orders in the unacknowledged state. Filters and rolling periods (not shown in the screenshot) may further adjust the resulting value.

Output Copied

Table Legend

Using the supplied template setup file Copied

FIX Analyser 2 is packaged with a template for a Gateway Setup Editor (GSE) include file. This provides:

• Several static variables defining useful user views, message detector views and column filters.
• An example FIX Analyser 2 sampler definition using these views and filters. The sampler configures its input file(s) via a number of references to variables.
• An environment definition giving example values for the file configuration variables.
• A managed entity type referencing the example sampler.

Many of the examples in the previous sections are based on this template.

Having referenced this template from the Includes section of your Gateway setup, you can make use of it in a number of ways, depending on how closely the definitions in the template match your immediate requirements.

To make full use of the template, you can add the “FixAnalyser2” type provided by the template to a new or existing managed entity and reference an environment definition that provides the symbols required by the template. The “FixAnalyser2Example” environment in the template defines these symbols for a particular format of input file; you can either reference it directly or use it as a model:

Below is a Managed Entity definition referencing the template type and using a custom environment:

Alternatively, you can reference the views and/or filters defined by the template in samplers located either in your main Gateway setup file or in another include file.

Configuring the Message Database connection Copied

Settings for the MySQL/MariaDB/PostgreSQL message database are contained in the Advanced configuration section. More information on these settings can be found in Database settings.

Users will need to configure the following to authenticate with the database:

• Database server hostname
• Database instance name and port
• Optional: SSL Configuration You can configure the following SSL settings if required:
  • SSL Mode
  • CA Certificate, Client Certificate, and Client Key if you are using SSL to secure the database connection.
• Optional: Username and password

These details will be determined or configured during database installation.

Web-based reports Copied

Show Orders report Copied

Using the report Copied

All FIX Analyser 2 dataviews, except for the Admin view, provide a right-click context-menu command ShowOrders… on rows. This command produces an HTML-based report giving additional details on the orders associated with the row, and optional message drill-down functionality assuming the MySQL/MariaDB/PostgreSQL message database has been configured.

To use the command simply right-click on a row and execute it.

Note

Individual rows in a message detector view will typically only return one order, although a message group row may return more data.

/PLUGIN:FIX-analyseR2:showOrders

Targets

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='User']/rows/row/cell

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell

Arguments

Executing the command will display the following dialog. The number of results returned for a report is limited to maintain performance. By default the most recently updated 1000 orders are returned. The number can be changed as desired in the dialog.

When the command is executed in Active Console, the report will be displayed using the default browser on the client machine. Example report output is displayed below. Orders associated with the selected row will be listed in a table.

This table supports several options for controlling its content, such as the number of entries per “table page”, or whether completed orders will be hidden (they are shown by default). The columns of this table can be sorted by clicking on the column headers, and the table content can be filtered dynamically by typing in the search field.

If an order row is selected, then a FIX message drill-down table will be displayed below the order table. This feature requires the MySQL/MariaDB/PostgreSQL message database to be configured and available. The message table displays the individual FIX messages detected that made up the order, including chained messages where orders are changed or replaced.

As per the order table, the message table can also be sorted by clicking on the table headings. The raw FIX message contents (the full input log file line) is available by clicking the “Show/HideFIX messages” button.

Customising report output Copied

The columns in the order table of the report output can be configured using the reportColumns setting for both user views and message detector views. This allows fields to be added or removed on a per-dataview basis so that order reports for different views can be more relevant to the data being displayed for that view.

For example, three additional columns have been added to the default columns using the following setup:

This configuration produces the following output with an additional customized columns:

FIX Message Query Copied

If the MySQL/MariaDB/PostgreSQL message database has been configured, you can right-click on any FIX Analyser 2 dataview to access the Fix-Analyser > Query command.

/PLUGIN:FIX-analyseR2:queryMessage

Target

/geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview

Arguments

Executing the Query command will display the dialog window allowing query parameters to be selected:

Below are the following considerations when supplying values for the parameters in the dialog window above:

• If the StartTime and EndTime parameters specify an empty range (for example, the times are the same), then it ignores the values.
• The ClientID and OnBehalfOfCompID fields are added for MySQL/MariaDB/PostgreSQL database schema version 1.1.
• The BidirectionalCompID tickbox allows you to filter out messages using SenderCompID or TargetCompID. Below are the search conditions and their respective results when the BidirectionalCompID setting is enabled:

When the query is executed, the report output will look similar to the example screenshot below. The output can be sorted by table heading, the number of entries per view is changeable, and the table is dynamically searchable using the search field.

Altering the report URL Copied

The reporting URL sent to Active Console clients is automatically generated by the FIX Analyser 2 Netprobe. By default, this URL contains the hostname reported by the host machine and the Netprobe listen port. In some circumstances (particularly if accessing the Netprobe from a different domain) the hostname in the URL may fail to resolve.

To solve this issue, users may directly configure the URL that FIX Analyser 2 will report using the reportURLPrefix setting. The correct URL may either be directly specified, or indirectly specified via variables in the Gateway setup.

We recommend specifying the variable via a macro, such that the hostname used is taken from the Gateway probe configuration (and therefore more likely to contain the fully qualified domain name). An example of this is shown in the following screenshot.

Note

These variables can be configured {anywhere permissible - they do not have to be configured specifically on an entity.

The XML configuration for these variables is as follows:

<var name="probeHost">
<macro>
<netprobeHost></netprobeHost>
</macro>
</var>
<var name="probePort">
<macro>
<netprobePort></netprobePort>
</macro>
</var>

Once configured, the variables can be referenced in the reportURLPrefix setting:

Direct web access to reports Copied

Report URLs generated by FIX Analyser 2 contain any necessary data required to re-run the command. Message query URLs contain the query parameters, while order queries contain the name of the view and row originally clicked on when the command was issued. Refreshing or reloading the URL will regenerate the same report using the most recent data.

In addition, reports may be generated from a simple top-level page allowing users to specify report parameters as form input. This can be useful for making additional Message Queries (although less useful for Order Reports). The page can be reached at the following URL, and in the case of multiple FIX Analyser 2 samplers running will use the first configured instance to service requests.

http://probeHost:probePort/FixAnalyser/

The access page appears as follows:

Technical Reference Copied

This section describes the configuration settings and options available for the FIX Analyser 2 plug-in. Configuration for the plug-in is placed in the Gateway setup file under the fix-analyser2 section of a sampler descriptor.

A FIX Analyser 2 configuration consists of 4 major parts:

• Files - the set of files to be monitored.
• Views - the statistical output to be displayed.
• Filters - referenced by the view configuration to filter messages.
• Database - optional database configuration for FIX message logging.

Detailed descriptions of these sections are described below.

All settings which specify regular expressions are evaluated using PCRE, the Perl 5 compatible regular expression library.

Files Copied

The files configuration section lists all the FIX log files that the plug-in will monitor during operation. These files will be scanned for FIX messages, and any messages found will then be parsed and passed to all configured views for processing.

File names can be specified with absolute or relative paths, and may contain wildcard or date-based pattern. Files can also be streamed from a remote host by using a Geneos File Agent.

At least one file must be specified in order for the plug-in to operate.

Mandatory: Yes

files > file > filename Copied

This setting specifies the path of a file (or files) to monitor. The path can be specified either as a relative path from the Netprobe working directory, or as an absolute path.

If the path contains wildcard characters (\* or ?) then Netprobe will evaluate this path to find files matching the pattern. Either the latest or all of these files will then be monitored, subject to the value of the wildcardMode setting for the file. Monitoring all files can be useful for FIX engines which create a separate log file per FIX session.

The path may also contain the special tags <today>, <yesterday> or <tomorrow>. These tags will be expanded each day to a numerical representation of the date, YYYYMMDD.

For example, on 27th May 2011 the filename fix\_<today>\_file.log will be expanded to fix\_20110527\_file.log. Yesterday and tomorrow tags will expand to 20110526 and 20110528 respectively.

The numerical format can be controlled by inserting time formatting codes inside the tag. For example, on the same date the filename fix-<today%m-%d-%Y>.log expands to fix-05-27-2011.log.

Mandatory: Yes

files > file > fileProperties > wildcardMode Copied

The wildcard mode specifies how files configured with a wildcard (\* or ? characters) will behave. Available wildcard modes are as follows:

Mandatory: No

Default: MATCH_ALL

files > file > fileProperties > maxLinesPerCheck Copied

This setting specifies the maximum number of lines to process, each time the file is checked for data. Files are checked every second. If the maximum limit is reached but the file still contains lines to read, the remaining lines will not be processed until the next check. This setting can be used to fine-tune how file reading operates, in addition to the file control options.

This setting is not applied when files are being read via a file agent.

Mandatory: No

Default: unlimited

files > file > fileProperties > multiLine Copied

Specifies settings for reading log multiple lines from a log file, and treating this as a single message to process.

Mandatory: No

Default: Each individual file lines is treated as a single message

files > file > fileProperties > multiLine > rtsLog Copied

Shorthand setting to specify the multi-line configuration for an RTS log file. This is the equivalent of setting a custom configuration with the values:

• startRegex as (received|sending) message:$
• dataRegex as ^[^:]\*:(?<data>.\*)
• endRegex as ^([^|]\*)$
• stripNewlines as true

Mandatory: One of rtsLog or custom must be specified.

files > file > fileProperties > multiLine > custom Copied

Allows users to specify a custom multi-line message format, using regular expressions.

Lines are read from the input log file, and then checked against the startRegex to find a start of a message. When inside a message, lines are also checked against the dataRegex (if specified) to extract a specific part of the line, and the endRegex (if specified) to determine the end of the message.

If a line is found to match the startRegex while “inside” a message, then the current message data is passed for processing, and a new message is started. Lines matching the endRegex will also be added to the message, and end the message at that point.

Mandatory: One of rtsLog or custom must be specified.

files > file > fileProperties > multiLine > custom > startRegex Copied

Specifies the start of a multi-line message. A line matching this regex will start a new multi-line message, and pass any previous message onwards for processing.

Mandatory: Yes

files > file > fileProperties > multiLine > custom > dataRegex Copied

The data regex allows users to specify how to extract the “data” part of a file line to add to a multi-line message. The regex must contain a named capture-group called data. Only the part of the line matching this capture-group will then be added to the message.

For example, with the regex:

EMSOutput.java:420\) (?<data>.*)

And the file line:

2010-03-04 18:04:23,691 INFO (EMSOutput.java:420) 8=FIX.4.2 9=460 ... 10=090

The data added to the message would be:

8=FIX.4.2 9=460 ... 10=090

Mandatory: No

Default: The whole line is added to the message.

files > file > fileProperties > multiLine > custom > endRegex Copied

The end regex allows users to specify a pattern for a line that indicates the end of a multi-line message. When this line is seen it is then added to the message, and the completed message is passed onwards for processing.

Mandatory: No

Default: A message remains “open” until the next line matching the startRegex is seen.

files > file > fileProperties > multiLine > custom > stripNewlines Copied

The strip newlines setting controls whether newlines are stripped from the file lines, when appending new file lines to the multi-line message. If set to true, newlines will be stripped so the end message will seem to be one long line of data. If false, newlines from the file will be kept.

Mandatory: No

Default: false (newlines are retained)

files > file > fileAgent Copied

If configured, this setting specifies that the file should be read from a remote machine via a TCP/IP connection to a Geneos File Agent process running on that machine.

The hostname of this machine must be specified.

The listen port of the file agent is optional, and if not specified will use the default port 7030.

Mandatory: No

files > file > timestamp Copied

The timestamp setting describes how to obtain a message time, for data read from the file(s) the setting is specified against. In addition to displaying the timestamp in user-configured views, the time is also required for performing order matching latency calculations, and is used in the session analysis view when detecting FIX conversations.

Mandatory: Yes

files > file > timestamp > sendingTime Copied

This setting configures the plug-in to read the SendingTime field (FIX field 52) of FIX messages from the file to obtain the message timestamp.

Note

This field may only contain timestamps to second resolution, depending upon the exchange or FIX engine being used. It is recommended that clients check the log files when configuring the plug-in, and choose the most accurate timestamps available. See the custom timestamp setting in the Technical Reference guide for more details.

Mandatory: Either sendingTime or custom timestamp must be configured per file.

files > file > timestamp > custom Copied

A custom timestamp configuration should be used when the FIX SendingTime field (field 52) of a FIX message is either not present, or does not contain the message timestamp to the desired level of accuracy.

Using this custom timestamp setting, users can configure a regex pattern to extract the timestamp portion of a file line. The plug-in will then interpret this time using the configured format.

Mandatory: Either sendingTime or custom timestamp must be configured per file.

files > file > timestamp > custom > regex Copied

This regular expression (regex) is used to extract the timestamp component of the message, which is then interpreted by the format. The first capture group (the regex component in parenthesis) is taken to be the timestamp.

In example 1, the regex extracts the first 23 characters of the file as the timestamp.

Example 2 is similar; it extracts 23 characters of timestamp following the text “low information”, which must occur at the start of the file.

For the last example (3) again we extract characters from the start of the file, but this time 24 characters are required as the timestamp is slightly longer.

The regex can optionally be configured with flags that change the pattern matching behaviour. By default no flags are set. Available flags are:

Mandatory: Yes

files > file > timestamp > custom > format Copied

The format is used to interpret the timestamp once it has been extracted by the timestamp regex. Timestamps are interpreted using a string containing special time formatting code characters.

The fields of the timestamp in example 1 are read directly in the pattern.

For example 2 there is a choice on how to read the milliseconds. The format code %f will read both the dot (.) and the millisecond digits, or the code %Of will read only the digits and so needs to be prefixed by dot.

The timestamp for example 3 contains a textual month description instead of a numerical value. This is read by using the format code %b to read the month as text (locale dependant).

Finally, for example 4 the date does not contain the year. When interpreting this pattern, the plug-in will fill in the missing fields (in this case only the year) using the system date and time. For files containing only a time (and not the date) in the timestamp, the plug-in will fill in the missing date fields (the entire date) using the current date.

Mandatory: Yes

files > file > messageDirection Copied

Each message processed by FIX Analyser 2 has an associated direction, either SEND or RECEIVE. The direction is assigned to the message according to the value configured for this setting.

The message direction is used by FIX Analyser 2 in order to determine whether messages apply to same FIX session, and to invert paired FIX fields for grouping in a user view.

For example, if SenderCompId (FIX field 49) has been configured as a grouping, then messages in the SEND direction will be categorised using this field. However, messages in the RECEIVE direction will use values from the paired field TargetCompId (FIX field 56) for grouping instead.

Mandatory: Yes. One of send, receive or sendAndReceive must be specified.

files > file > messageDirection > send Copied

This setting indicates that all FIX messages from this file have a SEND direction.

Mandatory: Mutually exclusive (one of send, receive or sendAndReceive must be specified).

files > file > messageDirection > receive Copied

This setting indicates that all FIX messages from this file have a RECEIVE direction.

Mandatory: Mutually exclusive (one of ref:send , receive or sendAndReceive must be specified).

files > file > messageDirection > sendAndReceive Copied

The send-and-receive message direction indicates that the monitored log file contains FIX messages in both directions; the exact direction for each message read will be set using the configured regex.

Messages are marked as having a RECEIVE direction if they match the receiveId regex pattern. FIX messages which do not match this pattern are assumed to have a SEND direction. Possible FIX messages that are neither send nor receive messages (e.g. error reports) should be excluded using inputTextFilters.

The receiveId pattern can additionally be configured with flags that change its behaviour. Available flags are:

Mandatory: Mutually exclusive (one of send, receive or sendAndReceive must be specified).

files > file > fieldSeparator Copied

This setting specifies the field separator character used within a FIX message. This may be of use if the FIX engine logs messages using a different separator character (e.g. | or #).

This character can be specified directly (e.g. #) or using hexadecimal notation (e.g. \x23) or octal notation (e.g. \043). Hexadecimal or octal notation is useful when specifying non-printing or whitespace characters, as these will not display directly in the Gateway Setup Editor (GSE).

Mandatory: No

Default: The Start-Of-Header (SOH) special character as defined by the FIX protocol, which is ASCII code 001.

Filters Copied

Filters are used to select subsets of FIX messages, primarily when computing statistics for user-defined columns. Because the column types are generic (e.g. a count column is just an incrementing integer value) filters are used to make the columns more specific. For example, by attaching a filter to a count column to only process “New Order” messages, the column value then displays a count of all “New Order” messages seen.

Filters can either be defined in the filters configuration section or in a separate filter list which is then referenced by the plug-in. Individual filters are then referenced by name from other parts of the setup.

At present there are three types of filter; Comparison filters, Regex filters and Boolean filters (and, or and not).

Mandatory: No

filters > filter > name Copied

This setting specifies the name of the filter. Filters are referenced by the name from other parts of the plug-in configuration, and so are required to be unique.

Mandatory: Yes

Comparison Filters Copied

Comparison filters compare the content of a FIX message against a set of preconfigured values. This is the most common type of filter and can be used (for example) to filter by message type or order volume.

To configure a comparison filter 3 settings are required; the FIX option to compare, a comparison operation, and a set of values to compare against.

When using an equality and inequality operators, if both the FIX message value and the configured value are numeric (i.e. are integer or floating-point values) the comparison will be performed numerically. In all other cases comparisons are performed as a case-sensitive string comparison.

When using the other relational operators (i.e. <, <=, > and >=), comparisons will only be performed if the values are the same data type. That is, these operators will only be performed for strings against strings (using dictionary order), or numbers against numbers (numerical comparison). Any other comparisons will result in a “false” output. (i.e. “A” < 23 gives false, as does “A” >= 23).

filters > filter > comparison Copied

The comparison section of a filter marks the filter as being a comparison-type filter. This filter compares a FIX message field value against a set of pre-configured values.

Mandatory: Yes (if specifying a comparison filter).

filters > filter > comparison > fieldId Copied

This setting specifies the FIX field id which will be compared for a comparison filter. The value of this field will be extracted to compare against the configured values. If the message does not contain a field with this id, then an empty value will be used for comparison.

Mandatory: Yes

filters > filter > comparison > comparison Copied

Using this setting you can specify the type of comparison operation which will be performed for a comparison filter. The table below lists the available comparisons:

Mandatory: Yes

filters > filter > comparison > values Copied

This setting specifies values which the FIX field value will be compared against for this comparison filter.

If configuring LESS\_THAN or GREATER\_THAN operators, only a single value needs to be configured. During comparison all values defined will be checked, however for LESS\_THAN operators the value of the largest value will determine whether the message passes or fails the filter. Similarly for GREATER\_THAN operators it is the smallest value which is important.

Mandatory: Yes - at least one value must be configured per comparison filter.

Regex filters Copied

filters > filter > regex Copied

Regex filters compare the raw message content (i.e. the un-parsed text containing the FIX message) against a configured regular expression to check for a match.

Regex filters require a minimum of a regex pattern to be specified. If the message text matches the regex, then the FIX message will pass the filter and be processed.

The optional invert setting reverses the result of the filter. If enabled (set to true) then a message which matches the regex pattern will not pass the filter.

Mandatory: Yes (if specifying a regex filter).

Boolean filters Copied

Boolean filters allow multiple sub-filters to be combined together using Boolean logic to produce more complex and meaningful filters. Sub-filters of a Boolean filter can either be a name-reference to another filter, or a contained inline filter definition. Boolean filters can be made up of other Boolean filters if desired.

Three types of Boolean filter can be configured - and, or and not filters.

filters > filter > and Copied

The andBoolean filter combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass all configured sub-filters. If any one of the sub-filters fail to pass the message then this filter also fails. If no sub-filters are configured, the and filter will pass all messages.

Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

filters > filter > or Copied

The orBoolean filter combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass any one of the configured sub-filters. If all of the sub-filters fail to pass the message, then this filter also fails. If no sub-filters are configured, the or filter will pass all messages.

Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

filters > filter > not Copied

The notBoolean filter inverts the result of the configured sub-filter. A FIX message which passes the sub-filter will fail the not filter, and vice versa.

The sub-filter can be either a name-reference to another filter, or be a new filter defined inline at the point-of-use.

Mandatory: Yes (one of and, or, not must be specified).

Filter Lists Copied

Filter lists allow filter definitions to be shared between samplers. This can help reduce configuration burden (fewer filters need to be configured) and also increase maintainability by providing a single location to check when filters need to be updated.

Filters lists are configured in the top-level Staticvariables section of the gateway setup, inside the Fix-analysertemplates sub-section.

To use a filter list in a FIX Analyser 2 plug-in the list must be referenced by name. Filters defined in this list are then available for use by other parts of the plug-in setup, and are referenced by name in the same was as for normal filters defined directly in the plug-in setup.

Filter names inside a single sampler definition must be unique. If a filter in a filter list has the same name as another filter (either defined in the same filter list, another referenced filter list, or directly in the plug-in configuration) then this is an error. The Netprobe log will contain diagnostic output in this case.

Views Copied

The views configuration section defines the different views that the plug-in will produce. Each view is passed all FIX messages read in from the configured files, and the output of the view is defined by the user-configuration.

There are three types of view which can be configured:

• User view - a custom view created by the user, which can generate statistics on a per-message or per-order basis.
• Message Detector view - rows are created dynamically when a message passing some user-configured criteria is detected. Selected values from the FIX message are extracted and placed in the row.
• Session Analysis view - the session analysis view monitors FIX sessions (or conversations) based on the FIX messages being processed by the plug-in. Session state information and statistics on these sessions are output to the dataview. See Session Analysis view for more details.

User view Copied

views > user Copied

User views are a customised view configured by the user. These views contain configuration settings for the rows (categories) and columns (statistical values) to be computed for the input FIX messages and displayed in the dataview. Column values can be driven using order data or values from individual FIX messages as specified by the column type.

The user view can be configured either by specifying the view settings directly in the plug-in configuration, or by referencing a user view template static variable. If the view is specified as a static variable, the setup can be shared by a number of plug-in instances.

Mandatory: No

views > user > fix-analyser-user-viewTemplate Copied

This setting references a user view from a static variable.

Mandatory: No

See Static Variables for more information on static variables.

views > user > data Copied

This setting specifies a user view configuration directly in the plug-in setup.

Mandatory: No

views > user > data > viewName Copied

Specifies the name of the user view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

Row configuration Copied

views > user > data > rowIdentifier Copied

The row identifier setting specifies how FIX messages are categorised when computing statistical data in the output dataview. The values of the specified FIX fields are extracted from the FIX message being processed, and used to form the name of the row which will process updates from the message.

This process works slightly differently depending upon the column type as follows.

Message columns

For each FIX message processed, the value of the rowIdentifier FIX field(s) are extracted and used to generate a row name. If this row does not already exist a new row is added to the view. Message columns in the named row are then updated with the content of the FIX message.

If a message does not contain the specified FIX field (or has an empty value) then the value of the field as seen by the associated order for this message (if any) is used. If there is no order, or the order does not contain a value, then the text “” is used as the row name.

Order columns

Operation is similar to the above, except that row names are generated directly from the order information associated with the FIX message. The appropriate row is found (and created if necessary) and order columns updated with any changes to the order state caused by the FIX message.

Note

Order updates are limited to a single dataview row. As a consequence, the FIX fields used as the rowIdentifier (for categorisation) should be chosen with care. Fields which change value during the lifetime of an order are unsuitable as rowIdentifiers.

It is allowable however, that the first message of an order contains a value for the rowIdentifier field, and that the remaining messages do not contain this FIX field at all. In this case the field value will be cached and all FIX messages for the order correctly processed by the same dataview row.

Mandatory: Yes

views > user > data > rowIdentifier > custom Copied

The custom row identifier allows the user to specify the FIX field tags, which will be extracted and used as the row name. FIX Analyser 2 allows 1 or 2 levels of identifier tags; primary and secondary. Secondary tags can only be specified once a primary tag has been configured.

If both primary and secondary row identifiers are configured, the result is a view with 2 levels of row name. Row names of the form primaryValue#secondaryValue are value rows, and the statistics (column values) on this row display values for those messages with matching identifier fields. Row names of the form primaryValue are summary rows, and summarise the data of the value sub-rows with the same starting prefix.

See the user guide configuration section for example screenshots.

Mandatory: Yes - one of custom or senderTarget must be configured.

views > user > data > rowIdentifier > senderTarget Copied

The senderTarget setting specifies that the view should use the FIX fields SenderCompId (49) and TargetCompId (56) for the row identifier.

If using these fields you may also optionally enable the session status columns, which allows you to create a customised session analysis view.

Mandatory: Yes - one of custom or senderTarget must be configured.

views > user > data > rowIdentifier > senderTarget > showSessionStatus Copied

If enabled, this setting adds the columns status, logon and logoff to the user view. These columns are normally displayed in a session analysis view, so enabling these allows the user to configure a customised session analysis view using the user view.

Mandatory: No

Default: false - session status columns are not displayed.

views > user > data > rowIdentifier > senderTarget > showSessionStatus > sessionTimeout Copied

The session timeout specifies the default maximum period of inactivity for a session in seconds. If no messages are sent or received for this period of time, the session is considered to be timed out. This is displayed in the logoff column.

When timeout delay setting is non-zero, the maximum period is increased by the value specified. This setting can be used to cater for delays in writing or processing FIX messages which would otherwise incorrectly display a session as timed out.

Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host.

Mandatory: No

Default: 0 - no timeout

views > user > data > rowIdentifier > senderTarget > showSessionStatus > timeoutDelay Copied

The timeout delay setting specifies an extra period (in seconds) of allowable inactivity for a FIX session. If a session is idle for a period equal to sessionTimeout +timeoutDelay seconds, then the session is considered to be timed out. This setting is ignored if timeouts are disabled.

Expected use of this setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough depending on normal message processing rates.

Mandatory: No

Default: 0 - no timeout delay.

views > user > data > rowIdentifier > senderTarget > showExpectedStatus Copied

This setting controls whether the expectedStatus column is displayed in the view. This column is normally present in a session analysis view if named FIX conversations have been configured. It can optionally be added to a user view to allow users to configure a customised session analysis view.

The expectedStatus column displays whether a session (senderTarget row) is expected to be Up or Down based on the time information configured in the conversation setting. Values in this column will be empty unless conversations have been configured.

Mandatory: No

Default: false- expectedStatus column is not displayed.

Column configuration Copied

The column configuration defines the data to be displayed for each row in the dataview output. Users can extract message or order values and compute selected statistics from these for display, using the configuration described below.

User views currently support two types of column; message columns and order columns.

Mandatory: Yes

views > user > data > columns > column > displayName Copied

Specifies the name of the column as it will appear in the dataview. The name must be unique among all other columns configured for the view.

Mandatory: Yes

views > user > data > columns > column > rollingPeriod Copied

This setting allows a rolling period to be configured for the column. This allows you to (for example) configure a column that shows a count of messages received in the last 5 minutes.

Values are specified as an integer in 3 different units: seconds, minutes and hours. To configure a non-integer period of 1.5 hours for example, configure the period as 90 minutes instead.

The period unit-type also specifies the granularity of the period. Values will be placed into “buckets” containing all values seen for a particular time. The time is taken from the FIX message timestamp truncated to the unit specified. The overall value of the period is then a statistical operation on the most recent X buckets (where X is the specified period length).

e.g. When using a period of minutes:

Therefore, for more accurate rolling period values users should choose a more granular unit type when only using a small number of units. For example, a rolling period of 60 minutes would be better than a period of 1 hour.

Note

There is a memory cost to more granular rolling periods however, as more values need to be stored.

Mandatory: No

views > user > data > columns > column > filter Copied

References a filter from the filters section by name. Filters are applied to input messages before the column processes any data. This allows the creation of (for example) a count of Cancel messages, by configuring a count column and filtering out all non-Cancel messages.

See the message column and order column sections in the Technical Reference guide for more details.

Mandatory: No

Default: No filter - column will process all updates.

Message columns Copied

views > user > data > columns > column > columnType > message Copied

Message columns process FIX messages individually when computing their statistics. There are 3 types of message column which can be configured; sum, count and value columns.

If a message column is configured with a filter, then only messages which pass the filter are processed by the column. Therefore, a count column (for example) would count only the messages which passed the filter. This could be used to create a count of Cancel messages, by applying a filter that accepts only Cancel messages.

When aggregating an empty data range, the sum and value message columns will output an empty call in the dataview display (count columns will display as 0). This behaviour is intended and allows for improved data processing when used in conjunction with the database logging or charting features of Geneos.

Mandatory: Yes

views > user > data > columns > column > columnType > message > type > sum Copied

The sum message column sums the values of the specified FIX field, for all messages processed by this column (i.e. messages that passed the filter, if configured).

Summary rows for sum columns will display a sum of all values displayed for the sub-rows.

A sum message column could be used to obtain the total size (in bytes) of FIX messages processed for a particular party, by summing FIX field 9 (BodyLength).

Sum columns will produce an empty value when no data is present. This may occur when summing messages that do not contain the configured FIX field, if a rolling period is configured and no values have been received within the configured period, or if the column has processed no FIX messages at all.

Mandatory: Yes - one of sum, count or value must be configured.

views > user > data > columns > column > columnType > message > type > count Copied

The count message column is an integer value which starts at zero and increments by one for each message processed by the column (i.e. messages that pass the filter, if configured).

Summary rows for count columns will display a sum of all counts displayed for the sub-rows.

A count column could be used to could all Reject messages for a particular client, by configuring a filter to match on FIX field 35=9 (OrderCancelReject).

Mandatory: Yes - one of sum, count or value must be configured.

views > user > data > columns > column > columnType > message > type > value Copied

The value message column stores the extracted FIX field or named value of the latest message processed by the column (i.e. messages that passed the filter, if configured).

Named property values which can be selected are as follows:

Summary rows for value columns will display the latest value (using the message timestamp) of the values displayed by the sub-rows. If no messages with the configured field are received (within the rolling period if configured) then an empty dataview cell is produced. See the message column description for a brief summary on this behaviour in the Technical Reference guide.

A message value column could be used to show the latest error for a particular session, by extracting FIX field 58 (Text). This field is typically used to describe the error for a Don’t Know Trade (35=Q) or an order reject execution (35=8 and 150=8).

Mandatory: Yes - one of sum, count or value must be configured.

Order columns Copied

views > user > data > columns > column > columnType > order Copied

Order columns display a statistical summary for all orders applicable to the given dataview row. For example if the row configuration specifies that rows equate to exchanges, then order columns for a specific row will display statistics for orders to that exchange.

Available column types are:

• Value
• Sum
• Min
• Max
• Average
• Standard deviation

Order statistics can be filtered by the current order status, as specified by the match states setting. See the order tracking section in Message Processing for more details.

Filters for order columns are applied to the first message of the order seen for each dataview row, so that the order is only filtered once for that column.

Order columns (with the exception of the count column type) may return an empty cell value when aggregating an empty data range. This behaviour is intentional and allows for improved data processing when used in conjunction with Geneos database logging or charting features. See the specific reference section for each column-type for further details.

Mandatory: Yes

views > user > data > columns > column > columnType > order > type > matchStates Copied

This setting specifies the states of orders which will contribute towards the column value. Orders with a state that has not been configured here will not be considered by the column.

Possible states are as follows:

See the order tracking section for more details on state changes.

For example, a count order column configured for the unacknowledged state would show a count of all unacknowledged orders. Orders which were subsequently acknowledged would then be removed from the count. By adding a filter to this count, we could (for example) count all unacknowledged orders of a particular type.

Mandatory: Yes

views > user > data > columns > column > columnType > order > type > count Copied

The count column shows the number of orders which pass both the configured filter and order match states, or 0 when there are no matching orders. An example usage of this column could be to display the number of open orders (i.e. states “new” or “partiallyFilled”) for that row.

Mandatory: Yes - a valid order-column type must be configured.

views > user > data > columns > column > columnType > order > type > value Copied

The value column shows the latest updated value for the specified FIX field or named property, from orders which pass the configured filter and order match states.

Value columns will display an empty if no matching values are present. This may occur if messages for matching orders do not contain the configured FIX field, if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > sum Copied

The sum column sums values for the specified FIX field or named property extracted for all applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

An example usage for this column could be to sum FIX field 151 (LeavesQty) for all orders in states new or partiallyFilled. The result would be the total quantity for all orders, which are the units remaining to be executed (per dataview row).

Sum columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain th configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > max Copied

The max column displays the maximum value of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the maximum acknowledgement-latency for all orders of a dataview row.

Max columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > min Copied

The min column displays the minimum value of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the minimum acknowledgement-latency for all orders of a dataview row.

Min columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > average Copied

The average column displays the average of the FIX field or named property values extracted for orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the average acknowledgement-latency for all orders of a dataview row.

Average columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > type > stdDev Copied

The standard deviation column computes the deviation of all FIX field or named property values extracted for applicable orders, subject to the filters and order match states selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero.

A column of this type could be configured to display the standard deviation acknowledgement-latency for all orders of a dataview row.

Standard deviation columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour.

Mandatory: Yes (a valid order-column type must be configured).

views > user > data > columns > column > columnType > order > fields > fieldId Copied

The fieldId setting specifies a FIX field id, which will be used to extract values from FIX messages for use by an order column.

Mandatory: Yes - one of fieldId or named must be specified.

views > user > data > columns > column > columnType > order > fields > named Copied

The named setting specifies a property which will be extracted from an order for use by an order column. Available properties are as follows:

For more detail on the order status, see the order tracking in the Message Processing section.

Mandatory: Yes - one of fieldId or named must be specified.

Custom Order Report output Copied

views > user > data > reportColumns Copied

The report columns setting specifies the set of columns to display in the “Show Orders” command output when run for this dataview. If no report columns have been specified for a view, the default set of columns is used:

Mandatory: No

Message Detector view Copied

views > messageDetector Copied

Message detector views extract and display values from “interesting” FIX messages. They do this by dynamically adding new rows to the view when a message is detected which matches a configured filter. Each row in the view represents a message which passed the filter.

The values displayed for the rows are determined by the column configuration as detailed below. The message detector view can be configured either by specifying the settings directly in the plug-in configuration, or by specifying a message detector template as a static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances.

Mandatory: No

views > messageDetector > fix-analyser-message-detector-viewTemplate Copied

This setting references a message detector view from a static variable.

Mandatory: No

See Static Variables for more information on static variables.

views > messageDetector > data Copied

This setting specifies a message detector configuration directly in the plug-in setup.

Mandatory: No

views > messageDetector > data > viewName Copied

Specifies the name of the message detector view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

views > messageDetector > data > filter Copied

The filter setting specifies the filter used to determine “interesting” messages. FIX messages which pass the filter will be added to the message detector view as a new row.

The filter is referenced by name from the filters available to the plug-in. To filter on multiple “interesting” message types, combine multiple filters using a Boolean filter.

Mandatory: Yes

views > messageDetector > data > Order state filter Copied

Filters the messages on the view based on their order state. The order state filter takes effect after all other filters have been applied to the dataview. When an order state is ticked, it is included in the dataview.

By default, all order states are included in this filter and appear on the dataview.

Possible values:

• Unacknowledged
• New
• PartialFill
• Filled
• Replaced
• Cancelled
• Rejected
• Unknown

views > messageDetector > data > messageTimeout Copied

The message timeout setting specifies the maximum age a message can be (in seconds) before it is removed from the message detector view.

If the difference between the current time (obtained from the local time of the Netprobe host) and the message timestamp is larger than this threshold, then the row representing this message is removed from the view.

Mandatory: No

Default: 0 seconds - no timeout

views > messageDetector > data > messageCount Copied

The message count setting controls the maximum number of messages which will be displayed in a message group. If the count it set to 1, then the group summary row in the dataview will be omitted.

A message group is a set of messages which share the same unique id. If the maximum limit is reached, a new incoming message will replace the oldest message in the group.

Mandatory: No

Default: No per group maximum

Row configuration Copied

views > messageDetector > data > uniqueId Copied

The uniqueId setting specifies how to construct a unique row identifier (row name) from a FIX message for display in the message detector view.

The uniqueId is made up of one or more id elements, each of which can extract a FIX field or named property from the message. If using more than one id, the field values will be compounded together separated by dashes (the - character) to form the uniqueId.

If several messages are found with the same id, the message detector view will display these all as sub-rows under a group row named with the id. This set of messages is then called a message group. The maximum number of messages per group can be configured using the messageCount setting.

Mandatory: Yes

views > messageDetector > data > uniqueId > id > fieldId Copied

The fieldId setting specifies which FIX field to extract from a message. This value will then be used to form the uniqueId (the row name) in the message detector view.

Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

views > messageDetector > data > uniqueId > id > named Copied

The named setting specifies a named FIX message property to extract. This value will then be used to form the uniqueId (the row name) in the message detector view.

Available properties are as follows:

Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

views > messageDetector > data > uniqueId > id > autoGenerated Copied

The autoGenerated setting generates a series of incrementing numbers starting from 1. These values will then be used to form the uniqueId (the row name) in the message detector view.

Users should use this setting if they are unsure whether the FIX fields they have specified as identifiers will be unique. If using only autoGenerated to create a unique id, it is recommended to disable the message group feature via the messageCount setting.

Mandatory: Yes - one of fieldId, named or autoGenerated must be specified.

Column configuration Copied

views > messageDetector > data > columns Copied

This setting specifies the columns which will be displayed in this message analyser view. For each message which passes the filter, field values from the FIX message are extracted as per the column configuration and displayed as a new dataview row.

Mandatory: Yes

views > messageDetector > data > columns > column > displayName Copied

Specifies the name of this column as it will appear in the dataview. Column names must be unique amongst all other columns configured for this message detector view.

Mandatory: Yes

views > messageDetector > data > columns > column > value Copied

This setting specifies the source of the column value. Choose between Field id and named.

If a FIX field id is specified, then the value will be extracted from the FIX message. If the specified field does not exist in the message an empty value will be displayed.

Users may alternatively specify a named FIX message property rather than a field value. The available properties are as follows:

Mandatory: Yes

Custom Order Report output Copied

views > messageDetector > data > reportColumns Copied

The output of the “Show Orders Report” command can be customised for message detector views, in the same way as for user view. Please see the user view custom order report section in the Technical Reference guide.

Session Analysis view Copied

views > sessionAnalysis Copied

The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day. For more information, please refer to the section on session state tracking.

Each Fix Analyser 2 plug-in can output a single session analysis view, which is created when this setting is enabled.

Expected FIX conversations can be configured using the conversations setting. This also allows other properties such as an alias to be specified. A start and end time can be specified for a conversation during which time the session is expected to be up.

The session analysis view can be configured either by specifying configuration directly in the plug-in configuration or by specifying a session analysis view template static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances.

Mandatory: No

Default: No session analysis view is created.

views > sessionAnalysis > fix-analyser-session-analysis-viewTemplate Copied

This setting references a session analysis view from a static variable.

Mandatory: No

See Static Variables for more information on static variables.

views > sessionAnalysis > data Copied

This setting specifies a session analysis view configuration directly in the plug-in setup.

Mandatory: No

views > sessionAnalysis > data > viewName Copied

Specifies the name of the session analysis view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced.

Mandatory: Yes

views > sessionAnalysis > data > showSessionStatus Copied

The sessionTimeout setting specifies a default timeout period for a FIX session conversation. This period will be overridden by the timeout (if specified) of the corresponding named conversation if available.

Combined with the timeoutDelay setting, this defines a default maximum period of (sessionTimeout+timeoutDelay) seconds of inactivity for a session. If no messages are sent or received for this period of time, the session is considered to be timed out. This condition is displayed in the logoff column.

Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host.

The expected use of the timeoutDelay setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough for normal message processing rates.

Mandatory: No Defaults: 0 seconds session timeout (no timeout), 0 seconds timeout delay (no delay).

Named FIX conversations Copied

conversations Copied

The conversations section allows users to define a set of named FIX conversations between known parties. This list is used both in the session analysis view and in user views where the showSessionStatus setting has been enabled.

The named conversations are then always displayed in those views, which can be used to detect unseen conversations which are down (without configuring sampler expect-rows).

Conversation entries can also be used to add more readable names, as well as configure a specific timeout time or a pair of times between which the conversation should be up to assist with Gateway rules and alerts.

Mandatory: No

conversations > configFile Copied

This setting specifies that the list of conversations is defined in an external file. At present this file is expected to have the same format as the Appia FIX engine configuration file (named appia.ini by default). The Netprobe must be able to read this file in order to load the conversation settings.

Conversations are detected within the appa.ini file by looking for connection configuration details. The keys local\_firm\_id and remote\_firm\_id are used as the SenderCompId and TargetCompId settings for the conversation configuration.The heartbeat\_interval key becomes the timeout.

The trading periods for the session are detected by resolving the trading\_session key value, and extracting the trading\_session\_dow, trading\_session\_start\_time and trading\_session\_end\_time values. The comments setting is populated with the text from that section of the ini file in case any other settings are useful.

The INI-file is only read once at when FIX Analyser 2 is started, therefore a Netprobe restart or plug-in setup change is required to pick up any changes made to this configuration file.

An example except from an INI file is shown below. The highlighted text indicates fields used by FIX Analyser 2 when parsing the file.

[MAIN]
connection = [BANK-EXCHANGE]
trading_session = [NORMAL_HOURS]
[BANK-EXCHANGE]
connection_type = FIX_44
contact = 24 hour support contact number +44 (0)20 xxx yyyy
description = Live Connection from Bank to Exchange
heartbeat_interval = 30
local_firm_id = BANK
remote_firm_id = EXCHANGE
trading_session = [NORMAL_HOURS]
[NORMAL_HOURS]
trading_session_auto_connect = OFF
trading_session_auto_disconnect = OFF
trading_session_auto_eod = ON
trading_session_auto_retry = OFF
trading_session_dow = MONDAY
trading_session_dow = TUESDAY
trading_session_dow = WEDNESDAY
trading_session_dow = THURSDAY
trading_session_dow = FRIDAY
trading_session_end_time = 21:10:00
trading_session_start_time = 21:40:00

Mandatory: Yes - one of conversations or configFile must be specified.

conversations > conversations Copied

This setting specifies the list of conversations as entries defined inside the plug-in configuration.

Mandatory: Yes - one of conversations or configFile should be specified.

conversations > conversations > conversation > compIdA Copied

Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the timeout) should be applied. compIdB must also be specified.

Mandatory: Yes

conversations > conversations > conversation > compIdB Copied

Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the timeout) should be applied. compldA must also be specified.

Mandatory: Yes

conversations > conversations > conversation > timeout Copied

This setting specifies the timeout interval (in seconds) for this particular FIX conversation. If a value of 0 is specified this session will not time out.

If this setting is not specified, then the default value is used instead. For a session analysis view the default is specified by the sessionTimeout setting. For a user view please see the showSessionStatus sessionTimeout setting in the Technical Reference guide.

Mandatory: No

Default: Uses sessionTimeout setting applicable to the view type.

conversations > conversations > conversation > alias Copied

Specifies an alternate name (an alias) for the conversation. This name will be displayed in the dataview instead of the sender and target ids.

Mandatory: No

Default: If no alias is specified the format “Sender: Target” is used to generate the row name.

conversations > conversations > conversation > comments Copied

Allows additional details to be added for this conversation. These details can be displayed by issuing the “Show session info” context-menu command in the session analysis view.

Mandatory: No

Default: No comment

conversations > conversations > conversation > tradingSessions > tradingSession Copied

Specifies a trading session for this conversation. Trading sessions are used to populate the expectedState session-analysis column. A conversation has an expected state of Up if the local time on the Netprobe host falls inside the times specified by a trading session.

The trading session must have a startTime and an endTime specified in the form HH:MM or HH:MM:SS using 24-hour clock. The end time may be before the start time.

A trading session definition must also specify the days the session is active for. At least one day must be specified.

Mandatory: No

Default: No trading session. The expectedState column will be empty for this conversation.

Advanced settings Copied

filterPlugin Copied

This deprecated setting controls the plug-in level message filtering feature. Users are recommended to switch to using the more powerful inputTextFilters replacement feature. Messages filtered at this level are not processed by any views configured in the plug-in, and are also not used for order tracking or database logging.

Messages must match all configured positive filters, and not match any configured negative filters to be processed.

Mandatory: No

Default: No filters - all messages are processed

filterPlugins > positiveFilter Copied

If positive filters are configured, then messages must pass the positive filters before they are processed by the plug-in. Positive filters are specified as a plain string, which must then be present in any raw lines read in from the source file(s). Regex syntax is not supported here.

Mandatory: No

filterPlugins > negativeFilter Copied

If negative filters are configured, then any message which matches a negative filter is discarded and not processed by the plug-in. Negative filters are specified as a plain string, which must then not be present in any raw lines read in from the source file(s). Regex syntax is not supported here.

Mandatory: No

inputTextFilters Copied

The inputTextFilters setting provides global message filtering on all input data. These filters are applied before any FIX message parsing is performed, and so will prevent processing of the message by any configured views, order tracking or database logging.

Filters are configured as an ordered list of regular expressions. Each input message read by the plug-in is checked against these filters in turn. The input is compared against each regular expression in order, and if a match is found the message will be either included (processed by the plug-in) or excluded (ignored) as defined by the filter configuration.

If a message does not match the filter regular expression, the next filter in the list will be checked. If no filters in the list match a message then:

• When only “include” filters are configured, the message is excluded (ignored),
• Otherwise, the message is included (processed).

This list of filters compares similarly to a list of ACCEPT/REJECT rules of a firewall configuration.

The regex for a filter can be configured via a variable, instead of being directly configured in the sampler definition. If using variables then the referenced variable must be of “regex” type.

Filter regexes can be made case insensitive by enabling the optional “i” flag. By default they are case sensitive.

Filter regexes can also be optionally “inverted”. This means that if a message does not match the pattern then it will be included or excluded according to the filter configuration. This setting should be used when you want to further filter any remaining messages that do not match the regex.

For example, “Exclude messages not matching (i.e. invert) regex “8=FIX4.2” to only process FIX4.2 messages but still allow further filtering on these messages.

Mandatory: No

Default: No filters - all FIX messages are processed.

publishToObcerv Copied

If this is present, this allows the users to enable or disable the publishing of all FIX messages read by the sampler to Obcerv. If enabled, all FIX messages are published regardless of filter settings, and can be consumed by the Obcerv FIX Monitor application. The configurations in the Obcerv Connection section of the Gateway setup are used by the Obcerv connection.

If the section is not present then FIX messages are not published to Obcerv.

Mandatory: No

Default: True

showOrders > reportURLPrefix Copied

Report output generated by the “Show Orders Report” and “Show Messages” commands are published by FIX Analyser 2 as a link to an internally-generated HTML webpage.

The reportURLPrefix setting controls the hostname component of the link URLs. By default the link contains the hostname as obtained from the local machine. Users can use this setting to provide a different (or more specific) hostname to use, which may be required if the Netprobe is hosted in a different domain or DNS environment from users attempting to view the reports.

For example, to use the same hostname value as configured in the gateway setup file, configure variables of type “macro > netprobeHost” and “macro > netprobePort” on your managed entity. These values can then be referenced in the reportURLPrefix, e.g. “$(FIXHOST):$(FIXPORT)”

Mandatory: No

Default: Uses the Netprobe machine hostname and Netprobe listen port.

resetTime Copied

The resetTime setting controls the time of day at which FIX Analyser 2 will reset its internal state. When a reset occurs, all stored order data is cleared and all output views are blanked. If any named conversations have been configured, then session analysis views (or user views configured to show session data) will contain empty rows representing the sessions.

It is recommended that the resetTime be set to a time of day when ideally, no FIX sessions are operating or when no messages are sent. If possible, the resetTime should also be different to the MySQL/MariaDB/PostgreSQL message database housekeeping time. The default housekeeping time is 00:15.

Mandatory: No

Default: 00:00 (midnight)

resetDays Copied

The resetDays setting sets the day of the week when resetTime will commence.

It is recommended that the resetDays be scheduled on a day when no FIX sessions are operating, or when no messages are sent.

Mandatory: No

Default: Everyday

Database settings Copied

This section specifies the database connection details for the FIX Analyser 2 message database. The message database is required for the message query report, and the message-level drilldown in the orders report.

Mandatory: No

database > enabled Copied

This Boolean setting specifies whether message database logging is enabled. This flag allows the feature to be disabled without un-configuring the database details.

Mandatory: Yes

database > type > mysql/mariadb Copied

This setting specifies the MySQL/MariaDB connection details and SSL configuration details.

Mandatory: Yes - if the database setting has been specified (SSL configuration is not mandatory).

database > type > postgresql Copied

This setting specifies the PostgreSQL connection details and SSL configuration details.

Mandatory: Yes - if the database setting has been specified (SSL configuration is not mandatory).

database > username Copied

The username to use when connecting to the database. This user must have the appropriate privileges to the configured FIX Analyser 2 database instance.

Mandatory: No

Default: No username

database > password Copied

An optional password to use when authenticating with the database server as the specified username. The password can be one of the following: external password, encrypted password, or a variable set within the GSE.

Mandatory: No

Default: No password

database > filter Copied

Optional filter. If specified, only FIX messages that pass the filter will be logged to the MySQL/MariaDB/PostgreSQL message.

Mandatory: No

Default: No filter

Local cache settings Copied

localCache Copied

The local cache section contains optional settings controlling the local order store, which FIX Analyser 2 uses to maintain intra-day order records. Currently the local cache is implemented by an on-disk SQLite database instance, as described in the Data persistence in the Message Processing section.

localCache > database > sqlite > file Copied

Specifies a custom filename used to store the SQLite order database. The file will be created and initialised automatically if it does not exist. The file must remain accessible while FIX Analyser 2 is running.

Mandatory: No

Default: The default filename is of the form managedEntityName.samplerName.db. Characters in the entity or sampler names which cannot be used in a file path will be replaced by an underscore _ character.

localCache > orderCacheSize Copied

The order cache size specifies how much memory space (in megabytes) to allocate for caching order records in FIX Analyser 2. This caching helps reduce how frequently data must be read or written to disk, thereby increasing performance.

Using the default of 5MB allows approximately 13k order records to be stored in memory, as each order record is just under 400 bytes in size.

Note

This figure relates only to order state, other information such as field values for reporting will increase the memory footprint of FIX Analyser 2.

Mandatory: No

Default: 5 (MB), the minimum allowable value. Maximum is 2048 MB.

localCache > writeBackCount Copied

The write back count controls how many orders are written out to disk at a time. This typically occurs when an order needs to be placed in the cache and there is no free space available.

Larger values mean that more orders are written leaving a larger amount of free cache space and thus improving performance for subsequent insertion operations. However, the written-out orders may potentially be required at a later time and so will need to be cached-in again which would decrease performance.

Mandatory: No

Default: 10

Debug settings Copied

debug Copied

The settings in the debug section enable different log outputs in FIX Analyser 2, to assist in diagnosing any problems. The available settings are:

Mandatory: No (the debug section and each setting inside are optional)

Appendices Copied

Appendix A - Time Format Codes Copied

The following time formatting codes can be used with this plug-in. These codes work both for the time format (when reading a timestamp from a log file) and also when configuring a filename to read from, where the filename contains the current date.

Some examples:

Appendix B - MySQL/MariaDB message database schema Copied

Version 1.3 schema Copied

The latest schema for the MySQL/MariaDB message database is packaged with the FIX Analyser 2 Netprobe. This section describes the latest schema version available (1.3). For PostgreSQL, the schema is similarly designed but adapted for PostgreSQL features and syntax. Instructions for database installation can be found in the database installation section.

Note

In the packaged MySQL database schema, the FixMessage column is initially configured with a limit of 2048 characters. If FIX messages longer than 2048 characters are required, you can modify the MySQL database schema by either increasing the VARCHAR length or changing the data type to MEDIUMTEXT. For PostgreSQL, the FixMessage column is similarly configured with VARCHAR(2048), which can also be adjusted as needed.

MySQL/MariaDB Schema Copied

-- --------------------------------------------------------
-- Table structure for table `data`
--
CREATE TABLE IF NOT EXISTS `data` (
  `TimeStamp` decimal(20,6) NOT NULL,
  `FileId` int(10) unsigned NOT NULL,
  `SenderCompID` varchar(128) COLLATE latin1_bin NOT NULL,
  `ClientID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `TargetCompID` varchar(128) COLLATE latin1_bin NOT NULL,
  `OnBehalfOfCompID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `MsgType` varchar(4) COLLATE latin1_bin NOT NULL,
  `ClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL,
  `BuySideInst` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `SellSideInst` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `BuySideField` int(11) DEFAULT NULL,
  `OrigClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL,
  `Symbol` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `SecurityID` varchar(128) COLLATE latin1_bin DEFAULT NULL,
  `ExecType` varchar(4) COLLATE latin1_bin DEFAULT NULL,
  `OrdStatus` varchar(4) COLLATE latin1_bin DEFAULT NULL,
  `OrderQty` float DEFAULT NULL,
  `CumQty` float DEFAULT NULL,
  `FixMessage` varchar(2048) COLLATE latin1_bin NOT NULL,
  KEY `FileId` (`FileId`),
  KEY `MsgType` (`MsgType`),
  KEY `Order` (`ClOrdID`, `BuySideInst`, `SellSideInst`, `BuySideField`),
  KEY `OrigOrder` (`OrigClOrdID`)
) DEFAULT CHARSET=latin1 COLLATE=latin1_bin;

-- --------------------------------------------------------
-- Table structure for table `files`
--
CREATE TABLE IF NOT EXISTS `files` (
  `FileId` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `Offset` bigint(20) NOT NULL,
  `Date` date NOT NULL,
  `File` varchar(1024) COLLATE latin1_bin NOT NULL,
  `Host` varchar(255) COLLATE latin1_bin NOT NULL,
  `UniqueFileID` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`FileId`)
) DEFAULT CHARSET=latin1 COLLATE=latin1_bin AUTO_INCREMENT=1;

-- --------------------------------------------------------
-- Table structure for table `version`
--
CREATE TABLE IF NOT EXISTS `version` (
  `major` int(11) NOT NULL,
  `minor` int(11) NOT NULL
) DEFAULT CHARSET=latin1 COLLATE=latin1_bin;
INSERT INTO `version` (`major`, `minor`) VALUES (1, 3);

-- --------------------------------------------------------
-- Data management procedure.
-- Culls data older than the specified threshold.
delimiter $$
CREATE PROCEDURE `removeExpiredFiles`(IN `daysOld` int)
BEGIN
  DELETE FROM `files`, `data`
  USING `files`
  LEFT JOIN `data` ON `data`.`FileId` = `files`.`FileId`
  WHERE TIMESTAMPDIFF(DAY, `files`.`Date`, NOW()) > `daysOld`;
END$$

PostgreSQL Schema Copied

For PostgreSQL, the schema structure and data types are adapted to PostgreSQL syntax:

-- --------------------------------------------------------
-- Table structure for table `files`
--
CREATE TABLE IF NOT EXISTS files (
  FileId serial PRIMARY KEY,
  "Offset" bigint NOT NULL,
  Date date NOT NULL,
  File varchar(1024) NOT NULL,
  Host varchar(255) NOT NULL,
  UniqueFileID bigint DEFAULT NULL
);

-- --------------------------------------------------------
-- Table structure for table `data`
--
CREATE TABLE IF NOT EXISTS data (
  TimeStamp numeric(20,6) NOT NULL,
  FileId integer NOT NULL REFERENCES files(FileId),
  SenderCompID varchar(128) NOT NULL,
  ClientID varchar(128) DEFAULT NULL,
  TargetCompID varchar(128) NOT NULL,
  OnBehalfOfCompID varchar(128) DEFAULT NULL,
  MsgType varchar(4) NOT NULL,
  ClOrdID varchar(256) DEFAULT NULL,
  BuySideInst varchar(128) DEFAULT NULL,
  SellSideInst varchar(128) DEFAULT NULL,
  BuySideField integer DEFAULT NULL,
  OrigClOrdID varchar(256) DEFAULT NULL,
  Symbol varchar(128) DEFAULT NULL,
  SecurityID varchar(128) DEFAULT NULL,
  ExecType varchar(4) DEFAULT NULL,
  OrdStatus varchar(4) DEFAULT NULL,
  OrderQty double precision DEFAULT NULL,
  CumQty double precision DEFAULT NULL,
  FixMessage varchar(2048) NOT NULL
);

-- --------------------------------------------------------
-- Table structure for table `version`
--
CREATE TABLE IF NOT EXISTS version (
  major integer NOT NULL,
  minor integer NOT NULL
);
INSERT INTO version (major, minor) VALUES (1, 3);

-- --------------------------------------------------------
-- Data management function.
-- Culls data older than the specified threshold.
CREATE OR REPLACE FUNCTION removeExpiredFiles(daysOld integer) RETURNS void AS $$
BEGIN
  DELETE FROM data USING files 
  WHERE data.FileId = files.FileId 
  AND files.Date < NOW() - INTERVAL '1 day' * daysOld;

  DELETE FROM files 
  WHERE files.Date < NOW() - INTERVAL '1 day' * daysOld;
END;
$$ LANGUAGE plpgsql;

Upgrading the database schema Copied

FIX Analyser 2 comes packaged with database schema upgrade files which can be used to upgrade a MySQL/MariaDB database instance to a later version while retaining contained data.

The current schema version is 1.3, which adds the SellSideInst column to the data table of the previous 1.2 version. This allows the FIX Analyser 2 to uniquely distinguish orders that have the same ClOrdID and BuySideInst

While PostgreSQL currently does not have any schema upgrades (only version is 1.3), MySQL/MariaDB supports schema upgrades through the procedures outlined below.

Attended Scripted Upgrade (MySQL/MariaDB) Copied

A schema installation / upgrade script is provided in the resources/db-schema folder of the FIX Analyser 2 package. The script should be run from within this folder and will require database administrator (DBA) access to the database. If remote DBA access has been disabled, then the script will need to be executed on the database server host itself (or remote access granted).

To execute the script to upgrade an existing FIX Analyser database:

1. Run the schema upgrade script:

cd resources/db-schema
./fix_analyser_schema_install.sh mysql -u

2. Provide the required details during the upgrade process:

• Database administrator (DBA) username. This account will perform administrative functions on the database.
• DBA password
• Hostname and port of the MySQL/MariaDB database server
• Name of the database to upgrade. This database must already exist on the server.
• Time for daily housekeeping (default: 00:15)
• Number of days to retain data (default: 3)

Unattended Scripted Upgrade Copied

The upgrade script described above can also be run in unattended (non-interactive) mode.

To perform an unattended upgrade, set the following environment variables and run the upgrade script:

Manual Upgrade (MySQL/MariaDB) Copied

If you are unable or do not wish to install the database schema using the installation script, you can install the schema manually using the following steps. The SQL statements below should be executed as a database administrator (DBA) account on the MySQL/MariaDB database server.

1. Connect to the database.

   mysql -u dba_username -p dba_password [ -h db_host -P db_port ] FIX_DB
   

2. Enable the MySQL/MariaDB event scheduler. Note, you should also enable the scheduler permanently as detailed in the server installation section of this document.

   SET GLOBAL event_scheduler = ON;
   SHOW PROCESSLIST;
   

   The output of the SHOW PROCESSLIST command should return a row with the text event_scheduler if successful.

3. Verify the current schema version.

   SELECT * FROM version;
   

4. Upgrade the schema by running the appropriate SQL files:

   mysql connection_parameters FIX_DB < sql_file
   

   The exact files to use will depend on the version of database being upgraded. For a version 1.0 schema, first upgrade to version 1.1 using the SQL file:

   mysql-fix-analyser-v1.0-v1.1-schema-upgrade.sql
   

   Then, upgrade from version 1.1. to version 1.2 using the SQL file:

   mysql-fix-analyser-v1.1-v1.2-schema-upgrade.sql
   

   Then, upgrade from version 1.2 to version 1.3 using the SQL file:

   mysql-fix-analyser-v1.2-v1.3-schema-upgrade.sql
   

5. Schedule the housekeeping event. In the following SQL, HH:MM is the daily execution time of the event and DAYS the number of days of message data to retain.

The following configurations are recommended for the MySQL message database housekeeping:

• Schedule it at a time of day when no FIX sessions are operating, or when no messages are sent. The installation script uses the default values of 00:15 and 3 for the run time of the daily data housekeeping and the number of days to retain message data, respectively.
• Schedule it on the same day as the resetTime setting.

CREATE EVENT FIX_DB_housekeeping
ON SCHEDULE
EVERY 1 DAY
STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE)
DO
CALL FIX_DB.removeExpiredFiles(DAYS);