FIX log ingestion
The following options for FIX log data ingestion are available.
Geneos FIX Analyser to Obcerv publishing Copied
You can seamlessly send all FIX messages captured by the FIX Analyser 2 plugin in Geneos directly to Obcerv. When you enable the publishToObcerv
setting in the FIX Analyser 2 plugin, all FIX messages will be published to Obcerv for consumption by the FIX Monitor app. See publishToObcerv.
Kubernetes log scraping Copied
If your FIX engine application is running in Kubernetes, you can configure Obcerv to stream the log data from your FIX engine application pod(s) into Obcerv.
OpenTelemetry (OTel) Copied
Follow the OpenTelemetry instructions to configure the OTel client on your FIX application. You do not have to modify the source code for you FIX application.
Set the following configuration parameters:
otel.exporter.otlp.endpoint
=https://ingestion.obcerv.local:443 (URL for the ingestion service in Obcerv)otel.exporter.otlp.headers
=otel-client-username
=username,otel-client-password
=password (username and password set on the ingestion service in Obcerv)otel.exporter.otlp.compression
=gzipotel.logs.exporter
=otlpotel.resource.attributes
=service.namespace
=fix,service.name
=sell,service.instance.id
=$$
Fluentd Collector Copied
The Fluentd Forward Protocol can be used to capture log data from your FIX engine(s) and send to a ITRS Fluentd Collection Agent plugin.
The Collection Agent then streams the logs data to the ingestion service in Obcerv.
Follow the two configuration examples below to guide you with the setup process.
Fluentd config Copied
The Fluentd configuration example streams log data from a single log file fix.log
using the <source>
directive, adds a set of dimensions and attributes using the <filter>
directive, and then sends the log data to an ITRS collection agent using the <match>
directive. This is a basic setup, and more complex configurations can be used. See the Fluentd Config Reference for more details.
# Source directive to set FIX log file location, position file (pos_file)
# and tag log events for future matching
<source>
@type tail
<parse>
@type none
</parse>
path ../fix-equities/fix.log
pos_file ./fix-sell-logs.pos
tag fix-equities-log
</source>
# Filter directive for adding dimensions and attributes
<filter fix-equities-log>
@type record_transformer
<record>
service.namespace fix-data
service.name equities
host.name "#{hostname}"
log.name fix-equities-log
level info
</record>
</filter>
# Match directive to stream logs to a Fluentd collection agent plugin
<match fix-equities>
@type forward
send_timeout 1s
<server>
name collection-agent
host localhost
port 24224
</server>
</match>
ITRS Fluentd Collection Agent config Copied
plugin-directory: ./plugins
monitoring:
enabled: true
# Plugin configuration
collectors:
- type: plugin
name: fluentd
class-name: FluentdForwardCollector
# listen port
port: 24224
# message event key combination
name-key: log.name
message-key: message
# dimesion mapping fluentd -> obcerv
dimensions:
service.namespace: service.namespace
service.name: service.name
host.name: host.name
workflow:
store-directory: ./run
metrics:
reporter: obcerv
logs:
reporter: obcerv
events:
reporter: obcerv
attributes:
reporter: obcerv
traces:
reporter: obcerv
# Ingestion service URL, username and password
reporters:
# Send logs to stdout
- type: logging
name: stdout
# Send logs to Obcerv ingestion service
- type: plugin
name: obcerv
class-name: GrpcInternalIngestionServiceReporter
hostname: ingestion-obcerv.itrslab.com
port: 443
username: ingestion-api
password: ingestion
use-compression: true
call-deadline: 3000