How to monitor if log files are updated with specific patterns?
Application log monitoring is a wide topic and can have various scenarios depending on the application behaviour. This article provides some examples making use of FKM and StateTracker plugins. Sometimes there can be multiple ways to perform similar tasks in Geneos. Here we aim to inspire users to adapt Geneos configurations as needed.
Case 1: An application which process incoming data generates a new log file per job. The logs should contain job started and then job completed after 5 minutes.
Solution: The FKM plugin may be considered in this case. We can configure FKM Tables to monitor a pattern job started, and with the pattern job completed as Clear Key.
We also need to modify some options in the FKM Advanced tab. We will enable the column Seconds Since Last Trigger, so it will show a timer if job started appears. And in order to show all filenames matching the file wildcard, the Wildcard Monitor All Matches option is turned on.
Then a Gateway rule can be created against the secondsSinceLastTrigger column, to compare if it is over 300 (sec). The following xml is for the FKM sampler configuration:
<sampler name="Case1_FKM_job_completed_clear">
<plugin>
<fkm>
<display>
<columns>
<column>status</column>
<column>permissions</column>
<column>lastModificationTime</column>
<column>fileSize</column>
<column>updateRate</column>
<column>triggerDetails</column>
<column>fileOwner</column>
<column>filename</column>
<column>secondsSinceLastTrigger</column>
</columns>
</display>
<wildcardMonitorAllMatches>
<data>true</data>
</wildcardMonitorAllMatches>
<files>
<file>
<source>
<filename>
<data>/tmp/case1/abc/app_*.log</data>
</filename>
</source>
<tables>
<table>
<severity>warning</severity>
<keyTable>
<data>
<keys>
<key>
<setKey>
<match>
<searchString>
<data>job started</data>
</searchString>
<rules>REGEXP</rules>
</match>
</setKey>
<clearKey>
<match>
<searchString>
<data>job completed</data>
</searchString>
<rules>REGEXP</rules>
</match>
</clearKey>
</key>
</keys>
</data>
</keyTable>
</table>
</tables>
</file>
</files>
</fkm>
</plugin>
</sampler>
Case 2: An application performs housekeeping tasks to get ready for market open Monday to Friday. The log file should contain the string housekeeping finished before 8 am in the morning.
Solution:
We can make use of StateTracker plugin with the Timeout option inside. In the configuration, we can detect the housekeeping finished message to go into Finished status, and if no message is detected until 08:00:00, it will go into Timed Out status.
And in the sampler’s Advanced tab, we can configure Active Time such that the sampler is only active on trading days.
The following xml is for the StateTracker configuration:
<sampler name="Case2_StateTracker_housekeeping_finished">
<plugin>
<stateTracker>
<trackerGroup name="TG1">
<trackers>
<tracker name="Housekeep">
<filename>
<data>/tmp/case2/def/app_<today>.log</data>
</filename>
<rewind>
<data>false</data>
</rewind>
<transitionStates>
<defaultState name="Start Of Day">
<timeout>
<atTimeOfDay>08:00:00</atTimeOfDay>
<state ref="Timed Out"/>
</timeout>
</defaultState>
<state name="Finished">
<keys>
<key>
<regularExpression>
<data>
<regex>housekeeping finished</regex>
</data>
</regularExpression>
</key>
</keys>
</state>
<state name="Timed Out"/>
</transitionStates>
</tracker>
</trackers>
</trackerGroup>
</stateTracker>
</plugin>
<activeTimes>
<activeTime>
<activeTime ref="Market Open Days"/>
</activeTime>
</activeTimes>
</sampler>
Case 3: An application routes trades to a remote system, each trade is recorded with an unique transaction ID in the log file. We would like to monitor the log that for each transaction ID, there should be a corresponding acknowledgement message within 60 seconds.
Solution: We will try to configure the StateTracker plugin for this requirement. In order to allow the capture of log pattern together with the transaction ID, the Template checkbox should be selected.
When a new trade is received, we use a Relative Timeout of 60 seconds to monitor the time. If the acknowledgement message is received, we use the Remove Tracker option to clear the corresponding trade.
The resulting Active Console will then show transactions under processing and those that have timed out.
The following xml is for the StateTracker configuration:
<sampler name="Case3_StateTracker_transaction_id">
<plugin>
<stateTracker>
<trackerGroup name="TG3">
<columns>
<column>
<data>Transaction ID</data>
</column>
</columns>
<trackers>
<tracker name="Trade Server">
<filename>
<data>/tmp/case3/ghi/app_<today>.log</data>
</filename>
<rewind>
<data>false</data>
</rewind>
<template>
<data>true</data>
</template>
<transitionStates>
<defaultState name="Processing">
<keys>
<key>
<regularExpression>
<data>
<regex>received trade with ID: (?<id>T[0-9]+), processing</regex>
</data>
</regularExpression>
</key>
</keys>
<timeout>
<relative>
<interval>60</interval>
<units>seconds</units>
</relative>
<state ref="Timed Out"/>
</timeout>
</defaultState>
<state name="Acknowledged">
<keys>
<key>
<regularExpression>
<data>
<regex>received acknowledgement with ID: (?<id>T[0-9]+) successfully</regex>
</data>
</regularExpression>
</key>
</keys>
<templateOptions>
<removeTracker>
<data>true</data>
</removeTracker>
</templateOptions>
</state>
<state name="Timed Out"/>
</transitionStates>
</tracker>
</trackers>
</trackerGroup>
</stateTracker>
</plugin>
</sampler>