FTM Plugin: Technical Reference

Plugin Configuration

Geneos Universal FTM Plugin allows for monitoring arrival times of files.

It is typically employed where the Exchange FTPs the previous day's trading reports to the client. These reports are expected to arrive at pre-defined times so that they can be automatically picked up by the back office systems for further processing.

If a report is late, but before the 'drop dead' time, it can be flagged as an amber warning condition. If the report is not available after the 'drop dead' time, this can be flagged as a critical RED condition.

For example: consider a report file that is expected to be FTPd from the Exchange at 05:00 with a back office system programmed to process it at 05:30. If the report arrives before 05:00, the FTM will show GREEN status, if the report arrives between 05:00 and 05:29:59, the status is AMBER, and if the report has not arrived or arrived after the 05:29:59 the status is RED.

FTM works by dividing the time into monitoring periods. By default, a monitoring period starts at 12:00 noon and finishes 11:59:59 the following day. FTM is aware of the current monitoring period and any files that exist, are ignored if their timestamp is outside of the monitoring period. This ensures that FTM monitors for 'fresh' file arrivals and does not confuse them with 'older' files that may be there from the previous monitoring period.

It's possible to switch the monitoring period between "noon to noon" (as described above) and "midnight to midnight".

Note: Monitoring periods for Saturday, Sunday and additional defined holiday dates are ignored only for daily expected period. See expectedPeriod for more details on expected periods.

View

The FTM plugin can be used to monitor arrival times of any file, for example the log file of a batch process that must be run at a specific time.

Headline Legend

Name	Description
totalFilesExpected	Number of files configured.
filesDelayed	Number of files that have not arrived yet and are past their due arrival time.
lateArrivals	Number of files that have arrived, but later than expected, i.e. where the Delay time is greater than zero.
totalReceived	Number of files received.

Table Legend

Name	Description
Filename	Name of the file or its alias.
expectedTime	Time the file is expected.
arrivalTime	Time the file has arrived.
Delay	Number of seconds the file is delayed by, counted from the expected time. If the file arrives before its expected time, the delay is set to zero.
currentFileSize	Current size of the file that was transferred. This can be used to detect files that have not transferred properly.
actualFilename	Actual file being monitored that is only visible if showActualFilename is set.
periodStart	Start of the current monitoring period.
periodEnd	End of the current monitoring period.

Plugin Configuration

The following parameters can be configured for this plugin:

files

Allows the configuration of files.

Mandatory: Yes

files > file > path

The full path to the file.

Example:

/usr/data/FileX.dat

If <today> (including the < > brackets) is specified as part of the file name, it will result in the automatic substitution of the date in yyyymmdd format.

Additionally, you can include one or more formatting codes to format the date - <today%d%m%Y>.

Also, in place of the <today> keyword, you can use these keywords when specifying the filename:

Date keyword	Description
`<today...>`	Current date.
`<today+x...>`	Current date + x actual days. This does not include the monitored days and holidays.
`<today-x...>`	Current date - x actual days. This does not include the monitored days and holidays.
`<yesterday...>`	Actual date yesterday. If monitored days is not configured in FTM, `Saturdays` and `Sundays` are considered holiday. Therefore, if today is `Monday`, the `<yesterday>` tag is the date of the previous `Friday`.
`<tomorrow...>`	Actual date tomorrow.
`<previous_monitored_day...>`	Previous day with respect to holidays and monitored days.
`<previous_monitored_day-x...>`	Previous monitored day minus the previous x number of monitored days. This includes the holidays and monitored days.
`<next_monitored_day...>`	Next day with respect to holidays and monitored days.
`<next_monitored_day+x...>`	Next monitored day plus the next x number of monitored days. This includes the holidays and monitored days.

Sample path configuration details for FTM plug-in:

Date keyword expansions

When using the +x or -x values into the date keywords to offset the date on the filename, note the following behaviours when introducing spaces (depicted as the ␣ character) in formatting:

Date keyword expansion format	Behaviour and outcome
`<today␣>`	Ignores the space and uses the default `%Y%m%d` format. If the date today is 30 June 2018, then the outcome is: `20180630.dat`
`<today␣%Y␣%m␣%d␣>`	Ignores the spaces around the date format, but includes the spaces in-between. If the date today is 30 June 2018, then the outcome is: `2018 06 30.dat`
`<today␣+␣10␣>`	Ignores the spaces and uses the default `%Y%m%d` format. This format also offsets by the value of the number indicated. If the date today is 30 June 2018, then the outcome is: `20180710.dat`
`<today␣+␣1␣0␣>`	Offsets by the value of the first digit, and interprets the subsequent digit as the format string. As a consequence, this format renders the offset invisible, even though the offset is in effect due to the first digit. Regardless of the date, the outcome is: `0.dat`
`<today␣+␣1␣0␣%Y-%m-%d␣>`	Offsets by the value of the first digit, and interprets the subsequent digit as part of the `%Y-%m-%d` format. If the date today is 30 June 2018, then the outcome is: `0 2018-07-01.dat`

The limitations for offsets to today, previous_monitored_day-x, and next_monitored_day+x path keywords are set to:

Lower bounds: -3650
Upper bounds: 3650

The invalid offsets get an expanded value equivalent to the base tag. If you have <previous_monitored_day-3651>, then the date becomes <previous_monitored_day>.

Formatting code	Description
%a	Abbreviated weekday name.
%A	Full weekday name.
%b	Abbreviated month name.
%^b	Abbreviated month name (but capitalised).
%B	Full month name.
%d	Day of month as decimal number (01 - 31).
%j	Day of year as decimal number (001 - 366).
%m	Month as decimal number (01 - 12).
%U	Week of year as decimal number, with Sunday as first day of week (00 - 53).
%w	Weekday as decimal number (0 - 6; Sunday is 0).
%W	Week of year as decimal number, with Monday as first day of week (00 - 53).
%x	Date representation for current locale.
%y	Year without century, as decimal number (00 - 99).
%Y	Year with century, as decimal number.
%Z	Time-zone name or abbreviation; no characters if time zone is unknown.

Examples:

Configured	Equivalent (if date is 24 September 2003)
filename<today%d%m%Y>.txt	filename24092003.txt
filename<today%d%b%Y>.txt	filename24Sep2003.txt
filename<today%d%^b%Y>.txt	filename24SEP2003.txt
filename<today%d-%b-%Y>.txt	filename24-Sep-2003.txt

Mandatory: Yes

files > file > expectedArrival

The expected time in the HH:MM:SS format.

Mandatory: Yes

files > file > expectedPeriod

Specifies how often the file is transferred.

Possible values:

Setting	Description
daily	File is expected every day.
weekly	File is expected every week. The day on which the file will arrive should also be specified.
monthly	File is expected every month. The date on which the file will arrive should also be specified. Last may be chosen if the file will always arrive on the last day of the month.

Caution: The monitoredDays and the files > file > expectedPeriod settings are not compatible. The behaviour is undefined, if these two settings are used in conjunction.

Mandatory: No

Default: daily

Daily

The following are the expected times depending on the configured day, time, and monitoring period:

Note: The results below assume that the monitoredDays setting is disabled. Therefore, Saturday and Sunday are considered as holiday, and if today is Monday, then the <yesterday> tag is the date of the previous Friday.

If the expected arrival time set is earlier than the current Netprobe machine time and the expected period is set on the same day, then the expected time in the dataview will fall on the same day with delay.

Today	Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Monday	Before Noon	Before Noon	Noon to Noon	Before Noon, Same Day
	Before Noon	After Noon	Noon to Noon	After Noon, Previous Friday
	Before Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	Before Noon	After Noon	Midnight to Midnight	After Noon, Same Day
	After Noon	Before Noon	Noon to Noon	Before Noon, Next Day
	After Noon	After Noon	Noon to Noon	After Noon, Same Day
	After Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	After Noon	After Noon	Midnight to Midnight	After Noon, Same Day
Tuesday, Wednesday, Thursday	Before Noon	Before Noon	Noon to Noon	Before Noon, Same Day
	Before Noon	After Noon	Noon to Noon	After Noon, Previous Day
	Before Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	Before Noon	After Noon	Midnight to Midnight	After Noon, Same Day
	After Noon	Before Noon	Noon to Noon	Before Noon, Next Day
	After Noon	After Noon	Noon to Noon	After Noon, Next Day
	After Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	After Noon	After Noon	Midnight to Midnight	After Noon, Same Day
Friday	Before Noon	Before Noon	Noon to Noon	Before Noon, Same Day
	Before Noon	After Noon	Noon to Noon	After Noon, Previous Day
	Before Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	Before Noon	After Noon	Midnight to Midnight	After Noon, Same Day
	After Noon	Before Noon	Noon to Noon	Before Noon, Next Monday
	After Noon	After Noon	Noon to Noon	After Noon, Same Day
	After Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
	After Noon	After Noon	Midnight to Midnight	After Noon, Same Day
Saturday, Sunday	Before Noon	Before Noon	Noon to Noon	Before Noon, Next Monday
	Before Noon	After Noon	Noon to Noon	After Noon, Previous Friday
	Before Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Friday
	Before Noon	After Noon	Midnight to Midnight	After Noon, Previous Friday
	After Noon	Before Noon	Noon to Noon	Before Noon, Next Monday
	After Noon	After Noon	Noon to Noon	After Noon, Previous Friday
	After Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Friday
	After Noon	After Noon	Midnight to Midnight	After Noon, Previous Friday

Holidays

Note: The results below disregard weekends. For example, if Monday is the monitoring day, "Previous Day" refers to Friday, not Sunday; and if Friday is the monitoring day, "Next Day" refers to Monday, not Saturday.

Scenario 1: The monitoring day and the holiday is set on the same day.

Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Before Noon	Before Noon	Noon to Noon	Before Noon, Next Day
Before Noon	After Noon	Noon to Noon	After Noon, Previous Day
Before Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Day
Before Noon	After Noon	Midnight to Midnight	After Noon, Previous Day
After Noon	Before Noon	Noon to Noon	Before Noon, Next Day
After Noon	After Noon	Noon to Noon	After Noon, Previous Day
After Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Day
After Noon	After Noon	Midnight to Midnight	After Noon, Previous Day

Scenario 2: The monitoring day is a weekday and the holiday is set the next day.

Note: This also applies to consecutive holidays.

Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Before Noon	Before Noon	Noon to Noon	Before Noon, Same Day
Before Noon	After Noon	Noon to Noon	After Noon, Previous Day
Before Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
Before Noon	After Noon	Midnight to Midnight	After Noon, Same Day
After Noon	Before Noon	Noon to Noon	Before Noon, Day/s After Holiday/s
After Noon	After Noon	Noon to Noon	After Noon, Same Day
After Noon	Before Noon	Midnight to Midnight	Before Noon, Same Day
After Noon	After Noon	Midnight to Midnight	After Noon, Same Day

Scenario 3: The monitoring day is on a weekend and the holiday is set on a Monday.

Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Before Noon	Before Noon	Noon to Noon	Before Noon, Next Tuesday
Before Noon	After Noon	Noon to Noon	After Noon, Previous Friday
Before Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Friday
Before Noon	After Noon	Midnight to Midnight	After Noon, Previous Friday
After Noon	Before Noon	Noon to Noon	Before Noon, Next Tuesday
After Noon	After Noon	Noon to Noon	After Noon, Previous Friday
After Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Friday
After Noon	After Noon	Midnight to Midnight	After Noon, Previous Friday

Scenario 4: The monitoring day is on a weekend and the holiday is set on a Friday.

Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Before Noon	Before Noon	Noon to Noon	Before Noon, Next Monday
Before Noon	After Noon	Noon to Noon	After Noon, Previous Thursday
Before Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Thursday
Before Noon	After Noon	Midnight to Midnight	After Noon, Previous Thursday
After Noon	Before Noon	Noon to Noon	Before Noon, Next Monday
After Noon	After Noon	Noon to Noon	After Noon, Previous Thursday
After Noon	Before Noon	Midnight to Midnight	Before Noon, Previous Thursday
After Noon	After Noon	Midnight to Midnight	After Noon, Previous Thursday

Note: If the expected arrival time set is earlier than the current Netprobe machine time and the expected period is set on the same day, the expected time in the dataview will fall on the same day with delay.

Weekly and Monthly

Sunday - Monday

Netprobe Machine Time	FTM Expected Arrival in GSE	Monitoring Period	FTM Expected Time in Dataview
Before Noon	Before Noon	Noon to Noon	Before Noon, Expected Period in GSE
Before Noon	After Noon	Noon to Noon	After Noon, Expected Period in GSE
Before Noon	Before Noon	Midnight to Midnight	Before Noon, Expected Period in GSE
Before Noon	After Noon	Midnight to Midnight	After Noon, Expected Period in GSE
After Noon	Before Noon	Noon to Noon	Before Noon, Expected Period in GSE
After Noon	After Noon	Noon to Noon	After Noon, Expected Period in GSE
After Noon	Before Noon	Midnight to Midnight	Before Noon, Expected Period in GSE
After Noon	After Noon	Midnight to Midnight	After Noon, Expected Period in GSE

Note: For monthly expected period, if the expected period set is earlier than the current Netprobe machine day, the expected time in the dataview will fall on the expected period set, next month.

Monitoring Day	Expected Period	Expected Time in Dataview
December 4	Monthly, Day 1	January 1

files > file > tzOffset

The time-zone offset is used when files have date / time formatting in the filename, but the file is produced in a time-zone which may lead to the file having a different date from the expected.

The parameter specifies the number of hours difference between the time-zone the probe is running in and the time-zone in which the file is produced. For example, if the Netprobe is running in London and the file is produced in New York with a with a time difference of -5 hours, then tzOffset should be set to -5. This way, if the file is produced at 10pm in New York (3am in London), FTM will know to expect a filename with what will be the previous days date.

                      Mandatory: No
                    
                      Units: Hours
                    
                      Default: 0

files > file > monitoringPeriod

Specifies the monitoring period, overriding the monitoring period set at the sampler level.

Mandatory: No

Default: Same as the sampler configuration.

files > file > monitoringPeriod > periodAlias

One of the period aliases for the monitoring period, e.g. NOON_TO_NOON or MIDNIGHT_TO_MIDNIGHT.

Mandatory: No

Default: Same as the sampler configuration.

files > file > monitoringPeriod > periodStartTime

A specific time in HH:MM format for the start of the monitoring period, e.g. 12:00 is the same as specifying NOON_TO_NOON.

Mandatory: No

Default: Same as the sampler configuration.

files > file > alias

A more readable name for the file, to be displayed in the first column of the view.

Mandatory: No

Default: The filename

files > file > activePeriod

An amount in seconds that gets added before and after the expected time of a file to provide a period during which file arrivals should be recognized and responded to and ignored otherwise. This is useful when you are monitoring the arrival of a file with the same name two or more times during the day. If a file arrives after (expectedTime + activePeriod), then none of the displayed metrics should be affected.

Mandatory: No

Default: Nothing

monitoringPeriod

Sets the monitoring period for the plugin.

monitoringPeriod > periodAlias

An alias for the start of the monitoring period.

Possible values:

Setting	Description
NOON_TO_NOON	Monitoring period starts at 12:00pm and ends at 11:59am the following day.
MIDNIGHT_TO_MIDNIGHT	Monitoring period starts at 12:00am and ends at 11:59pm on the same day.

Mandatory: No

Default: NOON_TO_NOON

monitoringPeriod > periodStartTime

A start time for the monitoring period in HH:MM format. A start time of 12:00 is the same as specifying a periodAlias of NOON_TO_NOON.

holidays

Lists holiday dates in DD-Mon[-YYYY] format. If the year is not specified, then the date applies to any year.

Caution: If monitoredDays is set, then any configuration in the holidays settings will not affect the FTM plugin.

Mandatory: No

Default: Saturday and Sunday

monitoredDays

This setting allows FTM to monitor only on certain days of the week. Each day will be monitored for a 24-hour period starting on the time configured in the monitoringPeriod.

The holidays configuration is ignored if this option is used.

Caution: The monitoredDays and the files > file > expectedPeriod settings are not compatible. The behaviour is undefined, if these two settings are used in conjunction.

Mandatory: No

Default: If monitoredDays is not set, the configuration from the files > file > expectedPeriod setting will be applied.

consistentDateStamps

By default, the FTM plugin uses the Netprobe host's date and timestamp as reference for its date and time resolution. For example, the filename function <previous_monitored_day...> uses the Netprobe host's date and timestamp and the value of the monitoredDays setting to determine the filename.

If consistentDateStamps is enabled, the expected filename is based on the date and time at the start of the monitoring period. For example, the filename function <previous_monitored_day...> uses the date and time of the start of the monitoring period and the value of the monitoredDays setting to determine the filename.

Mandatory: No

Default: Disabled

showPreviousDay

Plugin shows a separate view of the data relating to the previous day. (Previous day data is stored in a file by the plugin when the current day ends - at noon or midnight based on the monitoringPeriod setting).

This implements the monitoring period snapshot at the file level, instead of using the sampler level monitoring period.

The periodStart and periodEnd columns are added to the main and previous day dataview. It shows the start and end of the current monitoring period respectively.

Mandatory: No

Default: false

previousDayFile

The full path (including filename) of the file that the plugin should use for writing and reading data. The plugin should have write permissions to this file/location.

Mandatory: No

Default: FTMpredaydata.dat

showActualFilename

Adds an additional column "actualFilename" which shows the actual filename of each file (this is useful when files have aliases specified)

Mandatory: No

Default: false

delayUnit

The unit of time that the "delay" column should be displayed in.

Possible values:

Setting	Description
SECONDS	Displays the delay time in seconds.
MINUTES	Displays the delay time in minutes.
HOURS	Displays the delay time in hours.

Mandatory: No

Default: SECONDS

sizeUnit

The unit of size that the "currentFileSize" column should be displayed in.

Possible values:

Setting	Description
BYTES	Displays the current file size in bytes.
KILOBYTES	Displays the current file size in kilobytes.
MEGABYTES	Displays the current file size in megabytes.
GIGABYTES	Displays the current file size in gigabytes.

Mandatory: No

Default: KILOBYTES

useChangedTime

Determines whether ctime or mtime should be used when a monitored file has been determined to have arrived. If false or not defined, then mtime is used. Else if true, then ctime is used.

Mandatory: No
Default: false