File Transfer Monitor configuration
Overview Copied
Geneos Universal FTM plugin allows for monitoring arrival times of files.
It is typically employed where the Exchange FTPs the previous day’s trading reports to the client. These reports are expected to arrive at pre-defined times so that they can be automatically picked up by the back office systems for further processing.
If a report is late, but before the ‘drop dead’ time, it can be flagged as an amber warning condition. If the report is not available after the ‘drop dead’ time, this can be flagged as a red critical condition.
For example, consider a report file that is expected to be FTPd from the Exchange at 05:00 with a back office system programmed to process it at 05:30. If the report arrives before 05:00, the FTM will show green status, if the report arrives between 05:00 and 05:29:59, the status is amber, and if the report has not arrived or arrived after the 05:29:59 the status is red.
FTM works by dividing the time into monitoring periods. By default, a monitoring period starts at 12:00 noon and finishes 11:59:59 the following day. The FTM plugin is aware of the current monitoring period and any files that exist, are ignored if their timestamp is outside of the monitoring period. This ensures that FTM monitors for fresh file arrivals and does not confuse them with older files that may be there from the previous monitoring period.
It is possible to switch the monitoring period between Noon_to_noon
, as described above, and Midnight_to_midnight
.
Note
Monitoring periods for holiday dates and days are ignored only for daily expected period. See files > file > expectedPeriod for more details on expected periods.
View Copied
The FTM plugin can be used to monitor arrival times of any file, for example the log file of a batch process that must be run at a specific time.
Note
The date and time used by the FTM plugin are always in the timezone of the Netprobe running the sampler.
By default, all dates and times are displayed in a standard FTM format that is always in the local time of the Netprobe. The date and time format can be changed to ISO8601 format, which includes the local timezone offset. Please see displayTimeInIso8601Format in configuration.
Headline Legend Copied
Name | Description |
---|---|
totalFilesExpected | Number of files configured. |
filesDelayed | Number of files that have not arrived yet and are past their due arrival time. |
lateArrivals | Number of files that have arrived, but later than expected, i.e. where the Delay time is greater than zero. |
totalReceived | Number of files received. |
Table Legend Copied
Name | Description |
---|---|
Filename |
Name of the file or its alias. The Filename column can only display up to 255 characters. Paths or aliases longer than 255 characters will be truncated, but they can still be detected and monitored. The View File and Accept File Delay commands will still work as expected, even if the displayed path or alias is truncated. Caution: Beginning Windows 10, long paths can be enabled but the FTM plugin may not detect paths longer than 260 characters. For more information on long paths in Windows, see Enable Long Paths in Windows 10, Version 1607, and Later. |
expectedTime | Time the file is expected. |
arrivalTime | Time the file has arrived. |
Delay | Number of seconds the file is delayed by, counted from the expected time. If the file arrives before its expected time, the delay is set to zero. |
currentFileSize | Current size of the file that was transferred. This can be used to detect files that have not transferred properly. |
actualFilename | Actual file being monitored that is only visible if showActualFilename is set. |
periodStart | Start of the current monitoring period. |
periodEnd | End of the current monitoring period. |
Plugin Configuration Copied
The following parameters can be configured for this plugin:
files Copied
Allows the configuration of files.
Mandatory: Yes
files > file > path Copied
The full path to the file.
Example:
/usr/data/FileX.dat
If <today>
, including the < >
brackets, is specified as part of the file name, it will result in the automatic substitution of the date in yyyymmdd
format.
Additionally, you can include one or more formatting codes to format the date - <today%d%m%Y>
.
Also, in place of the <today>
keyword, you can use these keywords when specifying the filename:
Date keyword | Description |
---|---|
<today...>
|
Current date. |
<today+x...>
|
Current date + x actual days. This does not include the monitored days and holidays. |
<today-x...>
|
Current date - x actual days. This does not include the monitored days and holidays. |
<yesterday...>
|
Actual date yesterday with respect to holidays. For example, if today is |
<tomorrow...>
|
Actual date tomorrow. |
<previous_monitored_day...>
|
Previous day with respect to holidays and monitored days. If both holidays and monitoredDays are configured, then only holiday dates are honored while holiday days are ignored. |
<previous_monitored_day-x...>
|
Previous monitored day minus the previous x number of monitored days. This includes holidays and monitored days. If both holidays and monitoredDays are configured, then only the holiday dates are honored but the holiday days are ignored. |
<next_monitored_day...>
|
Next day with respect to holidays and monitored days. If both holidays and monitoredDays are configured, then only the holiday dates are honored but the holiday days are ignored. |
<next_monitored_day+x...>
|
Next monitored day plus the next x number of monitored days. This includes the holidays and monitored days. If both holidays and monitoredDays are configured, then only the holiday dates are honored but the holiday days are ignored. |
Sample path configuration details for FTM plugin:
Date keyword expansions Copied
When using the +x
or -x
values into the date keywords to offset the date on the filename, note the following behaviours when introducing spaces (depicted as the ␣
character) in formatting:
Date keyword expansion format | Behaviour and outcome |
---|---|
<today␣>
|
Ignores the space and uses the default %Y%m%d format. If the date today is 30 June 2018, then the outcome is: 20180630.dat |
<today␣%Y␣%m␣%d␣>
|
Ignores the spaces around the date format, but includes the spaces in-between. If the date today is 30 June 2018, then the outcome is: 2018 06 30.dat |
<today␣+␣10␣>
|
Ignores the spaces and uses the default %Y%m%d format. This format also offsets by the value of the number indicated. If the date today is 30 June 2018, then the outcome is: 20180710.dat |
<today␣+␣1␣0␣>
|
Offsets by the value of the first digit, and interprets the subsequent digit as the format string. As a consequence, this format renders the offset invisible, even though the offset is in effect due to the first digit. Regardless of the date, the outcome is: 0.dat |
<today␣+␣1␣0␣%Y-%m-%d␣>
|
Offsets by the value of the first digit, and interprets the subsequent digit as part of the %Y-%m-%d format. If the date today is 30 June 2018, then the outcome is: 0 2018-07-01.dat |
The limitations for offsets to today
, previous_monitored_day-x
, and next_monitored_day+x
path keywords are set to:
- Lower bounds: -3650
- Upper bounds: 3650
The invalid offsets get an expanded value equivalent to the base tag. If you have <previous_monitored_day-3651>
, then the date becomes <previous_monitored_day>
.
Formatting code | Description |
---|---|
%a | Abbreviated weekday name. |
%A | Full weekday name. |
%b | Abbreviated month name. |
%^b | Abbreviated month name (but capitalised). |
%B | Full month name. |
%d | Day of month as decimal number (01 - 31). |
%j | Day of year as decimal number (001 - 366). |
%m | Month as decimal number (01 - 12). |
%U | Week of year as decimal number, with Sunday as first day of week (00 - 53). |
%w | Weekday as decimal number (0 - 6; Sunday is 0). |
%W | Week of year as decimal number, with Monday as first day of week (00 - 53). |
%x | Date representation for current locale. |
%y | Year without century, as decimal number (00 - 99). |
%Y | Year with century, as decimal number. |
%Z | Time-zone name or abbreviation; no characters if time zone is unknown. |
Examples:
Configured | Equivalent (if date is 24 September 2003) |
---|---|
filename<today%d%m%Y>.txt | filename24092003.txt |
filename<today%d%b%Y>.txt | filename24Sep2003.txt |
filename<today%d%^b%Y>.txt | filename24SEP2003.txt |
filename<today%d-%b-%Y>.txt | filename24-Sep-2003.txt |
Mandatory: Yes
files > file > additionalPaths Copied
Allows the user to configure multiple paths with filename under one file configuration in the FTM plugin. If additional paths are specified, then the FTM plugin will monitor the files in files > file > path and the files in the additional paths with the following behaviours:
- If the FTM plugin finds a file, then the details of that file are displayed in the dataview.
- If the FTM plugin finds one or more files in the list, then the details of the latest modified file are displayed in the corresponding row in the dataview.
- If the FTM plugin finds two or more files with the same modification timestamp, then the details of the first match in the list are displayed in the dataview.
When using additional paths, note the following:
- It is highly recommended to configure files > file > alias to keep the Filename column value constant and the monitoring easier for you. When files > file > alias is set, then enable the showActualFilename setting to display the actualFilename column, which shows the detected file.
- Wildcards are not supported. If any path, either main or additional, has a wildcard character, then the FTM plugin will ignore the particular file set, samplingStatus will display
WARNING
, and logs will include the ignored paths. - Case insensitive filename setting will be ignored, which means that the paths under a file configuration with additional paths will always be processed as case sensitive in Unix platforms.
Limit: Only 20 additional paths can be set.
Mandatory: No
files > file > expectedArrival Copied
The expected time in the HH:MM:SS format.
Mandatory: Yes
files > file > expectedPeriod Copied
Specifies how often the file is transferred.
Possible values:
Setting | Description |
---|---|
daily |
File is expected every day. For more information, see Daily. |
weekly |
File is expected every week. The day on which the file will arrive should also be specified. For more information, see Weekly. |
monthly |
File is expected every month. The date on which the file will arrive should also be specified. Last may be chosen if the file will always arrive on the last day of the month. For more information, see Monthly. Beginning Geneos 5.4.x, you can also configure the file to arrive on the basis of an nth workday of the month. For more information, see Day of month. |
Caution
The monitoredDays and the files > file > expectedPeriod settings are not compatible. The behaviour is undefined, if these two settings are used in conjunction.
Mandatory: No
Default: daily
Daily Copied
The following are the expected times depending on the configured day, time, and monitoring period:
Note
The results below assume that the monitoredDays setting is disabled. Therefore,Saturday
andSunday
are considered as holiday, and if today isMonday
, then the<yesterday>
tag is the date of the previousFriday
.
If the expected arrival time set is earlier than the current Netprobe machine time and the expected period is set on the same day, then the expected time in the dataview will fall on the same day with delay.
Today | Netprobe Machine Time | FTM Expected Arrival in GSE | Monitoring Period | FTM Expected Time in Dataview |
---|---|---|---|---|
Monday | Before Noon | Before Noon | Noon to Noon | Before Noon, Same Day |
Before Noon | After Noon | Noon to Noon | After Noon, Previous Friday | |
Before Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
Before Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
After Noon | Before Noon | Noon to Noon | Before Noon, Next Day | |
After Noon | After Noon | Noon to Noon | After Noon, Same Day | |
After Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
After Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
Tuesday, Wednesday, Thursday | Before Noon | Before Noon | Noon to Noon | Before Noon, Same Day |
Before Noon | After Noon | Noon to Noon | After Noon, Previous Day | |
Before Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
Before Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
After Noon | Before Noon | Noon to Noon | Before Noon, Next Day | |
After Noon | After Noon | Noon to Noon | After Noon, Next Day | |
After Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
After Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
Friday | Before Noon | Before Noon | Noon to Noon | Before Noon, Same Day |
Before Noon | After Noon | Noon to Noon | After Noon, Previous Day | |
Before Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
Before Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
After Noon | Before Noon | Noon to Noon | Before Noon, Next Monday | |
After Noon | After Noon | Noon to Noon | After Noon, Same Day | |
After Noon | Before Noon | Midnight to Midnight | Before Noon, Same Day | |
After Noon | After Noon | Midnight to Midnight | After Noon, Same Day | |
Saturday, Sunday | Before Noon | Before Noon | Noon to Noon | Before Noon, Next Monday |
Before Noon | After Noon | Noon to Noon | After Noon, Previous Friday | |
Before Noon | Before Noon | Midnight to Midnight | Before Noon, Previous Friday | |
Before Noon | After Noon | Midnight to Midnight | After Noon, Previous Friday | |
After Noon | Before Noon | Noon to Noon | Before Noon, Next Monday | |
After Noon | After Noon | Noon to Noon | After Noon, Previous Friday | |
After Noon | Before Noon | Midnight to Midnight | Before Noon, Previous Friday | |
After Noon | After Noon | Midnight to Midnight | After Noon, Previous Friday | |
Weekly Copied
Sunday - Monday
Netprobe Machine Time | FTM Expected Arrival in GSE | Monitoring Period | FTM Expected Time in Dataview |
---|---|---|---|
Before Noon | Before Noon | Noon to Noon | Before Noon, Expected Period in GSE |
Before Noon | After Noon | Noon to Noon | After Noon, Expected Period in GSE |
Before Noon | Before Noon | Midnight to Midnight | Before Noon, Expected Period in GSE |
Before Noon | After Noon | Midnight to Midnight | After Noon, Expected Period in GSE |
After Noon | Before Noon | Noon to Noon | Before Noon, Expected Period in GSE |
After Noon | After Noon | Noon to Noon | After Noon, Expected Period in GSE |
After Noon | Before Noon | Midnight to Midnight | Before Noon, Expected Period in GSE |
After Noon | After Noon | Midnight to Midnight | After Noon, Expected Period in GSE |
Monthly Copied
Note
For monthly expected period, if the expected period set is earlier than the current Netprobe machine day, the expected time in the dataview will fall on the expected period set, next month.
Monitoring Day | Expected Period | Expected Time in Dataview |
---|---|---|
December 4 | Monthly, Day 1 | January 1 |
Day of month Copied
Beginning Geneos 5.4.x, the dayOfMonth
option is available to the FTM sampler.
Option | Description |
---|---|
Count |
Sets the nth day of the month in which you want the file to arrive. If the count value exceeds the possible days of the month defined, then the FTM sampler sets the file to arrive on the last day of the month. Possible values: Default value: |
Reference |
Sets the point of reference for counting the days of the month. Possible values:
If not defined, then the FTM sampler counts from the start of the month. |
Days |
Sets which days of the week to consider as a day of the month. If not defined, then it defaults to all days. |
Example usage Copied
Consider the following calendar month of October 2020:
FTM dayOfMonth setting | Behaviour | Result |
---|---|---|
The FTM sampler considers the working days of the month to be Monday, Tuesday, Wednesday, Thursday, and Friday. The file is expected to arrive on the fifth workday of the month. |
The file arrives on Wednesday, 7 October 2020. | |
The FTM sampler considers the working days of the month to be Monday, Tuesday, Wednesday, Thursday, and Friday. The file is expected to arrive on the third workday from the end of the month. |
The file arrives on Wednesday, 28 October 2020. | |
The FTM sampler considers the working days of the month to only be Friday. The file is expected to arrive on the first Friday of the month. |
The file arrives on Friday, 2 October 2020. | |
The FTM sampler considers the working days of the month to only be Monday. The file is expected to arrive on the last Monday of the month. |
The file arrives on Monday, 26 October 2020. | |
The FTM sampler considers the working days of the month to be Wednesday and Thursday. The file is expected to arrive on the 31st workday of the month. Since this workday exceeds the number of workdays in the month, the FTM sampler considers the last workday of the month, instead. |
The file arrives on Thursday, 29 October 2020. | |
The FTM sampler considers all days of the month. The file is expected to arrive every 10th of the month. |
The file arrives on Saturday, 10 October 2020. | |
files > file > tzOffset Copied
The time-zone offset is used when files have date / time formatting in the filename, but the file is produced in a time-zone which may lead to the file having a different date from the expected.
The parameter specifies the number of hours difference between the time-zone the probe is running in and the time-zone in which the file is produced. For example, if the Netprobe is running in London and the file is produced in New York with a time difference of -5 hours, then tzOffset should be set to -5. This way, if the file is produced at 10pm in New York (3am in London), FTM will know to expect a filename with what will be the previous days date.
Mandatory: No
Units: Hours
Default: 0
files > file > monitoringPeriod Copied
Specifies the monitoring period. If the monitoring period is also set at the sampler level, then it will not be applied to the file.
Mandatory: No
Default: periodAlias
> Noon_to_noon
files > file > monitoringPeriod > periodAlias Copied
Specifies the monitoring period using a period alias. The following are the possible values:
Mandatory: No
Default: Noon_to_noon
Setting | Description |
---|---|
Noon_to_noon
|
Monitoring period starts at 12:00pm and ends at 11:59am the following day. |
Midnight_to_midnight
|
Monitoring period starts at 12:00am and ends at 11:59pm on the same day. |
files > file > monitoringPeriod > periodStartTime Copied
Specifies the monitoring period using a specific time in HH:MM
format. For example, if 12:00
is set as the periodStartTime, then it is the same with the Noon_to_noon
period alias.
Mandatory: No
Default: 12:00
files > file > alias Copied
Specifies a more readable name for the file to be displayed in the first column of the view.
Mandatory: No
Default: Original filename
files > file > activePeriod Copied
An amount in seconds that gets added before and after the expected time of a file to provide a period during which file arrivals should be recognized and responded to and ignored otherwise. This is useful when you are monitoring the arrival of a file with the same name two or more times during the day. If a file arrives after (expectedTime + activePeriod), then none of the displayed metrics should be affected.
Mandatory: No
Default: Nothing
monitoringPeriod Copied
Specifies the monitoring period. If the monitoring period is also set at the file level, then the value in this setting will not be applied to the file.
Mandatory: No
Default: periodAlias
> Noon_to_noon
monitoringPeriod > periodAlias Copied
Specifies the monitoring period using a period alias. The following are the possible values:
Mandatory: No
Default: Noon_to_noon
Setting | Description |
---|---|
Noon_to_noon
|
Monitoring period starts at 12:00pm and ends at 11:59am the following day. |
Midnight_to_midnight
|
Monitoring period starts at 12:00am and ends at 11:59pm on the same day. |
monitoringPeriod > periodStartTime Copied
Specifies the monitoring period using a specific time in HH:MM
format. For example, if 12:00
is set as the periodStartTime, then it is the same with the Noon_to_noon
period alias.
Mandatory: No
Default: 12:00
holidays Copied
Specifies the holidays, which can be defined in the following ways:
- As a date with
DD-Month-[YYYY]
format whereDD
— day of the month. The leading0
in the date is optional. For example,1
and01
are accepted. -Month
— short or full name of the month. For example,Jan
andJanuary
are accepted. -YYYY
— year. When the year is not specified, the FTM plugin will automatically apply the holiday period every year.
- As a day such as
monday
,tuesday
,wednesday
,thursday
,friday
,saturday
, andsunday
.
While in the holiday period, the FTM plugin still monitors delays and file arrivals.
If the next day is set as a holiday, then the monitoring period is extended until the next monitoring period starts. For example:
- If the monitoringPeriod is
Noon_to_noon
andsaturday
andsunday
are set as holidays, then on Friday, the monitoring period will start on Friday 12:00:00 and will end on Monday 11:59:59. The next non-holiday period after this will start on Monday noon. - If the monitoringPeriod is
Midnight_to_midnight
andsaturday
andsunday
are set as holidays, then the monitoring period will starts on Friday 00:00:00 and will end on Monday 23:59:59. The next non-holiday period after this will start on Monday midnight.
The monitoring period typically lasts 24 hours less 1 second. If the end of the monitoring period falls on a holiday, then the monitoring period is extended until the start of the next monitoring period. The FTM plugin always sets the expected arrival date to a non-holiday date within the monitoring period. To further illustrate this, please see examples below:
Caution
If monitoredDays is set, then any configuration in theholidays
settings will not affect the FTM plugin.
Mandatory: No
Default: saturday
and sunday
Monitoring Period | Holidays | Files | Expected values in FTM on a Friday afternoon |
---|---|---|---|
Noon_to_noon
|
saturday and sunday |
File 1 arriving in the morning File 2 arriving in the afternoon |
periodStart: Friday 12:00:00 periodEnd: Monday 11:59:59 expectedTime for File 1: Monday morning expectedTime for File 2: Friday afternoon |
Midnight_to_midnight
|
saturday and sunday |
File 1 arriving in the morning File 2 arriving in the afternoon |
periodStart: Friday 00:00:00 periodEnd: Monday 23:59:59 expectedTime for File 1: Friday morning expectedTime for File 2: Friday afternoon |
monitoredDays Copied
This setting allows FTM to track arriving files and compute delays only on certain days of the week. Each day will be monitored for a 24-hour period starting on the time configured in the monitoringPeriod. On a non-monitored day, all the arriving files are ignored and delays are not computed.
The holidays configuration is ignored if this option is used.
Caution
The monitoredDays and the files > file > expectedPeriod settings are not compatible. The behaviour is undefined, if these two settings are used in conjunction.
Mandatory: No
Default: If monitoredDays is not set, the configuration from the files > file > expectedPeriod setting will be applied.
consistentDateStamps Copied
By default, the FTM plugin uses the Netprobe host’s date and timestamp as reference for its date and time resolution. For example, the filename function <previous_monitored_day...>
uses the Netprobe host’s date and timestamp and the value of the monitoredDays setting to determine the filename.
If consistentDateStamps is set to true
, the expected filename is based on the date and time at the start of the monitoring period. For example, the filename function <previous_monitored_day...>
uses the date and time of the start of the monitoring period and the value of the monitoredDays setting to determine the filename.
Mandatory: No
Default: false
displayTimeInIso8601Format Copied
Allows you to configure the date and time format in the date-time columns of the FTM dataview. If this setting is set to true
, then the dates and times are displayed in ISO 8601 format. If this setting is disabled or set to false
, then the dates and times are displayed in the FTM format.
The following are examples of the FTM and ISO 8601 date and time formats:
Mandatory: No
Default: false
Format | Sample value |
---|---|
FTM format | Wednesday September 30 12:59:59
|
ISO 8601 Format | 2022-09-30T12:59:59+08:00
|
showPreviousDay Copied
If set to true
, then the FTM plugin generates and displays a separate Previous Day dataview. The data displayed in this dataview is stored in a file by the plugin when the current day ends based on the monitoringPeriod setting.
If the files > file > monitoringPeriod and monitoringPeriod are set, then the files > file > monitoringPeriod setting will be followed.
If the monitoredDays setting is configured and today
is an unmonitored day, then the Previous Day dataview will display the file activities from the last monitored day.
Note
The Previous Day dataview is a snapshot of the last sample data in the previous monitoring period. Only the configured settings at the time of the last sampling of the previous monitoring period are applied in the Previous Day dataview. New settings applied in the current monitoring period will not reflect in the Previous Day dataview.
Mandatory: No
Default: false
previousDayFile Copied
The full path of the file, including the filename, that the plugin should use for writing and reading data. The plugin should have a write permission to this location and file.
Mandatory: No
Default: FTMpredaydata.dat
showActualFilename Copied
If set to true
, then an additional actualFilename column showing the actual filename of each file is displayed in the dataview. This is useful when you specify a file alias.
Mandatory: No
Default: false
delayUnit Copied
Specifies the unit of time of the values displayed in the Delay column.
The following are the possible values:
Mandatory: No
Default: Seconds
Setting | Description |
---|---|
Seconds
|
Displays the delay time in seconds. |
Minutes
|
Displays the delay time in minutes. |
Hours
|
Displays the delay time in hours. |
sizeUnit Copied
Specifies the unit of size of the values displayed in the currentFileSize column.
The following are the possible values:
Mandatory: No
Default: Kilobytes
Setting | Description |
---|---|
Bytes
|
Displays the current file size in bytes. |
Kilobytes
|
Displays the current file size in kilobytes. |
Megabytes
|
Displays the current file size in megabytes. |
Gigabytes
|
Displays the current file size in gigabytes. |
useChangedTime Copied
Determines whether ctime
or mtime
should be used when a monitored file arrived. If this setting is set to true
, then ctime
is used. Otherwise, mtime
is used.
Mandatory: No
Default: false