File Transfer Monitor (FTM)

Overview Copied

The universal File Transfer Monitor or FTM plug-in monitors the arrival of files. This plug-in mainly checks the file existence and its timestamp against the time the file is expected to be available.

FTM is typically employed where an Exchange File Transfer Protocol (FTPs) is used to send the previous day’s trading reports to a client. These reports are expected to arrive at predefined times so that these can be automatically picked up by back office systems for further processing.

The plugin is available in all platforms supported by the Netprobe as it is file-based monitoring. To view the FTM plugin configuration and settings, see FTM configuration.

Intended audience Copied

This guide is intended for experienced Geneos users who want to configure the FTM plugin to monitor the arrival of files, and for users who have completed the recommended Geneos course training.

As a user, you should be familiar with Unix and Windows file system structure and permissions, log files, and scripting operations.

Prerequisites Copied

The following requirements must be met prior to the configuration and setup of the FTM plug-in:

• You must use the supported versions of Active Console, Gateway, and Netprobe. For guidance, see Supported Geneos versions in 6.x Compatibility Matrix.
• You must have the servers of the Active Console, Gateway, and Netprobe running on the supported operating systems. For guidance, see Operating system support in 6.x Compatibility Matrix.
• You must connect your Active Console to the Gateway where this plugin is configured.
• The Netprobe user must have a read-access permission to the monitored file.

Introduction Copied

The File Transfer Monitor plug-in works by dividing time into monitoring periods. The other features of the plug-in include:

• Configurable monitoring periods (noon-to-noon or midnight-to-midnight).
• Selecting specific monitoring days when the plugin is only expected to check for files.
• Defining holidays when the plug-in would skip monitoring even if the date falls on a monitoring day.
• Supporting several timestamp formats, and time zone offsets.
• Monitoring delays in the system.

By default, a monitoring period starts at 12:00 noon and finishes at 11:59:59 the following day. FTM is aware of the current monitoring period and any files that exist, are ignored if their timestamp is outside of the monitoring period. This ensures the FTM monitors for the most recent file arrivals and does not confuse them with the older files that may be there from the previous monitoring period.

Note

The date and time used by the FTM plugin are always in the timezone of the Netprobe running the sampler.

By default, all dates and times are displayed in a standard FTM format that is always in the local time of the Netprobe. The date and time format can be changed to ISO8601 format, which includes the local timezone offset. For more information, see displayTimeInIso8601Format in configuration.

If a report is late, but before the ‘drop dead’ time, it can be flagged as an amber or a warning condition. If the report is not available after the ‘drop dead’ time, it can be flagged as a critical red condition. Any warning can be set to trigger an action, depending on the business requirement.

Create a sampler in GSE Copied

Ensure that you have read and can follow the prerequisites prior to installation and setup of this plug-in.

To setup and configure the FTM plug-in, follow these steps:

1. Open Active Console.
2. Double-click the running Gateway to open Gateway Setup Editor.
3. In Gateway Setup Editor, click Samplers in the Navigation panel.
4. Click New Sampler.
5. In the Samplers section, select FTM on the Plugin menu.
6. Click Save to apply changes.

Here are the fields that you can configure on the FTM tab:

Monitor the path of a file in FTM Copied

Now that you have created an FTM sampler, you can start configuring its basic settings to monitor a file. Creating a new sampler for each monitored log file is not required. One managed entity group can hold multiple samplers, and one sampler can monitor multiple logs files. Once completed, the newly created FTM plug-in displays in the Samplers section.

Note

The Netprobe user ID must have a read-access to read the source file.

1. Open the Basic tab of the FTM sampler in Gateway Setup Editor.
2. Enter the path location of the file you want to monitor in the Files > Path field. The Path field specifies the location of the file and must be completed for basic monitoring. The usual format is /usr/data/FileX.dat.

Note

If you wish to monitor additional file locations, then you can add them in the Files > File > Additional paths. For more information about this field, see files > file > additionalPaths.

5. Check the Active Console dataview if it displays the configuration details of the FTM plug-in.

If the FTM dataview does not show any results, these are the possible errors:

• File does not exist.
• File is inaccessible or no read-access from Netprobe.

To view the other options that you can configure, see FTM Plug-In: Technical Reference.

Configure FTM plug-in Copied

This sections provides other use cases and scenarios that you can configure in the FTM plug-in.

Check test file detection Copied

Start testing the files you want to monitor using the sample configuration use case presented below. This allows you to test the existence of the files and view them in the dataview with the expected arrival time.

For example purposes, create the identified paths in the table. Change the paths accordingly based on your own path location. In Windows, specify the path as C:\test1.txt:

Create a file for each setup in the Netprobe directory:

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. Enter the path location of the file in the Path field.

Note

If you wish to monitor additional file locations, then you can add them in the Files > File > Additional paths. For more information about this field, see files > file > additionalPaths.

5. Click Save to apply the changes.
6. Go to the FTM dataview to see the filenames of each identified path and their arrival time.

Monitor files during weekdays Copied

To monitor the files during a certain period allows users or admins to specify the monitoring time, based on business needs and requirements. If the business mandates a 24/7 monitoring operation, the FTM plug-in can be modified to extend its monitoring scope to include the weekend period.

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. Select Daily on the Expected period menu.

Monitor files during holidays Copied

The monitoring period lasts 24 hours less 1 second. If the end of the monitoring period falls on a holiday, then the monitoring period is extended until the start of the next monitoring period.

A list of holidays can be defined in the following ways:

• As a date with DD-Month-[YYYY] format where

  • DD — day of the month. The leading 0 in the date is optional. For example, 1 and 01 are accepted.
  • Month — short or full name of the month. For example, Jan and January are accepted.
  • YYYY — year. When the year is not specified, the FTM plugin will automatically apply the holiday period every year.

• As a day such as monday, tuesday, wednesday, thursday, friday, saturday, and sunday.

Note

If the expected arrival time set is earlier than the current Netprobe machine time, and if the expected period is set on the same day, the expected time in the dataview will fall on the same day with delay.

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. Open the Advanced tab.
3. Do one of the following:

• If you wish to set a specific holiday date, click Add new in the Holidays > Date field.

Enter the holiday date in the Holiday field. When using the DD-Month format, the name of the month is shortened to its first three characters. When the year is not specified, the system defaults the date to every year.

• If you wish to set a day or days as holiday, click the Holidays > Day field. This opens a dropdown list.

4. Click Save to apply the changes.

Remove the leading zero in the day of the month Copied

This is a common requirement to monitor a file through FTM with a date format dd-mm-yy, where as dd does not have a leading zero.

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. In the Files > Path field, use the ddmmyy format where dd does not have a leading zero when the day of the month is between 1-9.

For example, March 2, 2016 will translate into 2032016. When the <today%d%m%y>.txt is added in the Path field, the %d is expecting a leading zero if digit is between 1-9:

8. To remove the leading zero in the date format for Linux users, use this syntax: <today%-d%m%y>.

Restrict the user from reading the file monitored by FTM Copied

Configure the Authentication roles in the Gateway to limit user access to monitored files.

1. In Gateway Setup Editor, click Roles under the Authentication section.
2. Click New Role in the Create a new item section.
3. In the FTM metric dataview, add a specific permission that pertains to the View File and Accept File Delay commands.
4. Enter the role name in the Name field. The roles section represents a list of roles, and each role must be uniquely named. The Administrators role is built-in and any users with this role receive full permission by default.
5. Go to the Role > Administrators > Basic tab > Permissions.
6. Click Add new in the Permissions field.
7. Select Command on the Permissions menu. In the Permissions section, the Command option specifies a single user permission entry. Permission definition must be one of command, data, sections, or setup type. This is a mandatory field.
8. Click Add new in the Users field.
9. Enter the reference name you want to assign to this role in the Users > Ref field.

10. Click Save to apply the changes.

Note

You can use this with your other existing roles, but ensure that your current combineMode setting for Authentication is set properly as it can either inherit the highest possible permissions or the lowest. Before saving the changes, ensure that you have admin rights so you can reactivate your access. For more details on authentication, see Gateway Authentication.

Check file existence on the first x-day of every month Copied

FTM can be set up to monitor file existence on a specific day(s) within a week. If you want to monitor the first x-day of a month, it requires more checking. In this scenario, the FTM plug-in monitors if a file exists on the first Sunday of the month.

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. Open the Advanced tab.
3. On the Monitored days menu, select the specific day you want to monitor:

4. Set a rule to avoid it to alert if today’s date is greater than seven (7), every after first week of the month.
5. Create a new rule in the Rules section in GSE.
6. In the Rule > Block field, enter the sample rule:

ml" class="code">if printDate("%d", now()) > 7 then severity undefined

7. Click Save to apply the changes.

Use wildcards when you enter filenames Copied

FTM allows you to use wildcards in the file names configuration. This can be useful where you have several filenames with the same extension, for example .log, and you do not want to enter each file name individually into the sampler.

Caution

Wildcards are not supported for any paths when files > file > additionalPaths is configured.

1. In Gateway Setup Editor, open the FTM sampler you have created.
2. On the Basic tab, click Add new.

Enter the full path name in the newly added Path field. In the example below, an asterisk (*) is added at the end of each filename:

3. Click Save to apply the changes.
4. Go to the Active Console dataview to view the monitored files.

Note

Using the wildcard by adding an asterisk (*) before the log extension name only shows the latest modified file in the directory matching the wildcard expression on the set expected arrival time and log. Alternatively, you may use another wildcard by adding a question mark (?) to search for a specific character.

Use date functions in path Copied

The date keyword, such as <previous_monitored_day> and <next_monitored_day>, allow you to define them as part of the path’s filename.

You can use multiple formatting codes and date keyword expansions in a path’s filename. To view the other extended date keyword expansions, see files > file > path in FTM Plug-In: Technical Reference.

This results in the automatic substitution of the date in yyyy-mm-dd format.

To set this in FTM sampler, follow these steps:

1. Go to Gateway Setup Editor**> Sampler > File Transfer Monitor (FTM)**.
2. Go to Basic > Path.
3. Enter the filename in the Path field using the date keyword.

Note

If you wish to monitor additional file locations, then you can add them in the Files > Additional paths. For more information about this field, see files > file > additionalPaths.

4. Set the Expected arrival time.
5. Set the monitored days on the Advanced tab.
6. Click Save to apply your changes.
7. Go to Active Console dataview to see the previous monitored date result.

In this example, the FTM sampler uses the following:

• path is /home/scripts/ftm/<previous_monitored_day%Y/%m/%d>/ftm_<previous_monitored_day%Y%m%d>.zip.
• expected arrival time of 07:00.
• the monitored days from Monday to Friday.

This configuration ran on 2022/06/28 (Monday), and therefore the <previous_monitored_day> keyword shows the 2022/06/27 (Friday) date in the filename.

Further reading Copied

To know more about the FTM plugin configuration and settings, see FTM configuration.