Geneos
Geneos
Recommended Reads

File Transfer Monitor (FTM) Plug-In User Guide

Overview

The universal File Transfer Monitor or FTM plug-in monitors the arrival of files. This plug-in mainly checks the file existence and its timestamp against the time the file is expected to be available.

FTM is typically employed where an Exchange File Transfer Protocol (FTPs) is used to send the previous day’s trading reports to a client. These reports are expected to arrive at predefined times so that these can be automatically picked up by back office systems for further processing.

The plugin is available in all platforms supported by the Netprobe as it is file-based monitoring.

Intended audience

This guide is intended for experienced Geneos users who want to configure the FTM plugin to monitor the arrival of files, and for users who have completed the recommended Geneos course training.

As a user, you should be familiar with Unix and Windows file system structure and permissions, log files, and scripting operations.

Prerequisites

The following requirements must be met prior to the configuration and setup of the FTM plug-in:

Introduction

The File Transfer Monitor plug-in works by dividing time into monitoring periods. The other features of the plug-in include:

  • Configurable monitoring periods (noon-to-noon or midnight-to-midnight).
  • Selecting specific monitoring days when the plugin is only expected to check for files.
  • Defining holidays when the plug-in would skip monitoring even if the date falls on a monitoring day.
  • Supporting several timestamp formats, and time zone offsets.
  • Monitoring delays in the system.

By default, a monitoring period starts at 12:00 noon and finishes at 11:59:59 the following day. FTM is aware of the current monitoring period and any files that exist, are ignored if their timestamp is outside of the monitoring period. This ensures the FTM monitors for the most recent file arrivals and does not confuse them with the older files that may be there from the previous monitoring period.

If a report is late, but before the ‘drop dead’ time, it can be flagged as an amber or a warning condition. If the report is not available after the ‘drop dead’ time, it can be flagged as a critical red condition. Any warning can be set to trigger an action, depending on the business requirement.

Create a sampler in GSE

Ensure that you have read and can follow the prerequisites prior to installation and setup of this plug-in.

To setup and configure the FTM plug-in, follow these steps:

  1. Open Active Console.
  2. Double-click the running Gateway to open Gateway Setup Editor.
  3. In Gateway Setup Editor, click Samplers in the Navigation panel.
  4. Click New Sampler.
  5. In the Samplers section, select FTM on the Plugin menu.
  6. Click Save to apply changes.

Here are the fields that you can configure on the FTM tab:

Field Description
Name

Specifies the sampler name you created in the managed entity.

The name of the sampler must be unique among all other samplers. The first character of your sampler name cannot be a special character.

Mandatory: Yes
Group

Specifies a group name.

Mandatory: No
Description

Specifies a description for your plug-in.

Mandatory: No
Sample interval

Controls the period between two samples taken by the sampler, which is measured in seconds.

The value should be set to a non-negative integer value. A value of 0 indicates that regular sampling should be disabled.

Mandatory: No

Default: 20

Unit: Seconds
Basic tab See FTM Plug-In: Technical Reference.
Advanced tab See FTM Plug-In: Technical Reference.
   

Monitor the path of a file in FTM

Now that you have created an FTM sampler, you can start configuring its basic settings to monitor a file. Creating a new sampler for each monitored log file is not required. One managed entity group can hold multiple samplers, and one sampler can monitor multiple logs files. Once completed, the newly created FTM plug-in displays in the Samplers section.

Note: The Netprobe user ID must have a read-access to read the source file.

  1. Open the Basic tab of the FTM sampler in Gateway Setup Editor.
  2. Enter the path location of the file you want to monitor in the Files > Path field. The Path field specifies the location of the file and must be completed for basic monitoring. The usual format is /usr/data/FileX.dat.
  3. Enter the specific time following the HH:MM:SS format in the Expected arrival field.
  4. Select the target period on how frequent the file should be monitored on the Expected period menu.

  5. Check the Active Console dataview if it displays the configuration details of the FTM plug-in.

If the FTM dataview does not show any results, these are the possible errors:

  • File does not exist.
  • File is inaccessible or no read-access from Netprobe.

To view the other options that you can configure, see FTM Plug-In: Technical Reference.

Configure FTM plug-in

This sections provides other use cases and scenarios that you can configure in the FTM plug-in.

Check test file detection

Start testing the files you want to monitor using the sample configuration use case presented below. This allows you to test the existence of the files and view them in the dataview with the expected arrival time.

For example purposes, create the identified paths in the table. Change the paths accordingly based on your own path location. In Windows, specify the path as C:\test1.txt:

Path filename Expected result

/export/home/test1.txt

The file is detected with the correct arrival time and size.

/export/home/test2-<today>.txt

 

Create file test2-YYYYmmdd.txt

Where “YYYY” is the year, “mm” is the month and “dd” is the day of the month

The file is detected with the correct arrival time and size.

/export/home/test3-<today-%Y-%m-%d>.txt

Create file test3-YYYY-mm-dd.txt Where “YYYY” is the year, “mm” is the month and “dd” is the day of the month

The file is detected with the correct arrival time and size.

Create a file for each setup in the Netprobe directory:

Current path filename Change to
/export/home/test1.txt test_<today>.txt

/export/home/test2-<today>.txt

 

Create file test2-YYYYmmdd.txt

Where “YYYY” is the year, “mm” is the month and “dd” is the day of the month

 test_<yesterday>.txt

/export/home/test3-<today-%Y-%m-%d>.txt

Create file test3-YYYY-mm-dd.txt Where “YYYY” is the year, “mm” is the month and “dd” is the day of the month

 test_<tomorrow>.txt

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. Enter the path location of the file in the Path field.
  3. Click Add new to add the remaining path filenames.
  4. Enter the expected time in the Expected arrival field.

  5. Click Save to apply the changes.
  6. Go to the FTM dataview to see the filenames of each identified path and their arrival time. 

Monitor files during weekdays

To monitor the files during a certain period allows users or admins to specify the monitoring time, based on business needs and requirements. If the business mandates a 24/7 monitoring operation, the FTM plug-in can be modified to extend its monitoring scope to include the weekend period.

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. Select Daily on the Expected period menu.

    3. Values Description
    Daily The file is expected every day.
    Weekly The file is expected every week. The day on which the file will arrive should be specified.
    Monthly The file is expected every month. The date on which the file will arrive should be specified. Last may be chosen if the file will always arrive on the last day of the month.

  3. Select the period on the Monitoring period > periodAlias menu.

    4. Period alias Description
    Noon_to_noon The monitoring period starts at 12:00 PM and ends at 11:59 AM the following day.
    Midnight_to_midnight The monitoring period starts at 12:00 AM and ends at 11:59 PM on the same day.

  4. Open the Advanced tab.
  5. Select the specific days you want to monitor on the Monitored days menu. By default, the monitoring length is from Monday to Friday if not manually specified on the Advanced tab.

Monitor files during holidays

The monitoring periods for defined holiday dates are ignored for the daily expected period. A list of holiday dates can be defined using the “DD-Month” format. When the year is not specified, it will automatically apply the holiday period every year. There are multiple scenarios to be considered when monitoring a file during holiday:

  • The monitoring day and the holiday is set on the same day.
  • The monitoring day is a weekday and the holiday is set the next day.
  • The monitoring day is on a weekend and the holiday is set on a Monday.
  • The monitoring day is on a weekend and the holiday is set on a Friday.

Note: If the expected arrival time set is earlier than the current Netprobe machine time, and if the expected period is set on the same day, the expected time in the dataview will fall on the same day with delay.

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. Open the Advanced tab.
  3. Click Add new in the Holidays field. This opens the field for holiday.
  4. Enter the holiday date in the Holiday field. When using the DD-Month format, the name of the month is shortened to its first three characters. When the year is not specified, the system defaults the date to every year. For example, enter 31-AUG in the Holiday field.
  5. Click Save to apply the changes.

Remove the leading zero in the day of the month

This is a common requirement to monitor a file through FTM with a date format dd-mm-yy, where as dd does not have a leading zero.

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. In the Files > Path field, use the ddmmyy format where dd does not have a leading zero when the day of the month is between 1-9.

    3. For example, March 2, 2016 will translate into 2032016. When the <today%d%m%y>.txt is added in the Path field, the %d is expecting a leading zero if digit is between 1-9:

  3. To remove the leading zero in the date format for Linux users, use this syntax: <today%-d%m%y>.

Restrict the user from reading the file monitored by FTM

Configure the Authentication roles in the Gateway to limit user access to monitored files.

  1. In Gateway Setup Editor, click Roles under the Authentication section.
  2. Click New Role in the Create a new item section.
  3. In the FTM metric dataview, add a specific permission that pertains to the View File and Accept File Delay commands.
  4. Enter the role name in the Name field. The roles section represents a list of roles, and each role must be uniquely named. The Administrators role is built-in and any users with this role receive full permission by default.
  5. Go to the Role > Administrators > Basic tab > Permissions.
  6. Click Add new in the Permissions field.
  7. Select Command on the Permissions menu. In the Permissions section, the Command option specifies a single user permission entry. Permission definition must be one of command, data, sections, or setup type. This is a mandatory field.
  8. Click Add new in the Users field.
  9. Enter the reference name you want to assign to this role in the Users > Ref field.

  10. Click Save to apply the changes.

Note: You can use this with your other existing roles, but ensure that your current combineMode setting for Authentication is set properly as it can either inherit the highest possible permissions or the lowest. Before saving the changes, ensure that you have admin rights so you can reactivate your access. For more details on authentication, see Gateway Authentication.

Check file existence on the first x-day of every month

FTM can be set up to monitor file existence on a specific day(s) within a week. If you want to monitor the first x-day of a month, it requires more checking. In this scenario, the FTM plug-in monitors if a file exists on the first Sunday of the month.

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. Open the Advanced tab.
  3. On the Monitored days menu, select the specific day you want to monitor:

  4. Set a rule to avoid it to alert if today’s date is greater than seven (7), every after first week of the month.
  5. Create a new rule in the Rules section in GSE.
  6. In the Rule > Block field, enter the sample rule:

    7. if printDate("%d", now()) > 7 then severity undefined

  7. Click Save to apply the changes.

Use wildcards when you enter filenames

FTM allows you to use wildcards in the file names configuration. This can be useful where you have several filenames with the same extension, for example .log, and you do not want to enter each file name individually into the sampler.

  1. In Gateway Setup Editor, open the FTM sampler you have created.
  2. On the Basic tab, click Add new.
  3. Enter the full path name in the newly added Path field. In the example below, an asterisk (*) is added at the end of each filename:

    4.  

  4. Click Save to apply the changes.

Note: Using the wildcard by adding an asterisk (*) before the log extension name only shows the latest modified file in the directory matching the wildcard expression on the set expected arrival time and log. Alternatively, you may use another wildcard by adding a question mark (?) to search for a specific character.

Use date functions in path

The date keyword, such as <previous_monitored_day> and <next_monitored_day>, allow you to define them as part of the path's filename.

You can use multiple formatting codes and date keyword expansions in a path's filename. To view the other extended date keyword expansions, see FTM Plugin: Technical Reference in FTM Plug-In: Technical Reference.

This results in the automatic substitution of the date in yyyy-mm-dd format.

To set this in FTM sampler, follow these steps:

  1. Go to Gateway Setup Editor> Sampler > File Transfer Monitor (FTM).
  2. Go to Basic > Path.
  3. Enter the filename in the Path field using the date keyword.
  4. Set the Expected arrival time.
  5. Set the monitored days on the Advanced tab.
  6. Click Save to apply your changes.
  7. Go to Active Console dataview to see the previous monitored date result.

In this example, the FTM sampler uses the following:

  • path /home/scripts/ftm/<previous_monitored_day%Y/%m/%d>/ftm_<previous_monitored_day%Y%m%d>.zip.
  • expected arrival time of 07:00.
  • the monitored days from Monday to Friday.

This configuration ran on 2019/02/18 (Monday), and therefore the <previous_monitored_day> keyword shows the 2019/02/15 (Friday) date in the filename.