Migrate Gateways to Centralised Configuration

Overview Copied

Gateway Hub can function as a centrally accessible repository for Gateway setup and include files. Gateways in your organisation can then obtain their setup information from Gateway Hub, which simplifies the administration of large estates.

The centralised configuration migration tool allows you to migrate your Gateway configurations to centralised configuration. This tool enables the phased migration of large estates with careful planning and user input. Performing migration with this tool is not a fully automated process.

You can use the centralised configuration migration tool to:

You must validate and migrate each Gateway in your estate individually when setting up centralised configuration.

The ccmigrationtool package can be downloaded separately from ITRS Downloads. This tool is updated regularly, so using the latest available version is recommended.

Prerequisites Copied

The ccmigrate tool is compatible with Gateway versions 5.7.x and newer only.

Additionally, the ccmigrate tool requires that the following dependencies are installed:

Requirement	Versions Supported
Python	3.7 or newer.
requests	2.18.4 or newer.
urllib3	1.22 or newer.
PyYAML	5.1.0 or newer

A full list of requirements for performing installation can be found in ccmigrate/requirements.txt and installed by running pip3 install --user --requirement requirements.txt.

Network connections Copied

The ccmigrate tool must be run from your Gateway host and this host must be able to connect to your Gateway Hub. The ccmigrate tool can use your Gateway installation to download setup files from remote locations if these are accessible over the network.

Centralised configuration migration tool Copied

The ccmigrate tool will check your Gateway configuration against a list of requirements you can specify with a JSON profile file. A minimal defaults.json is provided. When you run ccmigrate, the tool will guide you through providing any information that is not currently listed in your JSON profile file and will update the file accordingly.

You can also use the tool to migrate your Gateways to centralised configuration by specifying the --migrate option.

The script has the following command-line options:

Option	Description
--help	Display a help message.
--version	Display the current version number.
--batch-mode	Run without prompting for user input. This option is useful when automating migration to centralised configuration. If the tool requires user input, it will exit with an error.
--migrate	Perform all checks and then migrate the specified Gateway to the specified Gateway Hub.
--output	Specify an output directory. If no output directory is specified, then the migration tool will use `./output/ccmigrate-<version>-<timestamp>`
--name	Name of the target Gateway.
--profile	Specify a JSON requirements file for this profile.
--edit-mode	Require user input for all options even if they exist in your JSON profile file. When running in edit mode, `ccmigrate` will treat options specified in your JSON profile file as defaults.

Multi-Gateway environments Copied

In large environments with multiple Gateways, you must perform a phased migration. On each Gateway host you should validate and then migrate your configuration.

When migrating a Gateway, all its include files are added to Gateway Hub as resources. Each Gateway Hub resource must have a unique name, and you will be prompted to provide a resource name for each Gateway file uploaded.

If a resource with a matching name already exists in your Gateway Hub, then the ccmigrate tool will report if the local copy matches an old version of the Gateway Hub resource or if the local copy is entirely distinct. You may see the following messages in the log file:

Resource '{res_name}' is in Hub but it has been edited. Please update the version on disk to match the one in Hub
Resource '{res_name}' in Hub does not match the one used by the gateway. Please update the version on disk to match the one in Hub or update the version in Hub to match the one on disk

In both cases, you must update either the local or the Gateway Hub copy of the resource so that they match exactly.

To update Gateway Hub resources use the configuration page from your Web Console.

Updating an existing local resource using the Gateway Setup Editor will cause a failure.

To update a local resource, you must:

Remove the existing resource.
Open your output directory.
The ccmigrate tool downloads a copy of each Gateway Hub resource to your output folder when performing a validation. Add the matching resource from the output folder to your Gateway.

Profiles Copied

A JSON profile file is created when you run the ccmigrate tool for the first time, or you can create one manually. A profile file contains a list of all the requirements you specified when prompted by ccmigrate.

When you run ccmigrate you can use the --profile option to specify an existing profile file. This allows you to run the ccmigrate tool using the same set of requirements on multiple Gateways without repeated user input.

Note the following when using profiles:

You can only reuse a profile file for migrating Gateways to the same Gateway Hub.
If you have mapped an include file to a shared resource, then when reusing a profile file ccmigrate will automatically map any includes with the same file name to that shared resource.

The following is an example of a profile file:

Profile example

{
  "gateways": {
    "Gateway1": {
      "binaries": "/opt/itrs/gateway",
      "local_includes": {
        "local-samplers.xml": "Samplers"
      },
      "aes_file_name": "/opt/itrs/gateway/gateway1/private.aes",
      "setup_file": "/opt/itrs/gateway/gateway1/gateway.setup.xml",
      "working_dir": "/opt/itrs/gateway/gateway1",
      "use_aes_file": "Yes",
      "use_krb5_file": "No"
    },
    "Gateway2": {
      "binaries": "/opt/itrs/gateway",
      "local_includes": {
        "local-samplers.xml": "Samplers",
        "local-Rules.xml": "Rules"
      },
      "aes_file_name": "/opt/itrs/gateway/gateway2/private.aes",
      "setup_file": "/opt/itrs/gateway/gateway2/gateway.setup.xml",
      "working_dir": "/opt/itrs/gateway/gateway2",
      "use_aes_file": "Yes",
      "use_krb5_file": "No"
    }
  },
  "general": {
    "max_validation_accepted": "Warn"
  },
  "hub": {
    "api_port": "8081",
    "auth_method": "BASIC",
    "hostname": "hub.example-bank.com"
  },
  "shared_includes": {
    "http://includes.example-bank.com/hub.xml": "Global/Hub/Location",
    "http://includes.example-bank.com/hub-mappings.xml": "Global/Hub/Mappings",
    "http://includes.example-bank.com/users.xml": "Global/Auth"
  }
}

Local and shared files Copied

It is common for multiple Gateways to share configuration files either as exact duplicates or with local variations. The ccmigrate tool allows you to tag each Gateway include file as either a local or shared Gateway Hub resource. A local include file is unique to a specific named Gateway, whereas a shared include file is present in an identical form on multiple Gateways.

In a profile file, a local resource is always associated with a specific named Gateway. If a local resource with the same name is found on multiple Gateways in a profile then the user is required to supply a name for each file. To avoid clashing with the local copy of another Gateway, each local resource is prefixed with its Gateway of origin so that the naming scheme is <Gateway_name>/<Resource_name>.

In a profile file, a shared resource is associated with all Gateways in a profile. If a shared include file with the same name is found in multiple Gateway entries in a profile then the ccmigrate tool will assume this is a single shared include. The ccmigrate tool will only request a single name which will be the full name used in Gateway Hub.

To manage multiple regions where there are include files with the same name that are shared inside a region but different between regions, you will need to use a different profile for each region.

Validate a Gateway configuration Copied

Validating a will check the specified against a list of requirements, including but not limited to:

Connectivity to Gateway Hub.
Minimum required versions of Gateway and Gateway Hub.
Duplicate configurations stored in your Gateway Hub.
Gateway setup validation.

You will be prompted for user input to provide any additional requirements and to specify how include files should be tagged. The tool will modify the JSON requirements file based on your response. This means that you will not be asked to repeat information when re-running the tool.

For more information about centralised configuration prerequisites, see Centralised Gateways user guide.

To validate a Gateway configuration:

Download the ccmigrate package from ITRS Downloads.
Copy the downloaded package to the target Gateway host and extract the tool by running:
```
tar -xvzf ccmigrate.tar.gz
```
Stop the Gateway.

Run the following:

./ccmigrate --name <gateway_name> --profile <profile_file.json>

If the validation is successful, then the ccmigrate tool will print the output of your tests and also record a log of all checks to an ./output directory using the ccmigrate-<timestamp> format.

STATUS #TESTS #WARN #FAIL DESCRIPTION
FAILURE    1     0     1   Hub Tests
FAILURE    2     0     1   Gateway Tests
     OK    3     0     0   Setup file 'common.xml'
     OK    3     0     0   Setup file '/mnt/c/work/geneos/files/CapacityPlannerHelper.xml'
     OK    3     0     0   Setup file 'test.xml'
     OK    3     0     0   Setup file 'awsMonitoring.xml'
FAILURE    4     1     1   Main Setup file '/mnt/c/work/geneos/files/gateway.setup.xml'
FAILURE    0     0     0   Hub Stage 2 Tests
Overall Test Result: FAILURE

If the test fails, then the ccmigrate tool will print a FAILURE status and indicate the steps to resolve it in the log.

Migrate a Gateway configuration Copied

When migrating a Gateway to centralised configuration, the ccmigrate tool will read your JSON requirements file and perform all validation checks before performing the migration.

No changes are made to the Gateway working directory or resource directories and the migration will create a backup of your configuration before moving it into Gateway Hub.

To migrate your data:

Stop the Gateway.

Migrate the Gateway configuration:

./ccmigrate --name <gateway_name> --profile <profile_file.json> --migrate

Restart the Gateway using centralised configuration:
```
./gateway2.linux_64 -gateway-name <gateway_name> -gateway-hub <https://hub.example.com:8080>
```
- To connect to a secure Gateway Hub using Kerberos, use the following command line options:
```
-kerberos-principal <user@LDN.ITRS> -kerberos-keytab <user.keytab>
```
- To connect to a secure Gateway Hub using an application key, use the following command line options:
```
-app-key <user.app-key-file>
```

If the migration is successful, then the ccmigrate tool will print the output of your tests and a summary of the steps required to restart your Gateway using centralised configuration.

 STATUS #TESTS #WARN #FAIL DESCRIPTION
     OK    4     0     0   Hub Tests
     OK    2     0     0   Gateway Tests
     OK    3     0     0   Setup file '../inc/ReadWrite.xml'
     OK    3     0     0   Setup file '../inc/SSOAuth.xml'
     OK    3     0     0   Setup file '../inc/Hub.xml'
     OK    4     0     0   Main Setup file './gateways/Base.xml'
     OK    5     0     0   Hub Stage 2 Tests

Overall Test Result:      OK
Gateway has been uploaded into hub

In order to run this gateway from Hub you will need to update your gateway start script and restart your gateway
  - Remove command line option  -setup  (if present)
  - Add command line option  -gateway-name "Base"
  - Add command line option  -gateway-hub "https://127.0.0.1:16100"
  - Add credentials so connect to Hub (see http: .....)
    - Either -app-key 
    - Or -kerberos-principal  and -kerberos-keytab

Revert to local Gateway configuration Copied

When performing a migration the ccmigrate tool will create a backup of your local configuration. To revert your Gateway, start it using the backed up files.

Common issues Copied

The following issues are common when migrating Gateways to centralised configuration.

Gateway not configured with auto reload Copied

Ensure each include file has auto-reload enabled so that Gateways run the latest configuration from Gateway Hub.

Gateway binary has not been uploaded Copied

Upload the required Gateway binary file to Gateway Hub. This can be done from the configuration page in the Web Console or using the Gateway Hub API.

Unsupported users Copied

Centralised configuration requires using the Gateway Hub SSO agent for authentication. Locally defined Gateway users are not supported once authentication is enabled via Gateway Hub SSO. For more information, see Gateway Hub SSO Agent.

Gateway validation inconsistencies Copied

The ccmigrate tool performs a Gateway schema validation when run using the latest Gateway binary stored in Gateway Hub. This may result in a different output than running a schema validation using the Gateway Setup Editor because the Gateway Setup Editor will also check connected Netprobes for newer schemas.

Mixed Gateway deployments Copied

In deployments where some Gateways use centralised configuration and others do not, it is possible that shared include files may not be synchronised correctly. If a shared include file is available from Gateway Hub then centrally configured Gateways will use that, however, other Gateways will continue to use the version of the include file on disk. These versions may diverge and cause problems.

To avoid this you can lock the copy of the include file outside of Gateway Hub and restrict changes to only the centrally stored file. The file used by Gateways without centralised configuration can then be automatically synchronised with the centralised copy using your preferred tools outside of Gateway.

Gateway setup includes a rest address Copied

If a Gateway setup includes the Publishing > > Rest addresses field, then the migration process fails. Remove this field to proceed.

Performing migration on a running Gateway Copied

It is possible to run a migration operation on a running Gateway. However, this is not recommended and you should always stop your target Gateway when performing a migration.

Appendix: Output files Copied

When you run ccmigrate, the tool creates the following files in its output directory:

stdout — copy of the output sent to user when ccmigate is run (ignoring the questions).
validate.log — log of details of the validation process. Includes a list of all users tested for authentication.
validation.json — validation results from running -validate on a Gateway.
setup_files — directory containing the setup files used by a Gateway.
main.xml — main Gateway setup.
val.xml — modified main Gateway setup file that uses the include files in this directory. Used when validating the setup.
hub_main.xml — modified main Gateway setup. This is the main resource that is uploaded to Gateway Hub.
inc_*.xml — individual include files downloaded into this directory.
index.json — JSON describing the mapping between the location of the include file (remote or local) and the inc_*.xml files used by ccmigrate.
latest_resources — directory containing any resources that have been downloaded from Gateway Hub.