Gateway Hub Reference Guide

Note

This is a one-pager reference guide for Gateway Hub. For complete documentation, including detailed procedures, configuration options, and troubleshooting guides, refer to the full Gateway Hub documentation available in the offline documentation package.

Overview Copied

Gateway Hub is a component of ITRS’ Geneos monitoring platform, designed to provide operational resilience through enhanced storage, analytics, and automation capabilities. It serves as a centralized data repository and management layer that sits alongside traditional Geneos components, enabling organizations to monitor at significantly larger scale with improved insights and reduced operational overhead.

Gateway Hub transforms real-time monitoring by providing scalable big data storage, historical analytics, and a modern web-based interface. It is built from the ground up to support elastic and cloud-based environments, making it ideal for modern enterprise monitoring requirements.

Architecture Copied

Gateway Hub integrates seamlessly with existing Geneos components. The architecture follows this data flow:

  1. Data collection — Netprobes collect monitoring data from target systems.
  2. Gateway processing — Geneos Gateways process and normalize the data.
  3. Hub ingestion — Gateways publish data to Gateway Hub for storage and analysis.
  4. Data access — Data is accessible via:
    • Web Console for visualization and administration.
    • Active Console for traditional monitoring views.
    • REST API for programmatic access.
    • Kafka publishing for downstream systems.

Gateway Hub consists of multiple services running in a clustered configuration:

Operations and management Copied

Configuration file reference Copied

Gateway Hub uses a YAML configuration file for installation, reconfiguration, and upgrades. The configuration file structure is as follows:

installation:
  hosts:
  - server1.example.com
  - server2.example.com
  - server3.example.com
  
  connection:
    private_key: ~/.ssh/hub-key.pem
    port: 22

  tls:
    pem_file: ~/tls/hub.pem  # Production: provide PEM with private key, certificate, and CA chain
    reuse_self_signed_ca_certificate: true  # Testing: generate self-signed certificates

hub:
  root_dir: /opt/hub
  user: hub
  group: hub
  
  runtime:
    java_home: /usr/lib/jvm/java-1.8.0-openjdk/jre

Gateway Hub keeps a copy of configuration files used in each operation in /opt/hub/hub-current/etc/hub-installer/configuration-history on each node.

Essential hubctl Commands Copied

You can manage the Gateway Hub cluster using the hubctl tool included with the downloaded binaries. By default, you can find hubctl in the hub directory of your installation machine.

The hubctl tool reads configuration information from a YAML file that you must create when installing Gateway Hub. You can use hubctl and your configuration file to reconfigure, upgrade, or uninstall Gateway Hub.

The hubctl tool is the primary way you should reconfigure and manage the Gateway Hub platform components. To perform many operations with hubctl, you will need a YAML configuration file.

To use hubctl, run commands in the following format:

hubctl <command> <options> <config_file>

The following commands are available:

Note

To manage Gateway Hub using hubctl, all nodes must be reachable from your machine. Making changes to configuration files directly may break Gateway Hub functionality.

systemd and reconfiguration Copied

If the Gateway Hub orchestration service has been registered with systemd, you will need to mask the orchestration service when reconfiguring Gateway Hub.

To perform reconfiguration commands:

  1. Mask and stop the orchestration service by running the following on each node:

    sudo systemctl mask hub-orchestration
sudo systemctl stop hub-orchestration

  2. Start Gateway Hub and perform the reconfiguration using hubctl, then stop Gateway Hub. You are only required to run this on one node, the changes will affect the whole cluster.

    hubctl start <config_file>
hubctl setup reconfigure <config_file>
hubctl stop <config_file>

  3. Unmask and restart the orchestration service by running the following on each node:

    sudo systemctl unmask hub-orchestration
sudo systemctl start hub-orchestration

Service validation Copied

To verify Gateway Hub service health:

Query the REST address Copied

Use a browser, a dedicated client such as Postman, or curl -k in the command line, to query the REST address followed by /v0/ping. The default REST address is https://<hostname>:8081/.

Success

If the installation is successful, this returns 200 OK, and no output.

Access the Geneos Web Console Copied

View the Geneos Web Console by entering https://<hostname>:<port> in a web browser, replacing <hostname> with the hostname of the Gateway Hub server. Replace <port> with the Web Console port, the default is 8443.

Success

If the installation is successful, the page shows the Geneos Web Console.

Enable self monitoring Copied

Each Gateway Hub node runs an internal Netprobe and Collection Agent to collect metrics on its own performance.

You can connect a Gateway to the internal Netprobe to monitor Gateway Hub from your Active Console. You must connect to each node individually. To do this, follow the steps in Gateway Hub self monitoring.

To use Gateway Hub self monitoring, you must have an up to date license that includes the required tokens to use the following plugins:

Geneos Gateway integration Copied

You can connect Gateway and Gateway Hub to share data and centralise administration. You must have at least version 4.12 of both Active Console and Gateway.

Connect using the Gateway Setup Editor Copied

To connect a Gateway to Gateway Hub and share data follow these steps:

  1. Open Active Console.
  2. Double-click the Gateway you wish to configure with Gateway Hub. This opens the Gateway Setup Editor (GSE).
  3. In the GSE, double-click in the Navigation tree. This creates the Gateway Hub section.
  4. In the Gateway Hub section:
    • In Publishing address, add the host name and port of one of your Gateway Hub Kafka brokers. Once you have connected, Gateway will fetch the locations of all existing brokers from Gateway Hub. The default address is <hostname>:9092.
    • In REST address, add the REST address location of Gateway Hub. The REST address uses HTTPS. The default REST address is https://<hostname>:8081.
  5. You must configure security in Additional Settings by specifying the location of the Gateway Hub CA certificate. You can obtain the certificate file from <hub_home>/tls/trust-chain.pem on any Gateway Hub node.
kafka.security.protocol=ssl
kafka.ssl.ca.location=<location of CA certificate PEM file>
  1. Click Validate current document .
  2. Click Save current document .

When configuring Gateway Hub connections in Gateway Setup Editor, note the following:

For more details, see Gateway Hub Configuration.

Caution

Gateway Hub will reject metrics from Gateway samplers or dataviews that include the / character in their name.

Connect using the command line Copied

You can connect to Gateway Hub by using the -gateway-hub <REST URL> command line option when starting the Gateway. The URL specified here will take precedence over any REST address value specified in the Gateway Setup Editor.

Note

To use anomaly detection features you must start the Gateway in centrally configured mode.

Connect securely using SSO or API keys Copied

You can connect to a Gateway Hub without authentication. This is useful in testing and development environments. However, this is not secure and you should always use the SSO Agent or API keys in production environments.

In order to use Gateway Hub for centralised configuration or to perform anomaly detection, you should connect securely.

To start the Gateway with an authenticated connection to the Gateway Hub, use the following command line options, replacing the parts in <> with the your information:

You can also place these command line options in a file for the Gateway to read at start up. See Command line options.

Web Console Copied

The Geneos Web Console simplifies and modernizes the Geneos user interface, allowing easy monitoring of significantly larger scaled enterprises.

It is accessed through a web browser and is served from Gateway Hub. To access the Web Console:

  1. Navigate to https://<hostname>:<port> in your browser.
  2. Replace hostname with the location of your Gateway Hub instance and <port> with the Web Console port.

Note

You cannot access most of the pages of the Web Console without a valid Gateway Hub license.

The Web Console is secured using the SSO Agent that is bundled with Gateway Hub.

The log in process for the Web Console differs depending on if Kerberos has been enabled in the SSO Agent:

If log in is unsuccessful, an error screen is displayed.

UI settings Copied

The UI settings allow you to configure the appearance of the Web Console. These settings do not affect how the data is stored. To access these settings, click Settings at the top right of the screen.

The page has the following options:

Section Description
General Changes the display language and preferred user interface style.
Date and Time Changes the time zone, date, and time format.

Note

UI settings are stored in your current browser.

You may also click About ITRS Geneos Web Console in the Web Console side panel to access the online help and view the information about the current Gateway Hub version, build date, and environment.

User interface Copied

The Geneos Web Console provides a comprehensive web-based interface for monitoring, administration, and configuration. The following pages are available:

Page Description
Monitoring Provides an overview of all monitored entities in your estate, displayed as color-coded squares based on their status. This page allows you to group entities by attributes, filter entities using basic or advanced queries, and navigate to detailed entity views. Entities can be organized hierarchically with up to two levels of grouping, and overflow indicators show when there are more entities than can be displayed at once. Clicking an entity opens the Entity overview page for detailed analysis.
Dashboards Enables you to create custom visualizations using metrics from one or many entities. Dashboards consist of configurable panels that can display data in various formats including line charts, bar charts, tables, and more. Each panel can be configured with data sources, visualizations, and general settings. Dashboards support global parameters, date range selection, and event overlays. You can create, import, export, and manage dashboards, with role-based permissions controlling edit access.
Entity Overview Provides detailed information about a specific monitored entity, including attributes, dataviews organized by sampler, metric history graphs, and related events. The page displays the current values of important attributes, all dataviews with their metrics, historical metric data with optional expected bounds for anomaly detection, and a table of events related to selected metrics. You can configure which attributes are displayed and their order, view deleted rows to track transient events, and compare current values with expected values based on historical data.
Events Provides an overview of all events data produced by changes in your estate, helping you identify sources of problems or alerting noise. The page displays events as bar charts grouped by entity attributes, with filtering capabilities using basic or advanced queries. You can specify time ranges, include or exclude events based on alert type, and view detailed event lists. Events are color-coded by severity, and clicking an event opens the entity overview page showing the history of the metric that triggered the event.
Gateway Configuration Manages centrally configured Gateways and their resources. This section consists of three pages: Gateways (shows all centrally configured Gateways and their status), Binaries (allows uploading Gateway binary files for schema validation), and Includes (allows uploading Gateway setup files to a centralized store). You can examine and modify Gateway configurations, view resource revision history with diff capabilities, manage resource permissions based on roles, and track which Gateways are using which resources.
Publishing Configures publishing of normalized metric and event data to a downstream Kafka instance. Gateway Hub normalizes data to a standardized JSON format, making it easier to consume programmatically than the human-readable strings output by Gateways. The page allows you to configure Kafka connection settings, security protocols (PLAINTEXT, SASL_PLAINTEXT, SSL, SSL_CLIENT_AUTH, SASL_SSL), filtering to publish subsets of data, and request snapshots or schemas. Only one downstream Kafka instance is supported per Gateway Hub cluster.
Application Keys Manages OAuth 2.0 client credentials for connecting external applications securely to Gateway Hub. You can create API key credentials (composed of client_id and client_secret), assign Gateway Hub roles to determine resource access permissions, regenerate client secrets, and delete API keys. Application keys enable external applications to authenticate and access Gateway Hub data programmatically, with role-based permissions controlling what resources can be viewed or edited.
Roles Manages Gateway Hub roles that control viewing and editing permissions for various resources and data. By default, Gateway Hub includes Administrator and Operator roles that cannot be edited or removed. You can create custom roles, configure data filters using basic or advanced queries to limit what data a role can access, grant read or write access to specific Gateway Hub resources (dashboards, Gateway configuration, publishing, storage, webhooks, application keys, roles), and map SSO groups or users to roles for automatic role assignment.
Storage Configures retention policies for metric and event data stored in Gateway Hub. You can set separate retention periods for metrics and events (between 1 and 999 days), with the system automatically discarding data older than the specified period. The page also displays information about metric compression, which compresses stored metric data older than three days to reduce storage costs and improve retrieval speed. Data loss warnings are displayed when reducing retention periods.
Licence Displays information about your current Gateway Hub license and allows you to upload new licenses. The page shows general license information including expiry date, licensed products in your Geneos estate, and other license details. You can upload new license files through the Web Console interface, which is required to maintain access to Gateway Hub features.
Webhooks Configures publishing of events data to external applications using HTTP API calls. You can create multiple webhooks to connect to different APIs or applications, enabling bespoke integrations with your preferred tools. Each webhook can be configured with target URL, HTTP method, authentication (Basic or None), event filtering, attribute selection to include in event payloads, and enable/disable toggles. The page displays the status of all existing webhooks and allows you to create snapshots of the latest events to reset external systems in case of failure.
Ingestion Errors Collects and displays dataview errors generated from Gateway Hub due to incorrect schema or data coming from connected Gateways. The page provides a searchable table of errors with details including dataview path, error code, error message, affected Gateway and sampler, and currently configured schema. You can filter errors by path or error code, view detailed error information in a sidebar, and use this information to resolve schema issues in the Gateway Setup Editor. Errors are automatically deleted after seven days, or can be manually removed via PostgreSQL.

Load balancing Copied

By default, the Web Console is accessed directly from a Gateway Hub node on port 8443.

For increased resiliency and simpler navigation, you may use a load-balancer to share traffic between different nodes. This can also be used to map traffic to port 8443, allowing simpler navigation (for example https://geneos.mydomain.com).

SAML SSO and LDAP configuration Copied

Gateway Hub supports two Single Sign-On (SSO) authentication methods that enable users to access the Web Console and REST API using their organization credentials without additional password prompts. Both SSO methods support role-based security, mapping user credentials to Gateway Hub roles (Administrator, Operator, or custom roles) to control access to resources and data.

Only one SSO method can be enabled at a time.

SAML SSO Copied

SAML (Security Assertion Markup Language) is an open standard that allows an Identity Provider (IdP) to pass authentication credentials to Gateway Hub acting as a Service Provider (SP). This enables enterprise SSO integration using industry-standard OAuth 2.0 protocol. SAML-based SSO is only available when using the integrated Gateway Hub SSO Agent and is not supported by other Geneos components such as Gateways and Active Console.

Before configuring SAML-based SSO in the Web Console, you must add your SAML Identity Provider metadata to Gateway Hub. Typically, a REST API endpoint exists to automatically generate Identity Provider metadata. Otherwise, contact your systems administrator.

SAML metadata is configured using the hubctl tool. Ensure that the SAML metadata file is a valid XML file before configuring Gateway Hub.

To add SAML metadata to Gateway Hub, follow these steps:

Note

During this process, it is recommended that you disable security for easier set up. If you prefer not to, then you need to get a token from the APID so you can add the service provider metadata using curl -H "Authorization: Bearer {token}".

  1. Set the Identity Provider metadata to the Web Console:

    hubctl config set --service-name webconsole --local-config-files saml_idp_metadata.xml installation-descriptor.yaml

  2. Download the service provider metadata XML from the Web Console:

    curl https://<host>:8443/saml/metadata -k

  3. Add the service provider metadata file to the APID:

    curl -k -X PUT -H 'Content-Type: application/xml' "https://<host>:8081/v0/security/saml/serviceprovider/metadata" -d @saml_sp_metadata.xml -v

  4. Set the Identity Provider metadata file to the APID:

    hubctl config set --service-name apid --local-config-files saml_idp_metadata.xml installation-descriptor.yaml

After configuring SAML SSO, an information bar is displayed at the top of this page providing the current status.

You can also configure SAML field mappings and token expiration via Web Console > Administration > SAML Config.

LDAP SSO Copied

LDAP (Lightweight Directory Access Protocol) SSO enables authentication using LDAP or Active Directory, allowing users to access Gateway Hub with their environment credentials. LDAP SSO supports both Active Directory and Open LDAP, with Kerberos or basic authentication protocols. Gateway centralised configuration features require LDAP SSO with Kerberos to be enabled.

Two authentication protocols are supported, Kerberos and Basic. In almost all circumstances, you should select Kerberos.

Using Basic authentication can be useful for simple tests and proofs of concept. However, Basic is generally not recommended as it is not supported by Gateway functions such as Gateway centralised configuration.

Before configuring SSO, you should acquire the following information from a system administrator within your organisation:

Note

Registering an SPN with a set of LDAP credentials is likely to require a change request within your organisation.

To use secure LDAP (LDAPS), you must ensure the CA certificate used by the LDAP service is included in the list of trusted certificates for the Java runtime environment on all Gateway Hub nodes. This list is located in a file called cacerts.

To add the LDAP CA certificate to cacerts:

  1. Acquire the LDAP CA certificate from your LDAP administrator.

  2. On each Gateway Hub node, run the following. You will be prompted for the key file password.

    keytool -importcert -alias MySecureLdap -keystore $JAVA_HOME/jre/lib/security/cacerts -file path_to_ldap_CA_cert.cer

  3. Restart the Gateway Hub apid service:

    /opt/hub/hub-<version>/tools/hub-admin/bin/hub-admin service restart -n apid

When importing the certificate, note the following:

After configuring LDAP SSO, an information bar is displayed at the top of the page providing the current status.

Gateway Hub applications Copied

Gateway Hub includes several integrated applications that extend monitoring capabilities and provide advanced analytics:

Capacity Planner Copied

The Capacity Planner application enables automatic publishing of Geneos data from Gateway Hub to Capacity Planner projects for capacity planning and forecasting.

Key Features:

Configuration:

Use Cases:

Monitoring Coverage Copied

The Monitoring Coverage application allows you to set policies for correct monitoring configuration and track compliance across your estate.

Key Features:

Configuration:

Use Cases:

Grafana Dashboards Copied

Gateway Hub integrates with Grafana to provide advanced visualization and alerting capabilities for Geneos data.

Key Features:

Configuration:

Prerequisites:

Use Cases:

Troubleshooting Copied

This guide is intended to help you troubleshoot your Gateway Hub instance.

Obtain diagnostics Copied

Check your Gateway Hub license status Copied

  1. Access your Web Console using your browser.
  2. Click Administration > Licence to navigate to the Licence page.

The status of your Gateway Hub license is displayed in the General section.

Configure self monitoring Copied

Each Gateway Hub node includes an internal Netprobe that can be used to monitor the node’s performance.

To configure Gateway to show data from internal Netprobes, see Gateway Hub integration.

Obtain logs Copied

By default Gateway Hub stores log files in the <hub_root>/logs directory. You can optionally specify another directory when installing.

Log retention policies:

You can adjust the default log retention policies by editing the log configuration files. The asterisk * symbol indicates a wildcard which can take any value. For example hub-svc-snapshotd.

Obtain an info file Copied

An info file containing basic information about your Gateway Hub installation can be sent to ITRS support to help diagnose problems with your Gateway Hub instance. You obtain this file using your Web Console.

To obtain an info file, follow these steps:

  1. Access your Web Console using your browser.
  2. Click About ITRS Geneos to open the About page.
  3. Click the Get Diagnostic Info button to start the download.

This creates a Info.txt file in your default downloads folder.

Obtain a diagnostic file from the command line Copied

You can create a comprehensive diagnostics file that packages the Gateway Hub log files from each node in the cluster as well as system information about the cluster and attached storage.

To obtain a diagnostic file from the command line, on any node run:

hubctl diagnostics <config_file>

This creates a temporary file on each node and downloads all these files to your local machine. The location of the file is printed to stdout.

Obtain query request payloads Copied

All data that is available in the Web Console is obtained from the REST API, this means that you can also access that data by making requests directly. For full documentation of the REST API see Gateway Hub REST API v1alpha.

To quickly identify request payloads to fetch data from the REST API you can use your browsers development tool to observe the requests sent by the Web Console.

  1. Navigate to the Web Console page that contains the data you are interested in. For example, the metric history of an entity.
  2. To open developer tools use the keyboard short-cut Ctrl + Shift + I or navigate to More tools > Developer tools. This will open a new panel on the left.
  3. Select Network from the topmost tab bar. You may be prompted to reload the page before network information can load.
  4. The Network view is composed of two panels. The left panel shows all requests, you can select any request to view information about it in the right hand panel. Select a request and then select the Payload tab to see the request payload sent by the Web Console.

Caution

These steps are correct for chromium based browsers. Other browsers, such as Firefox or Safari, also include development tools but the exact steps to view request payloads may be different.

Procedures Copied

Verify the REST endpoint is reachable Copied

Use a browser, a dedicated client such as Postman, or curl -k in the command line, to query the REST address followed by /v0/admin/info. The default REST address is https://<hostname>:8081.

If the REST endpoint is reachable, this returns output similar to below:

{
 "buildDateTime": "2020-08-13T17:17:46.297Z",
 "version": "2.2.0",
 "gitCommit": "ad4b43309b01ce0a5dd350f91673cceed05e4e3f",
 "gitCommitDateTime": "2020-08-13T17:17:16Z",
 "gitBranch": "release-2.2.0",
 "javaInfo": {
  "vendor": "Oracle Corporation",
  "version": {
   "major": 1,
   "minor": 8,
   "patch": 0,
   "update": 242,
   "arch": "x64"
  },
  "vm": "OpenJDK 64-Bit Server VM"
 },
 "os": {
  "name": "Linux(3.10.0-1127.el7.x86_64)",
  "other": ["CentOS Linux release 7.8.2003 (Core)", ...]
 }
}

Resolve an ingestion error Copied

The home page has a dashboard widget that shows the total number of ingestion errors. Details of each error is shown on the Ingestion Errors page.

To navigate to the Ingestion Errors page, either:

The Ingestion Errors page displays a table of errors, grouped by Dataview and Error.

Selecting an error in the list reveals the error details sidebar. This contains further information about the error such as the origin of the error and the currently configured schema.

To resolve an ingestion error, follow these steps:

  1. Access your Web Console using your browser.
  2. Click Administration > Ingestion Errors to navigate to the Ingestion Errors page.
  3. Select an error from the list to view the Error Details sidebar. The Error Details screen provides the information, including the affected Gateway and sampler in your setup. In the example below, the error message is 13033 - Invalid Unit in Schema.
  4. Open the dataview in Active Console to confirm the actual values.
  5. Open your Gateway Setup Editor.
  6. Navigate to the Publishing tab of the relevant sampler. Using the example above, the relevant sampler is gw-sql.
  7. In Schemas > Dataviews, click Data to open the schema configuration for the relevant dataview in the sampler.
  8. On the Data screen, correct the schema as required. In this example, the correct unit of measure is selected using the Unit of measure field.
  9. Click Close to close the window.
  10. Click Save to save the setup.

Success

Once the updated schema is saved, Gateway Hub will be updated and the previous error is removed from the system.

Remove ingestion errors Copied

Gateway Hub deletes old ingestion errors after seven days.

However, if you accumulate stale ingestion errors you may want to manually remove them.

To remove ingestion errors, perform the following steps on each node:

  1. Navigate to Gateway Hub’s built-in PostgreSQL database. By default, this is located at /opt/hub/hub-<hub_version>/services/postgres-timescale-<version>.

  2. Start a SQL prompt:

    ./run-psql.sh

  3. Connect to the database hub as the user postgres:

    \c hub

  4. Delete the ingestion errors:

    DELETE FROM errors;

  5. Exit the SQL prompt:

    exit

Note

Since you cannot perform SQL operations simultaneously on all nodes, some inconsistencies may occur.

Obtain the Subject Alternative Name of a certificate Copied

You can extract the Subject Alternative Name from a certificate using the OpenSSL command line tool. This allows you to ensure it matches the Gateway Hub domain.

Note

If you are using non self-signed TLS certificates, make sure that both server and client authentication are enabled. Otherwise, you might get a certificate verification error.

To extract the Subject Alternative Name, run:

openssl x509 -in <certificate_file> -text -noout

Which will return output similar to:

X509v3 Key Usage:
    Digital Signature, Non Repudiation, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
    DNS:DNS-name-1, DNS:DNS-name-2, ...

Add Gateway Hub certificate authority to Grafana Copied

In order for Grafana to connect to Gateway Hub securely, the certificate authority (CA) that has signed the TLS/SSL certificate used by Gateway Hub must be trusted by the system running Grafana.

If Gateway Hub is installed using certificates signed by a non-trusted CA, including the internal CA, you must add the relevant CA certificate to the trust store of the Grafana host. If Gateway Hub has been configured using production certificates that are trusted across an organisation, this is not required.

If you attempt to connect Grafana and Gateway Hub using non-trusted certificates, the connection will fail and Grafana will receive no data. The server logs will include a Failed to get access token error and state certificate signed by unknown authority.

To add Gateway Hub to a Linux system’s recognised certificate authorities:

  1. Locate the CA certificate used to sign Gateway Hub certificates. In a default installation, using an internal CA, this is /opt/hub/hub-version-GA/tls/trust-chain.pem.

  2. Copy the CA certificate to the trust store of the Grafana host. In a CentOS or Red Hat system this is located at /etc/pki/ca-trust/source/anchors/.

  3. To update the recognised certificate authorities, run:

    update-ca-trust extract

    You can verify the updated list by running:

    trust list

  4. Restart Grafana.

  5. In the Grafana web interface, open the ITRS Geneos Gateway Hub Datasource settings and disable Skip TLS Verify.

Restart Capacity Planner application Copied

To restart the Capacity Planner application, perform the following steps on each Gateway Hub node:

  1. Navigate to the <hub_install_dir>/hub-current/bin directory.

  2. Run the following command:

    hub-admin service restart -n capacity-planner

Test a Gateway connection Copied

When configuring a Gateway to publish data to Gateway Hub you may encounter a GatewayHubPublishing Failed sending: Local: Unknown topic error. This occurs when a Gateway cannot find the required topics or cannot connect to any brokers.

To diagnose the cause of the error use the kafkacat tool to test the connection to Gateway Hub and fetch a list of metadata, including topic names.

Run the kafkacat command, specifying as options each of the Additional Settings required by the Gateway Setup Editor. These options have the form -X setting.name=value where the setting.name matches the corresponding Additional Setting omitting the kafka prefix.

You should provide the same credentials used when configuring the connection in the Gateway Setup Editor.

kafkacat -X security.protocol=ssl -X ssl.ca.location=<hub_CA_certificate> -b <hostname>:9092 -L

If kafkacat returns a list of topics that does not include geneos-events or geneos-metrics-v1, the Gateway will not be able to publish metrics to Gateway Hub. You should check the Gateway Hub configuration.

If kafkacat cannot connect to Gateway Hub, the Gateway will also be unable to connect. You should check the network connection.

Kafkacat:

The kafkacat tool is an open source utility written and maintained by the author of the librdkafka library used by Geneos. This utility is shipped with Linux 64-bit Gateways to ease the testing of connecting to your Kafka infrastructure. For more information about kafkacat, see kafkacat Github.

To ensure that kafkacat uses the same Kafka and SSL libraries as the Gateway, kafkacat must be run with the following environment variables:

Increase Kafka message size for centralised configuration validation Copied

When running a Gateway using central configuration, setup files are validated by Gateway Hub.

Gateway Hub uses Kafka messages to distribute Gateway setup files to a dedicated daemon that validates them. If the setup file size is large, it may exceed the default Kafka message limit of 1 MB. In this case, Gateway Hub is unable to validate files and the Gateway setup files cannot be saved.

To resolve this issue, you must increase the maximum Kafka message size for Gateway setup validation.

However, the following additional memory limitations still apply:

If any of these are exceeded, Gateway setup files cannot be saved.

Caution

Kafka messages over 8 MB in size will also breach the maximum HTTP server request size of the API Daemon.

Diagnose:

The Kafka message size may be too small if both of the following behaviours are occurring:

2020-09-28 11:47:04.479Z ERROR [default-akka.actor.default-dispatcher-3] [ValidationQueryRequesterImpl] - Unhandled Kafka Exception publishing to topic 'unknown'
org.apache.kafka.common.errors.RecordTooLargeException: The message is 6747120 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
2020-09-28 11:47:04.479Z ERROR [default-akka.actor.default-dispatcher-3] [ValidationQueryRequesterImpl] - Failed to publish validation query
org.apache.kafka.common.errors.RecordTooLargeException: The message is 6747120 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

Increase message size:

To change the maximum Kafka message size for Gateway setup validation, perform the following steps:

  1. Open the API Daemon’s Kafka configuration in your default text editor:

    ./hubctl/hubctl config edit -n apid -c apid.yaml installation-descriptor.yml

  2. In the kafkaProducer > properties section, specify the max.request.size in bytes. For example:

    # Kafka producer for publishing to the Hub Kafka cluster
kafkaProducer:
  properties:
    max.request.size: 5242880
    bootstrap.servers: localhost:9092
    acks: all
    key.serializer: org.apache.kafka.common.serialization.StringSerializer
    value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer

  3. Open the Gateway configuration Daemon’s Kafka configuration in your default text editor:

    ./hubctl/hubctl config edit -n gateway-configd -c gateway-configd.yaml installation-descriptor.yml

  4. In the kafkaProducer > properties section specify the max.request.size in bytes, and in the kafkaConsumer > properties section specify the fetch.max.bytes in bytes. The configuration file should look similar to:

    # Kafka producer
kafkaProducer:
  properties:
    max.request.size: 5242880
    bootstrap.servers: localhost:9092
    security.protocol: SSL

# Kafka consumer
kafkaConsumer:
  properties:
    fetch.max.bytes: 5242880
    bootstrap.servers: localhost:9092
    security.protocol: SSL

  5. Open the Kafka server configuration in your default text editor:

    ./hubctl/hubctl config edit -n kafka -c server.properties installation-descriptor.yml

  6. In the Replication Settings section, specify the replica.fetch.max.bytes in bytes. The configuration file should look similar to:

    #### Replication Settings  ####

min.insync.replicas=3
replica.fetch.max.bytes=5242880

  7. Update the Kafka topic configuration. To do this, run the following on a Gateway Hub node:

    <hub_root>/hub-current/services/kafka-2.12-2.5.0/kafka_2.12-2.5.0/bin/kafka-configs.sh --zookeeper localhost:5181 --entity-type topics --entity-name hub-gateways-validations-requests --alter --add-config max.message.bytes=5242880
<hub_root>/hub-current/services/kafka-2.12-2.5.0/kafka_2.12-2.5.0/bin/kafka-configs.sh --zookeeper localhost:5181 --entity-type topics --entity-name hub-gateways-validations-queries --alter --add-config max.message.bytes=5242880

Renew Gateway Hub server certificate Copied

When installing Gateway Hub, you can choose to use self-signed certificates for TLS connections between Gateway Hub components.

The installer generates a self-signed CA certificate and uses it to sign the server certificate used by Gateway Hub. The CA certificate generated by the installer has a lifetime of 100 years. However, the server certificate has a lifetime of 397 days and you must generate a new certificate before it expires.

To generate a new server certificate, run:

hubctl setup reconfigure config.yml

Check the latest resource versions used by Gateways Copied

Make sure that all resources of the registered Gateways in the system use the latest version. You should always do this unless there is a critical issue with the system. Specifically, if there is an issue with etcd, it may result in a resource being updated but the Gateway update failing.

Use the cconfig_check_latest_resource_versions.sh script located in /opt/hub/hub-current/services/gateway-configd-<version> in the default installation directory.

To check the latest versions of the resources used by your Gateways, run:

./cconfig_check_latest_resource_versions.sh [-h] [-i] [-c <client_id> -s <client_secret>]

Where:

Note

When security is enabled, <client_id> and <client_secret> are required and must have admin rights. If not, the script won’t return results.

Prune the etcd key-value store history Copied

You can prune the etcd history of key-value (KV) entries used by the centralised configuration. This can free up unused disk space.

Use the kv_history.sh script located in /opt/hub/hub-current/services/etcd/etcd-gateway-<version> in the default installation directory.

To prune the KV store history in etcd, run:

kv_history.sh [-h] [-v] prune [OPTION]...

Where:

You can use either of the following for OPTION:

In addition, you can also run the following commands:

Restore PostgreSQL database files on a node Copied

If a PostgreSQL database file has been corrupted or deleted, you can restore PostgreSQL from another node. Since the PostgreSQL database is replicated across nodes, you can copy the entire database content from one node to another.

  1. Stop the Gateway Hub service on all nodes. If systemd is managing Gateway Hub, run the following commands:

    sudo systemctl mask hub-orchestration
sudo systemctl stop hub-orchestration

    If not, on each node, run:

    /opt/hub/current/bin/hub.sh stop

  2. Make sure that Gateway Hub is stopped on each node by running:

    /opt/hub/current/bin/hub.sh

    To check that PostgreSQL is effectively stopped, run:

    ps -edf|grep postgres:

  3. From a node with intact PostgreSQL database, create a TAR file for each of the following:

    • <hub data>/postgres-timescale/pgdata
    • <hub data>/postgres-timescale/pgwal

  4. Copy the created TAR files to the destination node.

  5. On the destination node, rename the following paths for backup:

    • <hub data>/postgres-timescale/pgdata to <hub data>/postgres-timescale/pgdata.<timestamp>
    • <hub data>/postgres-timescale/pgwal to <hub data>/postgres-timescale/pgwal.<timestamp>

  6. On the destination node, unzip the TAR files copied from another node. This will create new <hub data>/postgres-timescale/pgdata and <hub data>/postgres-timescale/pgwal file paths.

  7. On the destination node, start PostgreSQL manually by running:

    /opt/hub/current/services/postgres-timescale-<version>/start.sh

    Confirm that you see the following messages in the logs:

    LOG:  database system is ready to accept connections
LOG:  TimescaleDB background worker launcher connected to shared catalogs

  8. On the destination node, stop PostgreSQL by using the keyboard short-cut CTRL + C.

  9. Restart Gateway Hub normally using hubctl or systemd.

API Documentation Copied

Gateway Hub provides a comprehensive REST API for programmatic access to all functionality and data. The API enables integration with external applications, custom tools, and automation scripts.

For complete API documentation, including endpoint reference, authentication, query syntax, and examples, see Gateway Hub REST API v1alpha.

The REST API provides access to:

All data available in the Web Console is obtained from the REST API, meaning you can access the same data programmatically by making direct API requests.

Uninstall Gateway Hub Copied

You can uninstall Gateway Hub using the hubctl command. This command uninstalls all the nodes specified in the installation configuration file in the hosts list or linked hosts_file.

For more information about the installation configuration file, see Install.

In environments where you cannot connect to all nodes in the cluster, you can remove specific nodes by editing the hosts list or linked hosts_file. This allows you to uninstall the Gateway Hub in stages.

Caution

You cannot perform partial uninstallations. Removing a node unexpectedly will break the cluster.

To uninstall Gateway Hub, run:

hubctl setup uninstall config.yaml

This lists the nodes selected for removal and prompts for confirmation.

###########################################################################
# WARNING ! You are about to completely uninstall the Hub from all nodes. #
###########################################################################
Proceed ? [y/N]

Performing the uninstallation stops all processes and removes all programs and data associated with Gateway Hub on each node. You must also reboot each node to ensure temporary files and other miscellanea are removed.

After a completed uninstallation, the following prompt is displayed:

Some data is retained when uninstalling to avoid losing valuable information. You must remove these directories manually if required.

Systemd Copied

If you have registered the Gateway Hub orchestration service with systemd, then additional steps are required. The uninstaller will detect if you have registered the orchestration service with systemd and provide instructions to unregister the service.

["Geneos"] ["Geneos > Gateway Hub"] ["User Guide"]

Was this topic helpful?