Gateway Hub 1.x Release Notes

Gateway Hub 1.8.0 Copied

Released: March 2020

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Application keys Copied

You can now generate Application keys that allow you to connect third party applications to Gateway Hub securely.

Application keys are designed to be used by a single application and can be centrally revoked if required.

Grafana Data source Copied

The Grafana data source has been released as a full feature. The Geneos data source for Grafana lets you seamlessly integrate data from Gateway Hub into your Grafana dashboards.

Event browser Copied

You can now identify alerting irregularities using the new Events page in the Web Console. The Events page allows you to visualise events generated over time, enabling noisy entities to be identified and remedied.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-12613 The Apache Commons Compress library has been upgraded to prevent a security vulnerability.
STARK-12594 The Jackson Databind library has been upgraded to prevent a security vulnerability.
STARK-12384 Netty has been upgraded to prevent a security vulnerability.
STARK-12346 Webhooks are now fired correctly when Kafka attempts to commit a batch.
STARK-12305 The Gateway Hub sizing tool is now available as a separate download.
STARK-12300 You can now filter the list of entity attributes and possible values by the deletedEntities parameter.
STARK-12289 Gateway Hub can now ingest much larger dataviews without having to increase Kafka’s maximum message size.
STARK-12194 The UI controls for historical charts are now significantly more compact.
STARK-12187 The hubcheck script now checks that custom ports are free.
STARK-12129 Gateway Hub will now retain a maximum of 14 archived logs OR 1GB of log files, whichever is least. Logs are rolled (archived) each day or when the file size reaches 250MB.
STARK-11937 You can now specify the retention policy for Apache Spark logs. This can help conserve disk space.
STARK-11835 The Webconsole now shows a status bar when Webhooks are active.
STARK-11806 The historical metric chart now shows data from a 24 hour time window by default.
STARK-11778 You only need to specify the SSL password once during installation.
STARK-11183 When creating chained entity filters Gateway Hub now constrains the possible values of each new filter based on the preceding filters.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-12460 Gateway Hub now includes additional units of measurement.
STARK-12401 Gateway Hub now correctly handles errors when an expired oauth token is passed with an API request.
STARK-12388 Resources for Spark are now calculated correctly.
STARK-12168 Metric snapshot performance has been improved by running clean-up tasks more frequently.
STARK-12148 Significantly improved the performance of real-time updates to metrics, particularly when new metrics are added to an entity.
STARK-11971 You can now perform upgrades on installations that are using custom ports.
STARK-11879 The hubcheck script now ensures that disks are correctly shared.
STARK-11841 Upgrades using a non-default user now work correctly.
STARK-11805 The weekly view chart now calculates the time range correctly.
STARK-11692 The historical metric chart now correctly performs Y-axis rounding.
STARK-11689 Historical metric data crossing the Y-axis is now displayed correctly after panning.
STARK-11663 Ingestion Errors link is now styled correctly.
STARK-11609 The Web Console Home page no longer displays dual scrollbars in Firefox.
STARK-11069 The entity filter drop-down list is no longer obscured.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.7.3 Copied

Released: 13 February 2020

Note

Gateway Hub 1.7.3 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for in the ITRS Downloads page.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-12458 Gateway Hub now correctly calculates the required memory allocation for YARN during installation.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, “test;”. To resolve this issue update the dataview column nameremoving the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.7.2 Copied

Released: 1 February 2020

Note

Gateway Hub 1.7.2 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for in the ITRS Downloads page.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-12289 Gateway Hub can now ingest much larger dataviews without having to increase Kafka’s maximum message size.
STARK-12283 You can now configure Kafka’s maximum message size for persistent entity snapshots. Setting the maximum message size to a higher value than the default of 1 MB may be necessary when working with very large dataviews (in excess of 20,000 cells).

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.7.1 Copied

Released: 23 January 2020

Note

Gateway Hub 1.7.1 is a patch release containing a number of fixes for security and deployment issues.

This release delivers the updated binaries for in the ITRS Downloads page.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-12168 Significantly improved metric snapshot performance by running clean-up tasks more frequently.
STARK-12148 Significantly improved the performance of real-time updates to metrics, particularly when new metrics are added to an entity.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.7.0 Copied

Released: 17 January 2020

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-11947 You can now configure Gateway Hub TLS/SSL using a PEM file. If a PEM file is not provided, Gateway Hub will default to using self-signed certificates.
STARK-11739 Improved error handling in Gateway Hub dataset generation.
STARK-11647 You can now use Webhooks to publish Geneos events to external services like ticketing systems.
STARK-11638 The hubcheck tool no longer asks for a password twice when creating ssh connections.
STARK-11203 Simplified the controls for the metric history chart.
STARK-11202 Improved date selection controls for the metric history chart.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-12073 Gateway Hub now correctly updates the snapshot daemon and Kafka client when upgrading from a configuration using a non-default password for SSL.
STARK-11702 Certificates are no longer stored in the /tmp folder after configuration is completed.
STARK-11635 You can now re-run hubctl setup ssl on a failed ssl setup.
STARK-11447 The hubcheck logs are now generated in the correct location.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11973 Kafka depends upon Apache Log4j 1.2.17 which has known vulnerabilities.
STARK-11723 BlackDuck scans show a critical risk in Apache Log4j if used for remote logging. Our investigation shows that Gateway Hub and MapR Control System do not use Apache Log4j remote logging and so are not affected by this vulnerability.
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name, for example, “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.6.0 Copied

Released: 12 December 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Web Console monitoring UI Copied

The Web Console monitoring UI brings a brand new look for real-time monitoring, to supplement Active Console. The following is available in the Web Console:

The Web Console is available free of charge for all Geneos users.

Event query API Copied

Events such as severity, snooze and user assignment data can now be queried using an open API, providing access to the raw data for further analysis of trends and problem entities.

Centralised configuration validation Copied

Updates to centralised configuration files can now be validated against all Gateways dependent on that configuration, in order to avoid updates causing issues remotely. By default, changes that would cause errors in any dependent Gateways are refused.

The API for centralised configuration has been finalised since the early access version, and as a result all Gateways using GA centralised configuration must be version 5.0.

Breaking changes Copied

These are the breaking changes in this release:

Issue key Release description
STARK-9578 The API for centralised configuration has changed since Early Access. Upgrade to Gateway 5.0 to use centralised configuration with Gateway Hub 1.6 and above.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-10951 Metric and event endpoints now accept the Content type application/x-yaml.
STARK-10830 Basic authentication is refreshed more frequently, reducing the likelihood that the user will need to re-enter their credentials.
STARK-10676 Error messages now contain more detailed information.
STARK-10674 When security is disabled, Gateway Hub responds to GET requests to either of /security and /security/server/roles/mappings passing a Basic Authorization header with a 200 OK response.
STARK-10556 The Web Console now contains a User Profile page, which shows user details such as roles.
STARK-10440 The Web Console now remembers the open or closed state of the sidebar menu between sessions.
STARK-10358 New publishing filters can now be added using drop-down lists.
STARK-9578 Updated the centralised configuration API.
STARK-8703, STARK-8391 Changes to centralised configuration resources are now validated centrally.
STARK-8390 Added KPIs to the home dashboard showing:
  • The number of events raised over time.
  • The number of current entities and their severities.
STARK-7798 Improved the scrollbar style.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-11157 The description for CDF metric aggregation is now correct.
STARK-10834 The Web Console no longer supplies an empty diagnostic file.
STARK-10817 Improved the readability of 401 error screens when in dark mode.
STARK-10744 Application buttons are no longer shown on error pages when the user is not authenticated.
STARK-10706 Gateway Hub now correctly handles errors produced by incorrectly configuring the Kerberos settings.
STARK-10701 Gateway Hub now correctly handles an error when attempting to change roles with an expired token.
STARK-10678 Gateway Hub now returns a 400 error when an authorisation token expires.
STARK-10616 The Web Console will now show an error when you provide an invalid security configuration.
STARK-10593 Improved the error message when a user changes a role mapping that would remove their own admin access.
STARK-10539 The security endpoint no longer returns a 400 error when security is disabled.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is no longer published in the canonical unit (for example, Bytes). The values are published in the form originally set by the Gateway.
STARK-9665 Long error messages are now correctly formatted in the system and ingestion error page sidebars.
STARK-6928 The hubctl setup licence command no longer fails silently when a licence file is omitted.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-11411 The Archiver log might show an error the very first time the Archiver runs. The Archiver should heal and run successfully the next time it runs.
STARK-8239 BlackDuck scan shows a Critical risk in log4j if used for remote logging. Our investigation shows that Gateway Hub and MapR do not use log4j remote logging and so are not affected by this vulnerability.
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.5.4 Copied

Released: 13 November, 2019

Note

Gateway Hub 1.5.4 is a patch release containing a number of fixes for security and deployment issues. Versions 1.5.1 — 1.5.3 were only released internally.

This release delivers the updated binaries for in the ITRS Downloads page.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-11306 Upgraded to Kafka 2.3.1.
STARK-11226 It is now possible to allow the installer to do a self-installation without needing to use sudo and using a root user.
STARK-11032 Upgraded to jackson-databind library 2.10.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-11327 The Node Manager did not start if /tmp had noexec set. This is was fixed by the addition of a new setting in the hubctl JSON configuration: hub.install.java_io_tmpdiz. It should point to a directory that is mounted on a disk that does not have noexec set.
STARK-10046 Fixed an issue caused by a recent security update to systemd, that resulted in multiple copies of Warden and Gateway Hub services to be run and deployment to fail.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-10849 Web Dashboard is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10831 Webslinger is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10046 Gateway Hub is not compatible with systemd version 219-65 due to an issue with the underlyingMapR platform. A previous version of systemd is required.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-7596 The SSL certificate install incorrectly uses the default ‘hub’ user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:
"ansible" : { 
   "variables" : { 
  "hub_user":
    "gwh", 
    "hub_group":
    "gwh", 
  } 
}
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.5.0 Copied

Released: 8 October, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Anomaly detection rules Copied

One of the major sources of monitoring noise is due to simple thresholds set on metrics that are inherently variable over time. Many business processes follow a fairly predictable pattern, peaking around market open and close, and dormant over the weekends.

Anomaly detection rules provide a better way of monitoring these kinds of metrics. By analysing the history of a metric stored in Gateway Hub, an anomaly detection rule can determine what is normal, alerting only when this normality is deviated from.

For example, an anomaly detection rule can be set to trigger an alert if orders fall outside a certain tolerance for a given time and day of the week, for that specific server, or for all similar servers.

Switching to anomaly detection rules can result in significantly more accurate alerting and substantially reduced events over time for metrics that follow a pattern or cycle. Some metrics are more suited to anomaly detection rules than others.

For more information, see Anomaly Detection.

Kafka publishing filter Copied

In addition to the improved data format, normalisation and schemas published to Kafka from Gateway Hub, filtering options are now available to restrict the data that is published to an external Kafka cluster.

The publishing of metrics and events can now be individually controlled. In addition, publishing can be restricted to certain entities such as where Division = Securities and certain metrics, such as Bloomberg only.

For more information, see Publish to Kafka.

Role based security Copied

This release brings easy-to-configure single sign on (SSO) and role based security to Gateway Hub. With this version, predefined Administrator and Operator roles are available to provide write and read access to Gateway Hub. Gateway Access control is still configured at the Gateway, although this will be centralised in a future version.

Role based security requires integration with Active Directory or LDAP.

Breaking changes Copied

These are the breaking changes in this release:

Issue key Release description
STARK-9970 The UI and SSO endpoints were previously accessed on unique ports (defaulting to 8888 and 9999, respectively). These are now accessed using the same port as the rest of Gateway Hub API calls, defaulting to 8080.
STARK-9969 Before you upgrade to 1.5, save your SSO configuration separately. The configuration will be lost during the upgrade. Once 1.5 is successfully installed, SSO will need to be configured using the Security section of the Web UI.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-10538 Apache Commons BeanUtils library was updated.
STARK-10452 The user’s display name is now shown in the application when SSO is enabled.
STARK-10302 Improved YARN scheduler to allow important jobs to be prioritised correctly.
STARK-10186 YARN logs are now included in diagnostics files.
STARK-10106 The licence is now checked before enabling metric and event storage.
STARK-9970 The SSO Agent and UI are now served by the Gateway Hub API, and the standalone agents have been removed. Note that the UI and SSO API calls are now done using the same port as the rest of the Gateway Hub API.
STARK-9969 The previous standalone SSO and UI agents are removed on upgrade from version 1.4.
STARK-9908 Resources are now more fairly allocated between the various components within Gateway Hub, providing better stability and performance particularly for large queries.
STARK-9560 Dashboard panels are now always displayed while the data is still loading.
STARK-9444 Publishing endpoint was updated to allow message filtering.
STARK-9398 Improved robustness of installer when attempting to request a licence key on a slow internet connection.
STARK-9236 The Web Console is now served on the same port as the REST API.
STARK-9179 Entity change events are now published as a new message type. They indicate if a new entity has been detected, removed, or the attributes of an entity have changed.
STARK-9086 Various improvements were introduced to the Dark theme of the application.
STARK-9039 Published events can now be filtered by event type, such as severity.
STARK-8859 The /admin/retention endpoints now require ISO 8601 duration formats in line with other endpoints.
STARK-8821 The Storage page in the Web Console now allows metric and event storage to be explicitly enabled and disabled.
STARK-8547 React was upgraded to version 16.8.x.
STARK-8540 Various changes were made to Spark and YARN configuration, resulting in better performance and robustness, particularly with smaller clusters.
STARK-8402 The Web Console now defaults to using the browser time zone in the absence of a configured user preference.
STARK-8401 The layout of the errors tables has been improved when viewed on larger monitors.
STARK-8388 The number of system errors in the system is now displayed as a KPI on the home dashboard.
STARK-8258 GET /versions is now available to provide the API versions supported by Gateway Hub.
STARK-8136 Now that the SSO endpoints are hosted on the same port as the rest of the Gateway Hub endpoints, the Open API definition has been unified.
STARK-8135 The Web Console now routes SSO calls via the same port as other Gateway Hub API calls.
STARK-8134 The SSO REST API endpoints are now hosted on the same port as the rest of the Gateway Hub REST API endpoints.
STARK-7742 Material UI was updated to version 3.9.3.
STARK-7512 Kafka publishing can now be filtered by sampler name.
STARK-7511 Kafka publishing can now be filtered by entity query, such as all entities where Division = Securities.
STARK-7510 Kafka publishing can now be filtered by message type, such as metrics or events.
STARK-7233 retainMetricsAndEvents: true/false has been added to the retention API, allowing metric and event history to be explicitly enabled and disabled.
STARK-6828 Administrator and Operator roles are available for assignment to LDAP users and groups.
STARK-6705 Centralised configuration resources can now be locked and unlocked via API.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-10330 Metric queries now return data in the same unit of measure as the one displayed in Active Console, rather than the canonical unit of measure.
STARK-10087 Fixed an issue where retention could not be updated after previous invalid input.
STARK-9925 An error is now displayed when a metric query contains one or more invalid metric names.
STARK-9681 Fixed an issue where archiving was getting stuck in m5.2xlarge AWS instances.
STARK-9634 Errors without a message body, for example, a URL with more than 2048 characters, are now handled correctly.
STARK-9612 Configuring SSL using hubctl setup ssl no longer misallocates YARN resources.
STARK-9337 An issue with the archiving process occasionally getting stuck due to a lack of resources is now resolved through better allocation of resources on install.
STARK-9231 Kafka publishing configuration is no longer validated when publishing is disabled.
STARK-9161 Handling of invalid metric query API parameters was improved.
STARK-9138 Uploading a licence is now supported in the Web Console when the system is unlicensed.
STARK-9128 The ingestion errors loading indicator is now correctly positioned inside the table control.
STARK-9063 Fixed colour of Show related errors control in the Errors pages.
STARK-9014 Various fixes were introduced to the Web Console when used in Firefox.
STARK-8664 Repeatedly misconfiguring Kafka publishing in the Web Console no longer gives the impression that the invalid config has been saved successfully.
STARK-8535 Fixed an issue where the error search control only showed tooltip intermittently.
STARK-8510 Invalid publishing configuration no longer causes the Web Console to hang.
STARK-8506 Publishing status bar is now refreshed when configuration is updated.
STARK-8448 Attributes with special characters are now encoded correctly in the URL.
STARK-8432 The ingestion and system errors pages are now fully responsive down to mobile screens.
STARK-8387 Refreshing the Web Console while a page is loading no longer displays an incorrect error screen.
STARK-7672 Archiving is now done more frequently, instead of inside the maintenance window.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-10849 Web Dashboard is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10831 Webslinger is not compatible with SSO Agent in Gateway Hub. This will be addressed in the next release.
STARK-10046 Gateway Hub is not compatible with systemd version 219-65 due to an issue with the underlyingMapR platform. A previous version of systemd is required.
STARK-9730 The operating system language must be set to English, otherwise the build fails.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-7596 The SSL certificate install incorrectly uses the default ‘hub’ user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:
"ansible" : { 
   "variables" : { 
  "hub_user":
    "gwh", 
    "hub_group":
    "gwh", 
  } 
}
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.4.1 Copied

Released: 14 August, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-9778 Gateway Hub can now be installed to custom directories. However, please note that MapR must still be installed in /opt/mapr.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-9730 The operating system language must be set to English, otherwise the build will fail.
STARK-9095 Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:
fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-7596 The SSL certificate install incorrectly uses the default ‘hub’ user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:
"ansible" : { 
   "variables" : { 
  "hub_user":
    "gwh", 
    "hub_group":
    "gwh", 
  } 
}
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.4.0 Copied

Released: 31 July, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Improved Gateway integration Copied

Integration between Gateway and Gateway Hub has been optimised, resulting in almost 25% less disk required for a standard Gateway Hub with metric storage compared to the previous version. Data will also be available in the Gateway Hub significantly faster than before due to the improved processing. These improvements require Gateway version 4.12.x.

Centralised Gateway configuration locking Copied

Centralised configuration files can now be centrally locked, preventing concurrent updates from causing issues. This feature requires Gateway 4.12.x.

For more information, see Centralised Gateways User Guide.

Improved API documentation Copied

API documentation is now published using redoc allowing better navigation and examples.

For more information, see Gateway Hub API documentation.

Custom install directory Copied

You can now perform new installations of Gateway Hub in custom locations, allowing Gateway Hub, the Web Console SSO, and Kafka to be installed in any location.

Note that the install location for MapR cannot be changed from /opt/mapr.

Note

Changing install directories as part of an upgrade is not supported.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-9518 The maintenance window section has been removed from the Storage screen. Archiving is now done several times per day rather than during a maintenance period.
STARK-9469 Apache Commons Collections library was updated.
STARK-9355 The allocation of resources is now more sensibly calculated, providing more resources to the components that need it, such as Spark and YARN. This will prevent a number of OOM issues seen previously.
STARK-9247 The raw and enriched internal Kafka topics have been replaced with a single merged topic for 4.12 Gateways and above.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-9463 Fixed instances where a metric query would fail with the following exception:

java.lang.RuntimeException: org.apache.spark.SparkException:
Job aborted due to stage failure:
Exception while getting task result: scala.MatchError: None (of class scala.None$)
STARK-9337 An issue with the archiving process occasionally getting stuck due to a lack of resources is now resolved through better allocation of resources on install.
STARK-9316 Querying a metric that had previously changed schema no longer returns an empty result set.
STARK-9269 Added retry and wait steps to the install, providing more robustness when installing Gateway Hub over an unreliable connection.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-9730 The operating system language must be set to English, otherwise the build will fail.
STARK-9515 Data with a unit of measure assigned (for example, Kilobytes) is currently published in the canonical unit (for example, Bytes). This means that certain metrics may have a misleading name (for example, KilobytesInQueue).
STARK-9095 Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:
fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-7596 The SSL certificate install incorrectly uses the default ‘hub’ user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:
"ansible" : { 
   "variables" : { 
  "hub_user":
    "gwh", 
    "hub_group":
    "gwh", 
  } 
}
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.3.0 Copied

Released: 14 June, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Metric Query API Copied

Metric data can now be retrieved using an open REST API call:

POST /metrics/query

This endpoint can be used to retrieve metric data, grouped by time bucket with a number of built in aggregations such as median, mean, and percentiles, supporting simpler display of large amounts of monitored data. As of version 1.3 this endpoint is tuned for reporting and results can take several minutes, which is a substantial improvement over querying a relational database. Performance will be improved further in a future release.

A list of the aggregation functions is available using the following REST API call:

GET /metrics/aggregations

Entity directory Copied

This release marks the first version of the Entity directory, providing the basis for the next generation of the state tree. The entity directory brings a number of significant advantages over the existing Geneos state tree, including:

The entity store is an important part of our long term goal to simplify the data model, in particular removing Gateways and Netprobes from the XPath hierarchy and moving towards dealing with Entities and Metrics only. This will have a number of benefits, including easier configuration of Self-announcing Netprobes that rebalance across different Gateways frequently, as well as ultimately simplifying the way monitoring is configured in Geneos. Although the existing state tree will remain available for existing setups, over time we anticipate you will choose to switch to the simpler model.

In 1.4, the Entity Directory will also serve as the backbone for the new web-based monitoring UI.

Centralised Gateway configuration Copied

Gateway Hub can now store and serve Gateway configuration centrally. This version allows include files to be uploaded to Gateway Hub and used by connected Gateways. This is the first step in the journey to significantly simplify administration of Gateways at scale. More features will be added in the coming release, including centralised validation and locking, as well as administration UI screens in the Web Console.

For more information about this feature, see Centralised Gateways User Guide.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-8477 JMX instrumentation by various components has now been secured.
STARK-8315 Processing of schema messages from the Gateway has been substantially improved.
STARK-8248 The style for snackbar error and information messages in the Web Console has been updated.
STARK-8020 Web Console has improved handling of concurrency when changing configuration.
STARK-7913 Added a confirmation modal when leaving a Web Console page with errors.
STARK-7832 Improved design for system and ingestion error pages on mobile devices for the Web Console.
STARK-7713 In the Web Console, save buttons are now disabled in the presence of validation errors.
STARK-7628 Improved memory management for various Gateway Hub components.
STARK-7625 The Archiving jobs have been combined into a single Spark job for increased performance.
STARK-7623 A ITRS-Doc-Version HTTP header is required on all PUT endpoints, preventing concurrency issues. GET endpoints now return the current document version.
STARK-7302 Password field types in the Web Console have been improved to allow better password input.
STARK-7250 The documentation link in the About page is now correctly linked to the current Web Console version.
STARK-7228 The Web Console will now preserve the collapsed/expanded status of the sidebar between sessions.
STARK-7182 The status of Kafka publishing is now displayed in the publishing screen in the Web Console.
STARK-7132 Web Console now handles an unlicensed system better.
STARK-7074 Ports are now configurable for Kafka, REST API, Web Console, SSO API, and MCS monitoring UI.
STARK-6877 DD MMM YYYY has been added to the list of supported date formats in the Web Console.
STARK-6688 MapR Enterprise Components (MEP) have been upgraded to 6.1.
STARK-6686 The MapR instance underpinning Gateway Hub has been upgraded to 6.1.
STARK-6526 System errors are no longer displayed alongside ingestion errors and have their own endpoint and Web Console screen instead.
STARK-6525 The Kafka offsets have been removed from ingestion errors and replaced with the raw message, so that older errors did not become outdated by Kafka rebalances and retention settings.
STARK-6520 The ingestion errors pages in the Web Console have been reworked for significantly better performance and robustness. Outdated links to old errors that have been resolved are now handled correctly.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-8991 Dataview columns and headline cells with a full stop/period in the name no longer cause an error when queried for metrics.
STARK-8931 Dataview columns with backticks (`) in the name no longer cause the archiver to crash.
STARK-8668 Updating a metric’s type no longer causes the metric to be marked as deleted.
STARK-7617 Various improvements to the way Zookeeper is configured to improve stability.
STARK-7585 The /dataview endpoint has been updated to include a path parameter, allowing filtering by partial dataview path.
STARK-7227 The Web Console sidebar will no longer get stuck when collapsed under certain circumstances.
STARK-7174 Special characters (for example, ñ) are now handled correctly in Licence page of the Web Console.
STARK-7116 When publishing is disabled, schema and snapshot requests are no longer attempted.
STARK-6726 Column name escaping has been improved, preventing crashes when querying data from certain plug-ins.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-9095 Occasionally a container synchronisation issue occurs during an upgrade, resulting in the following error message:
fatal: [ec2-34-247-183-246.eu-west-1.compute.amazonaws.com]: FAILED! => changed=true 
attempts: 30 
cmd: /opt/mapr/server/mrconfig info containers resync local 

If this occurs, resync the containers manually using the following command:

/opt/mapr/server/mrconfig info containers resync local
STARK-7691 Due to a bug with Spark SQL, metric history cannot be retrieved from dataview columns with a semicolon in the name e.g. “test;”. To resolve this issue update the dataview column name removing the semicolon.
STARK-7596 The SSL certificate install incorrectly uses the default ‘hub’ user when applying an SSL certificate. To specify a different install user please add the following to the hubctl installation file:
"ansible" : { 
   "variables" : { 
  "hub_user":
    "gwh", 
    "hub_group":
    "gwh", 
  } 
}
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named “sampleTime”, “netprobeTime” or “dataViewId”.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.
STARK-4350 Data that is received without a timestamp has a default timestamp incorrectly applied with the timezone of the local machine.

Gateway Hub 1.2.0 Copied

Released: 11 March, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Kafka publishing Copied

The Kafka Publishing available with Gateway Hub builds upon the feature previously available in the Gateway. With Gateway Hub, schemas can now be published along with the metric and event data, in order to provide easier integration with other applications, and the published data is now normalised, enabling simpler usage downstream.

Web Console Copied

This release brings the new administration screens for the Web Console, allowing licence management and configuration of Publishing.

Breaking changes Copied

These are the breaking changes in this release:

Issue key Breaking change description
STARK-6346 Corrects the spelling of the “licence” keyword in the installation json file to UK English. Update the previous installation files to reflect this change.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-6860 When the Gateway Hub is not licensed, a placeholder appears in the Web Console.
STARK-6733 Dataview schemas can be requested to be published to Kafka using a REST command to Gateway Hub.
STARK-6732 Dataview snapshots can be requested to be published to Kafka using a REST command to Gateway Hub.
STARK-6619 Updates Web Console favicon.
STARK-6361 Gateway Hub can be used without saving metric and event data to disk.
STARK-6346 Standardises the use of the noun “licence” to UK English.
STARK-5983 Gateway Hub licences can be uploaded directly in the Web Console.
STARK-5980 Publishing to Kafka can be configured using a new Web Console screen.
STARK-5913 Sets the cluster name to hub.itrsgroup.com.
STARK-5613 Nodes can be added to the cluster.
STARK-4749 Dates in the Web Console, such as 20 September 2018, are now localised correctly.
STARK-4403 Allows you to navigate Web Console UI entirely by the keyboard.
STARK-4308 Upgrades Kafka to 2.0.
STARK-3270 A screen showing the current installed version of Gateway Hub is available, along with a control to download the diagnostic info if required for a support ticket.
STARK-3269 A new Web Console page for viewing and uploading Gateway Hub licences.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-6431 The installation script validates that the runtime user is assigned to the correct group before the installation starts.
STARK-6423 The installation script validates that the install user is configured correctly before the installation starts.
STARK-5922 Changing the units of measures no longer prevents ingestion errors from being returned.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-7368 While performing an upgrade from 1.1 to 1.2, there may be some errors in the upgrade log. This is due to a change in the message format and will not affect metric ingestion. The error messages can be ignored.
STARK-7118 If a snapshot is requested from Gateway with an unrecognised snapshot type, then the request is accepted by Gateway Hub but ignored by the Gateway. The error appears in the Gateway log.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

Gateway Hub 1.1.0 Copied

Released: 21 January, 2019

This release delivers the updated binaries for in the ITRS Downloads page.

Highlights Copied

These are the highlights of this release:

Historical metric and event storage Copied

Historical metric and event data can now be stored in Gateway Hub and retrieved using Active Console. This is significantly more reliable than database logging and is designed to handle all metrics being logged, rather than a subset.

Metric and event storage in Gateway Hub also provides the foundation for improved monitoring over the next few releases, including adaptive rules (anomaly detection), automatic baselining, predictive and causal analysis through machine learning.

Web Console Copied

his release brings the first release of the new web-based UI for Geneos, the Web Console, focusing on administration of Gateway Hub. The Web Console will be expanded upon in future releases to initially complement, and eventually replace, Active Console.

Breaking changes Copied

Note

These changes are applicable to Early Access customers only.

These are the breaking changes in this release:

Issue key Breaking change description
STARK-5233 The unit of measure endpoints have been simplified significantly.
STARK-3441 Dataview was previously written using incorrect case (e.g. “dataView”) in the REST endpoints. These have been corrected.
STARK-1566 Publishing to Gateway Hub is done using a secured connection. All publishing Gateway must be reconfigured to use SSL encryption as unsecured connections are no longer supported.

New features and enhancements Copied

These are the new features and enhancements of this release:

Issue key Release description
STARK-6565 Bundled SSO Agent upgrades to version 1.3.0.
STARK-6538 Increases message throughput due to improved MapR-DB batching strategy.
STARK-6344 Gateway Hub configuration can be updated through the command line.
STARK-6326 Kafka disk location can be specified as part of the installation parameters.
STARK-6316 Better validation when adding units of measure without a locale or text.
STARK-6191 Perfmon errors are correctly converted to null.
STARK-6167 Keystore password is configurable.
STARK-6007 Error codes are updated to make them unique.
STARK-5896 The kernel settings vm.swappiness, vm.overcommit.memory, and net.ipv4.tcp_retries2 are no longer modified as part of the install and must be managed separately as a prerequisite.
STARK-5768 Removes the need to store runtime user credentials for normaliserd.
STARK-5707 Dataview errors can be filtered by error code.
STARK-5591 Ingestion errors can be filtered by partial or full dataview path.
STARK-5582 Adds filter and skip parameters to ingestion errors endpoint to allow paging.
STARK-5541 Truncation of decimal data when attempting to save as an int generates ingestion errors.
STARK-4834 Passwordless SSH is no longer required. During installation, you can interactively enter the password.
STARK-4547 The installation is done using an offline package. Gateway Hub nodes do not longer require internet access.
STARK-4545 After installation, you can enter a custom SSL certificate.
STARK-3637 Web Console is packaged with Gateway Hub.
STARK-3020 Retention policies are now specified for metric and events.

Issues fixed Copied

These are the issues we have fixed in this release:

Issue key Release description
STARK-6736 Error messages with missing parameters can cause the error endpoints to fail.
STARK-6486 Zookeeper status is not reset correctly during factory reset.
STARK-6480 Logs entries are stored in UTC.
STARK-6290 Validation errors updated with unique titles.
STARK-6287 Better validation when uploading invalid licences.
STARK-6246 Duplicates unrecognisedField error message suppressed.
STARK-6237 Adds command line flag for generating faster diagnostic files.
STARK-6042 Resolves race condition causing “Unhandled Error: Failed to create table” log entries.
STARK-6039 Processing error messages no longer significantly slows down data ingestion.
STARK-6005 Systems with umask set to 077 no longer cause issues during installation.
STARK-5886 Simplifies user model to avoid permission issues in persistence.
STARK-5839 Fixes to skip and limit parameters for errors endpoint.
STARK-5783 Changes to Kafka ISR settings significantly improve resilience.
STARK-5751 Ingestion errors are no longer aggregated by dataview row name.
STARK-5283 Dataview queries return the most recent results first. If a limit is specified, the least recent results may be truncated.
STARK-5203 If a dataview had a computed row that was later removed, then the Gateway will now correctly report hasEnrichedRows as false.
STARK-5008 Changes to the casing of certain error parameters means that old error results will be cleared. These are repopulated as new errors are received and will not be an issue.
STARK-4965 Fixes an issue causing the error endpoint to return no data.
STARK-4957 Log files roll over correctly.
STARK-4517 Restarting Gateway Hub no longer causes Kafka to become corrupted under certain circumstances.
STARK-3962 Moves Kafka logs to the correct log location.

Known issues Copied

These are the known issues affecting this release:

Issue key Known issue description
STARK-7239 4.10 Gateways are not able to publish to Gateway Hub 1.1. This will be resolved as a matter of priority.
STARK-6975 Immediately after an upgrade from 1.0.3 to 1.1, the dataview query API returns empty results. This issue is resolved automatically after a few minutes and normal results will be returned.
STARK-6505 Gateway Hub will not correctly handle a dataview defined in a Gateway with columns named sampleTime, netprobeTime, or dataViewId.
STARK-4463 Dataview data from additional rows computed in the Gateway is not stored for regular/non-computed columns.

Other Gateway Hub releases Copied

Release Notes Release Date
2.x Release Notes Released: 8 April 2020
1.x Release Notes Released: 21 January 2019

Disclaimer

The information contained in this document is for general information and guidance on our products, services, and other matters. It is only for information purposes and is not intended as advice which should be relied upon. We try to ensure that the content of this document is accurate and up-to-date, but this cannot be guaranteed. Changes may be made to our products, services, and other matters which are not noted or recorded herein. All liability for loss and damage arising from reliance on this document is excluded (except where death or personal injury arises from our negligence or loss or damage arises from any fraud on our part).
["Gateway Hub"] ["Release Notes"]

Was this topic helpful?