Gateway Hub Overview

Quick Reference

This is a one-pager quick reference guide for Gateway Hub. For complete documentation, including detailed procedures, configuration options, and troubleshooting guides, refer to the full Gateway Hub documentation available in the offline documentation package.

Overview Copied

Gateway Hub is a core component of ITRS’ Geneos monitoring platform, designed to provide operational resilience through enhanced storage, analytics, and automation capabilities. It serves as a centralized data repository and management layer that sits alongside traditional Geneos components, enabling organizations to monitor at significantly larger scale with improved insights and reduced operational overhead.

Gateway Hub transforms real-time monitoring by providing scalable big data storage, historical analytics, and a modern web-based interface. It is built from the ground up to support elastic and cloud-based environments, making it ideal for modern enterprise monitoring requirements.

Key Features & Functionalities Copied

Scalable Data Storage Copied

Gateway Hub provides reliable, high-performance storage for all Geneos metrics and events using advanced compression techniques. Unlike traditional SQL database logging, Gateway Hub can store the complete dataset without forcing users to choose a subset of metrics, ensuring comprehensive incident investigation capabilities.

• Historical Data Retention — Store all metrics and events with configurable retention policies
• Advanced Compression — Minimize storage costs while maintaining data accessibility
• High-Throughput Ingestion — Handle large volumes of real-time monitoring data
• Data Retrieval — Access stored data via REST API for integration with external systems

Web Console Copied

Gateway Hub includes the Geneos Web Console, a modern, browser-based user interface that complements and will ultimately replace Active Console and Gateway Setup Editor.

Key Web Console Features:

• Real-time Monitoring — Monitor entity status and metrics with live updates
• Advanced Filtering — Use powerful filter syntax to find entities in large enterprise environments
• Dashboard Creation — Build custom dashboards with grouping and visualization capabilities
• Historical Analysis — View historical data for context and incident investigation
• Entity Overview — Detailed views of entity attributes, metrics, and events
• No Installation Required — Access via web browser without additional client software

Anomaly Detection & Analytics Copied

Leverage historical data to enable smarter monitoring rules and predictive capabilities:

• Baseline Establishment — Determine normal operating conditions using historical patterns
• Anomaly Detection — Identify deviations from established baselines
• Correlation Analysis — Detect relationships between metrics across entities
• Cyclical Pattern Recognition — Account for time-based variations in thresholds
• Incident Investigation — Analyze historical data to understand root causes

Centralized Configuration Management Copied

Simplify administration of large Geneos estates through centralized configuration:

• Centralized Gateway Configuration — Store and manage Gateway setup files in Gateway Hub
• Policy Enforcement — Apply best practices and policies at enterprise scale
• Simplified Administration — Reduce administrative burden for large monitoring deployments
• Version Control — Track configuration changes and maintain configuration history

Open APIs & Integration Copied

Gateway Hub is designed for interoperability and easy integration:

• REST API — Full programmatic access to Gateway Hub functionality and data
• Kafka Publishing — Publish normalized metric and event data to external Kafka instances
• Standardized JSON Format — Normalized data format for easy consumption by downstream systems
• Third-Party Integration — Connect external applications, visualization tools, and analytics platforms

Clustered Architecture Copied

Gateway Hub uses a distributed, clustered architecture for high availability and scalability:

• High Availability — Multi-node clusters provide redundancy and fault tolerance
• Horizontal Scaling — Add nodes to scale capacity as monitoring requirements grow
• Cloud-Ready — Built to support elastic and cloud-based environments
• Resilient Design — Automatic failover and recovery capabilities

Architecture Copied

Gateway Hub integrates seamlessly with existing Geneos components. The architecture follows this data flow:

1. Data Collection — Netprobes collect monitoring data from target systems
2. Gateway Processing — Geneos Gateways process and normalize the data
3. Hub Ingestion — Gateways publish data to Gateway Hub for storage and analysis
4. Data Access — Data is accessible via
   • Web Console for visualization and administration
   • Active Console for traditional monitoring views
   • REST API for programmatic access
   • Kafka publishing for downstream systems

Gateway Hub consists of multiple services running in a clustered configuration:

• PostgreSQL/TimescaleDB — Time-series database for metrics and events storage
• Kafka — Message streaming for data ingestion and publishing
• Zookeeper — Coordination service for cluster management
• etcd — Distributed key-value store for configuration
• Collection Agent — Collects metrics from Gateway Hub nodes
• API Service — REST API endpoint for external access
• Web Console — Web-based user interface

Operations & Management Copied

Essential hubctl Commands Copied

Configuration File Reference Copied

Gateway Hub uses a YAML configuration file for installation, reconfiguration, and upgrades. The configuration file structure:

Gateway Hub keeps a copy of configuration files used in each operation in /opt/hub/hub-current/etc/hub-installer/configuration-history on each node.

Service Validation Copied

Verify Gateway Hub service health:

Configuration & Integration Copied

Gateway Connection:

• Configure Gateway to publish to Gateway Hub (Gateway 4.10.x+)
• Use authentication: Kerberos keytab, API key, or SSO Agent
• Enable centralized configuration (Gateway 5.0+ on Linux, Gateway Hub 1.6+)

Self-Monitoring:

• Connect Gateway to internal Netprobes on each node (port 7036)
• Download and configure Gateway Hub self-monitoring include file
• Requires valid license with tokens for: kafka-plugin, prometheus-target-plugin, zookeeper-plugin, postgresql-plugin, linux-infra-plugin, system-plugin, StatsD-plugin

Authentication Setup:

• LDAP/SSO — Configure via Web Console > Administration > LDAP Config
• SAML — Configure via Web Console > Administration > SAML Config
• API Keys — Create via Web Console > Administration > Application Keys

Web Console Access:

• Default HTTPS port: 8443
• Default REST API port: 8081
• Access via: https://<hostname>:8443

Integration Copied

Gateway Integration Copied

Connect Geneos Gateways to Gateway Hub to enable:

• Data Publishing — Publish metrics and events from Gateway to Gateway Hub
• Centralized Configuration — Store Gateway configuration files in Gateway Hub
• Self-Monitoring — Monitor Gateway Hub performance from Gateway

Requirements:

• Gateway version 4.10.x or higher (for publishing)
• Gateway version 5.0 or higher (for centralized configuration on Linux)
• Gateway Hub version 1.6 or higher (for centralized configuration)

Authentication Options:

• Kerberos (using keytab files)
• API keys
• SSO Agent

For integration details, see Gateway Hub Quickstart.

External System Integration Copied

Gateway Hub provides multiple integration points:

REST API Integration:

• Access all Gateway Hub functionality programmatically
• Query metrics, events, and entities
• Manage configuration and administration tasks
• Integrate with custom applications and tools

Kafka Publishing:

• Publish normalized data to external Kafka instances
• Support for multiple security protocols (PLAINTEXT, SSL, SASL)
• Filter data by message type and entity attributes
• Enable downstream analytics and processing systems

Application Integration:

• Grafana — Connect Grafana to Gateway Hub for advanced visualization
• Capacity Planner — Use Gateway Hub data for capacity planning and forecasting
• Monitoring Coverage — Analyze monitoring coverage across your estate

Web Console Integration Copied

The Web Console provides integration capabilities:

• Dashboard Sharing — Share dashboards across teams
• Webhooks — Configure webhooks for event notifications
• SAML Authentication — Integrate with enterprise SSO providers
• LDAP Integration — Connect to LDAP directories for user management

Benefits Copied

Gateway Hub delivers significant value to organizations:

• Operational Resilience — Enhanced automation and predictive capabilities help prevent costly outages
• Scalability — Support for much larger estates and cloud environments
• Cost Reduction — Advanced compression and efficient storage reduce infrastructure costs
• Simplified Administration — Centralized configuration reduces administrative overhead
• Better Insights — Historical data enables smarter rules, baselines, and anomaly detection
• Modern Interface — Web-based console eliminates client installation and maintenance
• Open Architecture — APIs and Kafka publishing enable integration with existing toolchains
• Future-Proof — Foundation for future Geneos enhancements and capabilities

Troubleshooting Copied

Diagnostic Procedures Copied

Obtain Diagnostics:

Check Logs:

• Default log location: <hub_root>/logs (typically /opt/hub/logs)
• Log retention: 14 days by default, configurable
• Log file size: 250MB by default, configurable

Verify Service Health:

Common Issues Copied

1. Ingestion Errors:

• Access: Web Console > Administration > Ingestion Errors
• Resolve: Update schema in Gateway Setup Editor (Publishing tab > Schemas > Dataviews)
• Errors auto-delete after 7 days, or manually delete via PostgreSQL:

  \c hub
  DELETE FROM errors;
  

2. Gateway Connection Issues:

• Test Kafka connection: kafkacat -X security.protocol=ssl -X ssl.ca.location=<ca_cert> -b <host>:9092 -L
• Verify topics exist: geneos-events, geneos-metrics-v1
• Check network connectivity and firewall rules

3. Certificate Issues:

• Extract SAN from certificate: openssl x509 -in <cert_file> -text -noout
• Renew self-signed certificate: hubctl setup reconfigure <config_file>
• Certificate lifetime: 397 days (self-signed), CA certificate: 100 years

4. Kafka Message Size Limits: If Gateway setup validation fails due to large files:

• Increase max.request.size in API daemon Kafka producer config
• Increase fetch.max.bytes in gateway-configd Kafka consumer config
• Increase replica.fetch.max.bytes in Kafka server config
• Update topic config: max.message.bytes

5. PostgreSQL Database Restoration:

6. etcd History Pruning:

Upgrade Procedures Copied

Prerequisites Copied

• Installation machine with required tools
• All nodes reachable from installation machine
• Same Gateway Hub version on all nodes
• Valid installation configuration file
• Gateway Hub in healthy state (check self-monitoring metrics)

Upgrade Process Copied

1. If using systemd, mask orchestration service:

2. Perform Upgrade:

3. Unmask and restart (if using systemd):

Important Notes:

• Data Transformation — Upgrading transforms data to new format. Existing data is NOT preserved.
• Backup Required — Create manual backup before upgrading if you may need to downgrade
• Version Compatibility — Upgrades from pre-2.2.x to 2.2.x+ are not supported
• Gateway Include Files — Update Gateway self-monitoring include files after upgrade

Operating System Upgrade Copied

Configuration Management Copied

Web Console Configuration Copied

SAML SSO Setup:

1. Set Identity Provider metadata: hubctl config set --service-name webconsole --local-config-files saml_idp_metadata.xml <config_file>
2. Download Service Provider metadata: curl https://<host>:8443/saml/metadata -k
3. Add SP metadata to APID: curl -k -X PUT -H 'Content-Type: application/xml' "https://<host>:8081/v0/security/saml/serviceprovider/metadata" -d @saml_sp_metadata.xml
4. Set IdP metadata to APID: hubctl config set --service-name apid --local-config-files saml_idp_metadata.xml <config_file>

LDAP Configuration:

• Configure via Web Console > Administration > LDAP Config
• For secure LDAP (LDAPS), add CA certificate to Java keystore:

  keytool -importcert -alias MySecureLdap -keystore $JAVA_HOME/jre/lib/security/cacerts -file <ldap_ca_cert.cer>
  # Restart apid service
  /opt/hub/hub-current/tools/hub-admin/bin/hub-admin service restart -n apid
  

Kafka Publishing:

• Configure via Web Console > Administration > Publishing
• Create topics on downstream Kafka: ${prefix}metrics, ${prefix}events, ${prefix}entities
• Default prefix: itrs-
• Supports filtering by message type and entity attributes

Gateway Configuration Management:

• Upload Gateway binaries via Web Console > Administration > Gateway Config > Binaries
• Upload include files via Web Console > Administration > Gateway Config > Includes
• Manage Gateway resources and permissions via Gateways page

Service Configuration Copied

Edit Service Configuration:

Restart Services:

Operational Maintenance Copied

Regular Tasks Copied

Monitor Health:

• Check Gateway Hub self-monitoring metrics via Gateway/Active Console
• Review ingestion errors in Web Console
• Monitor disk space for logs and data directories
• Check cluster quorum status

Log Management:

• Default retention: 14 days
• Default max file size: 250MB
• Location: <hub_root>/logs (configurable)
• Total size cap: 1GB per service (configurable)

Backup Considerations:

• PostgreSQL data: <hub_data>/postgres-timescale/pgdata and pgwal
• etcd data: <hub_data>/etcd/data and wal
• Configuration files: Store installation YAML files securely
• Gateway Hub keeps configuration history in: /opt/hub/hub-current/etc/hub-installer/configuration-history

Capacity Planning:

• Use hubsize tool to estimate hardware requirements
• Monitor metrics retention (default: 90 days for metrics, 3 days for Kafka)
• Adjust based on workload and storage capacity

Key Ports Copied

• Web Console — 8443 (HTTPS)
• REST API — 8081 (HTTPS)
• Kafka — 9092
• Zookeeper — 5181 (client), 5188 (leader), 5189 (election)
• PostgreSQL — 5432
• etcd — 2379 (client), 2380 (peer)
• etcd Gateway — 2378
• Netprobe (internal) — 7036
• Collection Agent — 9097 (health), 8125 (statsd), 9137 (reporting)