Getting Started with Collection Agent

Overview Copied

This guide outlines the step-by-step procedures you need to start running the Collection Agent in a standard environment.

Caution

Make sure to read Upgrading Collection Agent, which outlines the breaking changes that may have an impact on your upgrade, before you upgrade to the latest version of Geneos and Collection Agent.

For information on running the Collection Agent in an orchestrated environment, see Data collection in an orchestrated environment.

Note

Beginning Geneos 5.1, the Collection Agent is included in the Netprobe binaries for Windows and generic Linux platforms.

Prerequisites Copied

System requirements Copied

The following requirements must be met on the machine where the Netprobe and Collection Agent are installed:

Java requirements Copied

Geneos components requirements Copied

Choose how to run the Collection Agent Copied

Configure your Collection Agent depending on how you want to run it. You can run the Collection Agent in two ways:

Regardless of how you run your Collection Agent, it will always run as a Java process.

Running via Netprobe vs as a standalone Java process Copied

Using AWS monitoring as an example, the following sections compare running the Collection Agent via Netprobe versus as a standalone Java process.

Via Netprobe Copied

This path is the usual Geneos deployment, where the Collection Agent runs with the Netprobe on the same host, configuration is driven from the Gateway, and datapoints reach Geneos over the managed TCP path. Use it when you want AWS metrics displayed in the Active Console dataviews with Dynamic Entities.

  1. In Gateway Setup Editor, open Dynamic entities > Collection agent parameters, then New Collection agent parameters.
  2. In the Collection Agent dropdown, select Managed.
  3. In the Yaml dropdown, choose your YAML mode and complete the Collection Agent parameters (ports, paths, and any bundled or custom YAML your site uses).
  4. Add or enable AWS plugin JARs on the Netprobe host as needed. If a collector is not in the shipped package, download it from the ITRS Resources page and add it to the collection_agent directory on the Netprobe machine.

Then configure the AWS Collection Agent plugin itself and Dynamic Entities:

  1. Set up the Collection Agent plugin — define AWS collectors either in Dynamic entities > Collectors in Gateway Setup Editor or in collection-agent.yml on the machine, using the same collector types documented in Configure Geneos to deploy the AWS plugin. This includes full YAML examples for multiple collectors.
  2. Configure your mapping — use the template include templates/aws_mapping.xml, a built-in mapping, or a custom mapping under Dynamic entities > Mapping, so metrics and events appear correctly in the State Tree. See Configure your mappings.
  3. Complete Dynamic Entities — add any remaining Dynamic Entity configuration in the Gateway. For the detailed procedure, see Create Dynamic Entities.

As a standalone Java process Copied

You can run the Collection Agent without a Netprobe to pull AWS CloudWatch metrics through the Geneos AWS Collection Agent plugin.

Prerequisites:

The example below is a minimum configuration setup, following the same structure as the configuration reference above: pluginDirectory, a reporters entry with type: logging (each datapoint is written to standard output), a plugin collectors entry for AwsCollector, and workflow so the metrics pipeline sends data to that reporter. Collection is limited to EC2 in one region to minimize the CloudWatch API usage.

pluginDirectory: /opt/geneos/collection-agent/plugins

reporters:
  - type: logging
    name: logging
    level: info

collectors:
  - name: aws
    type: plugin
    className: AwsCollector
    regions:
      - us-east-1
    enabledServices:
      - AWS/EC2

workflow:
  storeDirectory: .
  metrics:
    reporter: logging

Run the Collection Agent with the same command, passing this file as the configuration argument.

java -jar collection-agent-<version>-exec.jar /path/to/collection-agent.yml

For full collector options, see the AWS Collection Agent plugin documentation.

Comparison summary Copied

Standalone Managed via Gateway Setup Editor
Where the Collection Agent runs Any host where you run java -jar and can be separate from the Netprobe Same host as the Netprobe
Where you define collectors Entire collection-agent.yml on disk (for example collectors with AwsCollector) Either YAML on the Netprobe host or collector definitions under Dynamic entities > Collectors in the Gateway Setup Editor
How datapoints leave the Collection Agent You choose the reporter (for example logging to stdout, or tcp to a Netprobe) Geneos wires the managed Collection Agent to the Netprobe
Gateway setup beyond the Collection Agent Optional (for example, mappings only if you use TCP and Dynamic Entities elsewhere) Collection agent parameters, Mapping and mapping types, and related Dynamic Entities are part of the normal setup

Create Dynamic Entities Copied

Netprobes receive datapoints from Collection Agents. Each datapoint can have a different set of dimensions, attributes, and properties. You should create mappings to construct a Geneos hierarchy from datapoints. This Geneos hierarchy determines how data is displayed in your Active Console. You should also create a mapping type, which allows you to easily deploy the same configuration on multiple probes.

  1. Open Gateway Setup Editor.
  2. Click Dynamic entities > Mapping, and then click New Mapping.
  3. Enter a mapping name in the Name field.
  4. In Options, choose custom or builtin. For more information, see Dynamic entities > Mapping > Options in Dynamic Entities.

    Note

    A built-in Collection Agent V3 mapping is packaged with the Netprobe, which automatically sets up the default mappings needed to display the Collection Agent self-monitoring metrics. Using this mapping requires setting the HOSTNAME as an environment variable to ensure that it returns valid metrics.

  1. Click Dynamic entities > Mapping types, and then click New Mapping type.
  2. If you configured a managed Collection Agent with a managed YAML type in the previous steps, then in the Collectors field, click Add new to add the name of the collectors you created in Run Collection Agent via Netprobe.
  3. In the Mappings field, click Add new to add the name of the mapping created in Step 4.

Mapping type

  1. To monitor the logs collected by the Collection Agent plugin, you can add an FKM plugin sampler in the Samplers field. The FKM plugin consumes these logs as streams. To configure your FKM plugin to monitor these streams, see files > file > source > stream in configuration.

Add Dynamic Entities in your Netprobe Copied

  1. Open Gateway Setup Editor.
  2. In Probes, select your Netprobe and go to the Dynamic Entities tab.
  3. Choose a mapping type in the Mapping type dropdown list.
  4. Choose a collection agent parameter in the Collection agent parameters dropdown list.

Collection agent parameters

Note

If mappings are defined in both the Mapping type and in the Collector, then the Netprobe will apply the mappings defined in the Mapping type first then the mappings defined in the Collectors.

Collection Agent self-monitoring metrics Copied

If you started the Collection Agent with the built-in Collection Agent V3 mapping, then the Collection Agent self-monitoring metrics are displayed in your Active Console.

Note

The HOSTNAME must be set as an environment variable to ensure that the built-in Collection Agent V3 mapping returns valid metrics.

  • On Debian-based systems such as Ubuntu, the HOSTNAME variable is typically set as a shell variable. Therefore, you must explicitly export it as an environment variable before running the Netprobe.
  • On RHEL-based systems, HOSTNAME is usually already set as an environment variable by default.

Setting the HOSTNAME as an environment variable Copied

There are two ways to set HOSTNAME as an environment variable:

Dynamic Entities Health Copied

The Dynamic Entities Health plugin monitors the performance of Collection Agent and Netprobe communication. It tracks the state of all metrics received by a Netprobe from the Collection Agent, and all dynamic entities associated with a Netprobe.

For guidance on how to set up the Dynamic Entities Health sampler, see Dynamic Entities Health.

Collection Agent log file Copied

The Collection Agent uses a separate log file from the Netprobe. The log settings for Collection Agent use Logback and are stored in logback.xml.

The logback has the following limitations by default:

You can use the following configuration:

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${COLLECTION_AGENT_DIR:-.}/collection-agent.log</file>
    <append>true</append>
    <immediateFlush>true</immediateFlush>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${COLLECTION_AGENT_DIR:-.}/collection-agent-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
      <maxFileSize>100MB</maxFileSize>
      <maxHistory>30</maxHistory>
      <totalSizeCap>10GB</totalSizeCap>
    </rollingPolicy>

For more information on the other available options for modifying the logger settings, see Logback.

By default, the Collection Agent log file is stored in the collection-agent.log of the working directory. If you are running a managed Collection Agent and you configured the Log directory parameter in GSE, then the collection-agent.log will be in that specified location.

Enable log masking Copied

You can enable log masking in Collection Agent by configuring Logback to use the Geneos log masking layout and a rules file.

The following example shows a Logback configured with log masking:

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${COLLECTION_AGENT_DIR:-.}/collection-agent.log</file>
    <append>true</append>
    <immediateFlush>true</immediateFlush>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${COLLECTION_AGENT_DIR:-.}/collection-agent-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
      <maxFileSize>100MB</maxFileSize>
      <maxHistory>30</maxHistory>
      <totalSizeCap>10GB</totalSizeCap>
    </rollingPolicy>
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
      <layout class="com.itrsgroup.platform.util.logging.logback.LogMaskingPatternLayout">
        <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n</pattern>
        <rulesFile>${COLLECTION_AGENT_DIR:-.}/log-masking-rules.yaml</rulesFile>
      </layout>
    </encoder>
  </appender>

The rulesFile element specifies the path to the log masking rules file. This configuration is specific to Logback-based components such as Collection Agent.

Example log-masking-rules.yaml:

  - name: FQDN_example_com_or_con
    regex: '(?i)\b(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+example\.(?:com|con)(?!\.[A-Za-z0-9-])\b'
    replacement: '[redacted-host]'

For more information, see Log masking.

["Geneos"] ["Geneos > Netprobe"] ["User Guide"]

Was this topic helpful?