OpenTelemetry

Overview Copied

The OpenTelemetry plugin is a Collection Agent plugin that provides data collection capability for the OpenTelemetry protocol over gRPC/protobuf.

The OpenTelemetry plugin requires an instrumented system that emits signals. The signals are the following telemetry categories that are supported by the OpenTelemetry specifications:

Prerequisites Copied

Geneos environment Copied

The latest version of the OpenTelemetry plugin requires Gateway and Netprobe version 6.0.x. The same version must be used for the GSE schema.

The OpenTelemetry plugin binaries are packaged with Netprobe 6.0.x and are stored in the collection_agent/plugins folder.

Alternatively, you can download separate binaries for the OpenTelemetry plugin from the ITRS Downloads.

OpenTelemetry environment Copied

The OpenTelemetry plugin requires an instrumented system that emits signals. Please see the OpenTelemetry documentation for language-specific instrumentation details.

Protocol and Security Copied

Protocol Copied

The OpenTelemetry plugin supports the following requests:

If the server responds with an OK status, then all records in the batch are ingested to persistent storage and will never require retransmission. There will be no retransmission either because they are already persisted or are considered non-critical.

The subset of gRPC error codes used are described in the table below:

gRPC code Retyrable Description
UNAVAILABLE Yes

An ingestion error has occurred and a retry is likely to succeed after a delay.

A RetryInfo record is also sent as additional details. The retry delay should be regarded as indicative of the initial delay period and clients should respect the suggested exponential backoff mechanism as per the protocol specification.

Caution

In some circumstances, gRPC emits UNAVAILABLE due to transport-related issues, such as bad TLS/SSL handshake. This is a non-retryable error but is outside the control of this plugin.
INVALID_ARGUMENT No An unknown error has occurred, and a retry of the same batch is unlikely to succeed and should not be attempted.
UNAUTHENTICATED No When an application authentication is enabled, and a client fails to authenticate.

Using the above mechanisms, the plugin attempts to guarantee at least once semantics. It is possible that retrying a batch will result in 1 or more individual records being ingested multiple times. You may minimize this possibility by sending records sequentially, one per batch. However, this is likely to be far less efficient from a network bandwidth and throughput perspective.

Security Copied

This plugin supports multiple layers of security:

Specifically, the following transport layer security modes are supported:

By default, when TLS is enabled, TLSv1.3 and TLSv1.2 are the only protocols allowed. It is possible to explicitly enable others, but they are generally considered less secure.

In addition to TLS, this plugin supports application layer authentication against a suitable IAM provider. In this case, aside from transport layer security, you must provide appropriate keys in the gRPC metadata, which can be used for authentication purposes.

Configure Gateway to receive OpenTelemetry data Copied

The OpenTelemetry plugin supports Collection Agent publication into Geneos using dynamic managed entities. To set up the OpenTelemetry plugin in Geneos, follow these steps:

  1. In GSE, go to Dynamic entitiesCollectors then create an OpenTelemetry collector with the following configuration:
    collectors:
    - type: plugin
        name: opentelemetry-otlp
        class-name: OpenTelemetryCollector
    
        # Optional. Default is 4317.
        port: 4317
    
        # Optional. gRPC inbound queue capacity. Default is 64.
        queue-capacity: 64
    
        # Optional. gRPC inbound queue processing threads. Default is 4.
        thread-pool-size: 4
    
        # Optional. Inbound username/password authentication against an OpenID IAM provider.
        # This mechanism is compatible with the internal IAM mechanism in the Obcerv Platform.
        authentication:
        # Mandatory. IAM provider credentials.
        iam:
            hostname: iam.domain.com
            port: 443
            client-id: ingestion
            client-secret: password
            admin-realm: master
            client-realm: obcerv
        # Optional. Username whitelist. If not specified then all users are allowed.
        allowed-users:
            - user1
            - user2
    
        # Optional. Use to enable TLS.
        # If omitted, the server accepts any connections over plain text.
        # Otherwise the server enables TLS and can either authenticate clients
        # (i.e. use mTLS) or not depending on whether the trust-chain-file is specified.
        tls:
        # Mandatory.
        cert-file: /path/to/cert.pem
        # Mandatory.
        key-file: /path/to/key.pem
        # Optional. Used only when mTLS is desired.
        trust-chain-file: /path/to/trust-chain.pem
        # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
        protocols: [ TLSv1.3, TLSv1.2 ]
    
        # Optional. Defaults to true.
        # By default, metrics from each trace span are extracted and emitted.
        # This flag allows toggling of that behavior.
        emit-trace-metrics: true
    
        # EXPERIMENTAL. Optional. Defaults to false.
        # By default trace spans are not emitted, however, this toggles
        # trace span emission. This is experimental because it only enables
        # reporting to another OpenTelemetry endpoint. That is, this is
        # useful only in relay mode.
        emit-traces: false
    
        # Optional. Default is 200.
        # For the resource and span attributes mapped (according to the rules below) that are mapped into dimensions,
        # both attribute names and values are truncated to the length specified in order to pass data point validation.
        # A value of 0 switches truncation off completely.
        max-attribute-key-value-length: 200
    
        # Optional. Defines which resource attributes to map to dimensions.
        # If not specified, then the default is to map all 'semantic' attributes
        # (see 'semantic conventions' above).
        resource-attributes:
        # Optional metric resource attributes to map to dimensions. Defaults to 'semantic'.
        metrics:
            # Optional. Defaults to 'semantic', which implies default to mapping any of the
            # resource attributes defined as having special semantic meaning
            # (see 'semantic conventions' above).
            # 'all' implies default to mapping all resource attributes to dimensions.
            # 'none' implies default to mapping no resources attributes to dimensions.
            base: semantic|all|none
            # Additional attributes to include.
            include:
            - custom.attribute
            # Attributes to exclude.
            exclude:
            - another.attribute
        # Optional log resource attributes to map to dimensions. Defaults to 'semantic'.
        logs:
            base: semantic|all|none
            include:
            - custom.attribute
            exclude:
            - another.attribute
        # Optional trace resource attributes to map to dimensions. Defaults to 'semantic'.
        traces:
            base: semantic|all|none
            include:
            - custom.attribute
            exclude:
            - another.attribute
    
        # Optional. Defines which scope attributes to map to dimensions.
        # If not specified the default is to map no scope attributes.
        scope-attributes:
        # Optional scope metric attributes to map to dimensions. Defaults to 'none'.
        metrics:
            # Optional. Defaults to 'none'.
            # 'all' implies default to mapping all resource attributes to dimensions.
            # 'none' implies default to mapping no resources attributes to dimensions.
            base: all|none
            # Additional attributes to include.
            include:
            - custom.attribute
            # Attributes to exclude.
            exclude:
            - another.attribute
        # Optional scope log attributes to map to dimensions. Defaults to 'none'.
        logs:
            base: all|none
            include:
            - custom.attribute
            exclude:
            - another.attribute
        # Optional scope trace attributes to map to dimensions. Defaults to 'none'.
        traces:
            base: all|none
            include:
            - custom.attribute
            exclude:
            - another.attribute    
    
        # Optional. Custom metric unit mappings from external string to a known
        # internal data model representation.
        units:
        # Map Otel metric unit 'By' to internal unit 'bytes'
        # See below for a reference to available internal units.
        By: bytes
    
    # EXPERIMENTAL. Only suitable for relaying traces to another OpenTelemetry endpoint.
    reporters:
    - type: plugin
        class-name: OpenTelemetryReporter
        name: otel-out
    
        # Optional. Reporting target hostname. Defaults to 'localhost'
        hostname: localhost
    
        # Optional. Reporting target port. Defaults to 4317.
        port: 4317
    
        # Optional. Switch compression on/off. Defaults to true (gzip).
        use-compression: true
    
        # Optional. gRPC call deadline in milliseconds. Defaults to 10000.
        call-deadline: 10000
    
        # Optional. Defaults to false.
        # When true, overrides any TLS configuration and uses a plain text connection.
        # When false (default), and there is no 'tls' section, default is to use
        # 'insecure' TLS (i.e. we trust the server certificate presented during TLS handshake)
        # over the more recent and supported TLS protocol out of TLSv1.3 and TLSv1.2.
        use-plain-text: false
    
        # Optional. Used only when use-plain-text is false (the default).
        # When not using plain text this section is only required when mTLS is desired,
        # else the client trusts whatever public key it receives from the server during the
        # TLS handshake.
        tls:
        # Optional. Used only for mTLS.
        cert-file: /path/to/cert.pem
        # Optional. Used only for mTLS.
        key-file: /path/to/key.pem
        # Optional. Used for mTLS and trusted server TLS (i.e. contains trusted server keys).
        trust-chain-file: /path/to/trust-chain.pem
        # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
        protocols: [ TLSv1.3, TLSv1.2 ]
    

    Note

    OpenTelemetry units may be translated to any available internal units, either by symbol or description. For more information, see Metric Unit Definitions.
  2. In Dynamic entitiesMapping, configure the mappings to be used. For an example mapping, see Mappings.
  3. If you will be monitoring logs, set up an FKM sampler. For more information, see File Keyword Monitor Configuration.
  4. In Dynamic entitiesMapping types, create a mapping type with the collector, mapping, and sampler you created in the previous steps.

Note

To check if there are any errors in the mapping, you can set up the Dynamic Entities Health or look at the Collection Agent log file in Collection Agent setup.

Mappings Copied

The data mappings of the signals supported by the OpenTelemetry plugin are as follow:

OpenTelemetry plugin allows you to create custom mappings, so you can modify the display of information in the Geneos dataview. The mappings can be configured differently by using available dimensions or properties.

To create custom mappings for OpenTelemetry:

  1. Navigate to your OpenTelemetry mapping in Dynamic entities > Mappings.
  2. In Options, select custom.

Example mappings and dataviews Copied

Traces Copied
<mapping name="opentelemetry-traces">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>trace_id_hex</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
   </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated:

open-telemetry-traces-dataview

Metrics Copied
<mapping name="opentelemetry-metrics">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>service.name</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
    </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated:

open-telemetry-metrics-dataview

Logs Copied
<mapping name="opentelemetry-logs">
 <custom>
  <filters>
   <filter>
    <label>__itrs_namespace__</label>
    <equals>
     <value>itrsgroup.com/c2/opentelemetry-plugin/traces</value>
    </equals>
   </filter>
  </filters>
  <geneosItems>
   <geneosItem>
    <label>service.name</label>
    <entity>
     <required>true</required>
     <useInDisplayName>true</useInDisplayName>
    </entity>
    </geneosItem>
  </geneosItems>
 </custom>
</mapping>

Using the custom mappings above, the following dataview is generated:

open-telemetry-logs-dataview

["Geneos"] ["Geneos > Netprobe"] ["User Guide"]

Was this topic helpful?