HTTP-JSON Plugin

Overview Copied

The HTTP-JSON plugin is a Collection Agent plugin that collects JSON data over HTTP. This plugin includes two separate collectors:

You can transform the returned JSON into a known schema for internal translation. To enable this, it is possible to specify a JQ program via an external file executed for each received JSON payload.

HTTP Security Headers Copied

Both collectors capture and publish HTTP security headers as Entity Attributes for compliance, audit, and security monitoring purposes. The following headers are collected when present:

The collectors publish these headers using the naming convention http_header_<header-name> (for example, http_header_authorization, http_header_user_agent).

Internal JSON schema Copied

High level description Copied

Regardless of the input, the output of the JQ program must output a JSON document that adheres to the following:

The following example shows a valid JSON document specifying a single gauge data point:

[
  {
    "name": "some.gauge.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "important.property": "something.important"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "gauge": {
      "value": 98.9,
      "unit": "%"
    }
  }
]

Formal internal JSON definitions Copied

“name” field Copied

The "name" field represents the name of the data point series. The combination of "name" and "dimensions" must be globally unique.

“dimensions” object Copied

The "dimensions" object represents the identity of the entity or resource emitting the data point series. The combination of "name" and "dimensions" must be globally unique.

Dimension ordering Copied

The plugin preserves the insertion order of dimensions as they appear in the JSON input. The plugin uses SequencedMap<String, String> for dimension storage, which maintains the order in which dimensions were added.

For example:

{
  "dimensions": {
    "datacenter": "us-west",
    "application": "web-service", 
    "cluster": "prod-cluster",
    "instance": "server-01"
  }
}

In this example, the dimensions are accessible in the exact order they appear in the JSON:

  1. datacenter (first)
  2. application
  3. cluster
  4. instance (last)

The plugin preserves this ordering throughout the data processing pipeline. You can rely on this ordering for consistent dimension iteration and display.

“properties” object Copied

The "properties" object represents additional information about the specific data point being exported.

“timestamp” field Copied

The "timestamp" field represents the sample timestamp of the specific data point. By default, timestamps must be in UTC and expressed in ISO-8601 format.

JQ has limited non-platform-specific date/time processing. If you cannot emit a timestamp in UTC and/or ISO-8601 format, you can use the following alternative formats:

For custom timestamp formats, if the timestamp represents non-UTC time, you can specify a timezone as either:

You specify the timestamp format as part of the jq section of the plugin’s configuration. Refer to the collector configuration sections for more information.

If no "timestamp" field is present, the collector uses the current time.

Type object Copied

The JSON object must contain exactly one field named from either one of the following:

Object Description
"gauge" "value": Floating point numeric (required). "unit": Unit symbol as defined here (optional).
"counter" "value": Fixed point numeric (required). "duration": The duration, in nanoseconds by default, over which the counter was incremented. The "timestamp" field represents the end of the timing window (optional). "unit": Unit of the "duration" field (optional). If specified, must be one of "ns", "ms", "s", "min", "h", "d". If you do not specify "duration", the plugin converts the counter into a gauge with no unit.
"status" "value": String representing the status.
"attribute" "value": Boolean, numeric, string, or list of strings (required). "unit": Numeric unit (optional).
"log" "severity": One of "none", "trace", "debug", "info", "warn", "error", "critical" (required). "message": Log message (required).

Example JSON document Copied

The following example shows a JSON document specifying data points (associated with the same entity/resource by dimensions) of each type:

[
  {
    "name": "some.gauge.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "some.property.key": "some.property.value"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "gauge": {
      "value": 98.9,
      "unit": "%"
    }
  },
  {
    "name": "some.counter.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "some.property.key": "some.property.value"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "counter": {
      "value": 98,
      "duration": 10,
      "unit": "s"
    }
  },
  {
    "name": "some.status.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "some.property.key": "some.property.value"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "status": {
      "value": "OK - nothing to see here"
    }
  },
  {
    "name": "some.numeric.attribute.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "some.property.key": "some.property.value"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "attribute": {
      "value": 128,
      "unit": "MB"
    }
  },
  {
    "name": "some.log.name",
    "dimensions": {
      "host.name": "some.host.com",
      "service.name": "some.service"
    },
    "properties": {
      "some.property.key": "some.property.value"
    },
    "timestamp": "2024-07-13T08:56:34Z",
    "log": {
      "severity": "info",
      "message": "Something happened"
    }
  }
]

Example JQ program Copied

Given the following JSON document:

[
  {
    "info": "opsview_resultsexporter 2024-07-16 08:08:48",
    "message": {
      "hostname": "opsview",
      "servicecheckname": "Opsview - Autodiscovery Manager - Status",
      "current_state": 0,
      "problem_has_been_acknowledged": false,
      "is_hard_state": true,
      "check_attempt": 1,
      "last_check": 1721117325,
      "execution_time": 0.214479,
      "stdout": "METRIC OK - CPU Usage is 0.00%, Memory Usage is 2.00%, Memory Used is 160.53MB, Child Count is 0, Uptime is 22h 42m ",
      "perf_data": {
        "CPU": "0.00%",
        "Memory": "2.00%",
        "Memory_Usage": "160.53MB",
        "Children": "0"
      },
      "metadata": {
        "hostname_run_on": "opsview-appliance"
      }
    }
  },
  {
    "info": "opsview_resultsexporter 2024-07-16 08:08:48",
    "message": {
      "hostname": "opsview",
      "servicecheckname": "Opsview - TimeSeries Enqueuer - Status",
      "current_state": 0,
      "problem_has_been_acknowledged": false,
      "is_hard_state": true,
      "check_attempt": 1,
      "last_check": 1721117325,
      "execution_time": 0.417679,
      "stdout": "METRIC OK - CPU Usage is 0.00%, Memory Usage is 0.70%, Memory Used is 56.36MB, Child Count is 1, Uptime is 22h 34m ",
      "perf_data": {
        "CPU": "0.00%",
        "Memory": "0.70%",
        "Memory_Usage": "56.36MB",
        "Children": "1"
      },
      "metadata": {
        "hostname_run_on": "opsview-appliance"
      }
    }
  }
]

You can use the following JQ program to translate it into the required internal format:

def get_timestamp:
  scan("\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}") |
  sub(" "; "T") |
  sub("$"; "Z");

def get_dimensions:
  if .servicecheckname == null
  then
    {
      hostname: .hostname
    }
  else
    {
      hostname: .hostname,
      servicecheckname: .servicecheckname
    }
  end;

def status_metric($timestamp; $dimensions):
  def get_name($default):
    if . == null then $default else . end;
  {
    name: .servicecheckname | get_name("host_check"),
    timestamp: $timestamp,
    dimensions: $dimensions,
    status: {
      # Currently, CA drops status metrics with values > 64 chars in length.
      value: .stdout | scan(".{0,64}")
    }
  };

def perf_metrics($timestamp; $dimensions):
  def get_gauge:
    . as $value_unit
      | ($value_unit | scan("[0-9.+-]+")) as $value
      | ($value | length) as $value_length
      | {
          value: $value,
          unit: $value_unit[$value_length:]
        }; 
  . | to_entries | .[] | {
    name: .key,
    timestamp: $timestamp,
    dimensions: $dimensions,
    gauge: .value | get_gauge
  };
  
.[] as $in
  | ($in.info | get_timestamp) as $timestamp
  | ($in.message | get_dimensions) as $dimensions
  | ($in.message | status_metric($timestamp; $dimensions)) as $status_metric
  | [
      $status_metric,
      ($in.message.perf_data | perf_metrics($timestamp; $dimensions))
    ]

Push collector configuration Copied

Configure the push collector as follows:

collectors:
  - type: plugin
    class-name: HttpJsonPushCollector
    
    # Optional but recommended name.
    name: http-push
    
    # Required. Port on which to receive JSON documents.
    port: 8080

    # Optional. Acceptor thread pool size (default = 2)
    acceptor-thread-pool-size: 2

    # Optional. Worker thread pool size (default = 4)
    worker-thread-pool-size: 4

    # Optional. URL suffix.
    # If specified, the publisher must publish using a URL ending in the specified path.
    path: /ingest
    
    # Optional. Acceptable HTTP methods. Defaults to [ POST, PUT ]
    # If specified, only JSON documents received via  one of the specified methods are accepted.
    methods: [ POST, PUT ]

    # Optional. Maximum content length. Defaults to 1 MiB. Must be in the range [1KiB, 8MiB].
    max-content-length: 1048576

    # Optional. Defaults to false.
    # When set, the HTTP 'Content-Type' header must be set to 'application/json' else requests are
    # dropped.
    require-content-type: false
    
    # Optional. JQ configuration.
    # If not specified then received JSON documents are passed straight through which implies
    # they must be produced externally in the correct format, including any timestamps in ISO-8601
    # format.
    jq:
      # Optional. Specify EITHER 'path' (external file) OR 'content' (inline), not both.
      # If neither is specified, received JSON documents are passed straight through
      # which implies they must be produced externally in the correct format.
      # Option 1: External JQ file
      path: ./path/to/jq_program.jq
      
      # Option 2: Inline JQ statement (use YAML multiline string with |)
      content: |
        .[] | {
          name: .metric_name,
          dimensions: .tags,
          gauge: { value: .value }
        }
      
      # The following options are used to customise the interpretation of timestamps output by
      # the JQ program in case it is not able to produce ISO-8601 / UTC timestamps.
      
      # Optional. Defaults to "iso-8601".
      # Valid options are: "iso-8601", "custom", "epoch-seconds", "epoch-millis", "epoch-nanos".
      timestamp-type: iso-8601
      # Optional. Must be specified when "timestamp-type" is "custom".
      # Must be a valid timestamp format (see below).
      timestamp-format: yyyy-MM-dd HH:mm:ss
      # Optional. May be specified when "timestamp-type" is "custom".
      timezone-id: "America/New_York"
    
    # Optional TLS configuration.
    tls-config:
      # Required.
      cert-file: /path/to/cert_file.pem
      # Required.
      key-file: /path/to/private_key.pem
      # Optional. Specify when mTLS is required.
      trust-chain-file: /path/to/trust_chain.pem

Pull collector configuration Copied

Configure the pull collector as follows:

collectors:
  - type: plugin
    class-name: HttpJsonPullCollector
    
    # Optional but recommended name.
    name: http-pull
    
    # Optional. Polling interval in milliseconds. Default is 60000 (1 minute).
    collection-interval: 60000

    # Required. List of targets to poll.
    # At least 1 polling target must be specified.
    targets:
        # Optional. Host component of HTTP URL. Defaults to 'localhost'.
      - host: localhost
        
        # Required. Port component of HTTP URL 
        port: 8080

        # Required. Path and query (if any) components of HTTP URL.
        path: /resource
        
        # Optional. HTTP method. Defaults to GET. Supports GET, POST, PUT
        method: GET

        # Optional. Request body (for POST/PUT requests). Ignored for GET.
        # Provide EITHER inline JSON in 'body.content' OR the path to a file in 'body.path', not both.
        # Default Content-Type is application/json unless overridden in headers.
        body:
          # Inline JSON body
          content: '{"query": "select * from metrics"}'
          # External file containing body
          #path: ./path/to/body.json
        
        # Optional. Custom request headers (e.g., for authentication, API keys)
        # Default headers set are Connection: close, Accept-Encoding: gzip.
        # Connection and Accept-Encoding can be overriden.
        headers:
          Authorization: "Bearer ${env:HTTP_PULL_BEARER_TOKEN}"
          X-API-Key: ${env:API_KEY}
          X-Custom-Header: "custom-value"

        # Optional. Connection and poll timeout in milliseconds. Default is 10000.
        timeout: 10000
        
        # Optional. Maximum content length. Defaults to 1 MiB. Must be in the range [1KiB, 8MiB].
        max-content-length: 1048576
        
        # Optional. JQ configuration.
        # If not specified then received JSON documents are passed straight through which implies
        # they must be produced externally in the correct format, including any timestamps in ISO-8601
        # format.
        jq:
          # Optional. Specify EITHER 'path' (external file) OR 'content' (inline), not both.
          # If neither is specified, received JSON documents are passed straight through
          # which implies they must be produced externally in the correct format
          
          # Option 1: External JQ file
          path: ./path/to/jq_program.jq
          
          # Option 2: Inline JQ statement
          content: |
            [.metrics[] | {
              name: .name,
              dimensions: .labels,
              gauge: { value: .value }
            }]

          # The following options are used to customise the interpretation of timestamps output by
          # the JQ program in case it is not able to produce ISO-8601 / UTC timestamps.
          
          # Optional. Defaults to "iso-8601".
          # Valid options are: "iso-8601", "custom", "epoch-seconds", "epoch-millis", "epoch-nanos".
          timestamp-type: iso-8601
          # Optional. Only specified when "timestamp-type" is "custom".
          # Must be a valid timestamp format (see below).
          timestamp-format: yyyy-MM-dd HH:mm:ss
          # Optional. Only (optionally) specified when "timestamp-type" is "custom".
          timezone-id: "America/New_York"
    
        # Optional TLS configuration.
        tls-config:
          # Required.
          cert-file: /path/to/cert_file.pem
          # Required.
          key-file: /path/to/private_key.pem
          # Optional. Specify when mTLS is required.
          trust-chain-file: /path/to/trust_chain.pem

Authorization Copied

Configure the authentication by setting the Authorization header under a target’s headers map.

Bearer authentication Copied 

headers:
  Authorization: "Bearer ${env:HTTP_PULL_BEARER_TOKEN}"

Basic authentication Copied

Provide basic authentication as a Base64-encoded token in accordance with RFC 7617. Use the following format for the Authorization header:

Authorization: Basic <base64(username:password)>

To generate the Base64-encoded token, use the following command:

echo -n 'username:password' | base64

Then reference the encoded value in your configuration:

headers:
  Authorization: "Basic ${env:BASIC_AUTH_B64}"
["Geneos"] ["User Guide"]

Was this topic helpful?