Results Exporter

Opsview Results Exporter exports results to services such as Splunk, syslog, local files, and arbitrary HTTP endpoints.

Known issue

When logging data to Rsyslog via TCP, the first message has the correct priority, but all subsequent messages use the user.notice priority.

Installation

The Opsview Results Exporter component can be installed on the Orchestrator system.

To install the Results Exporter:

# Debian
sudo apt-get install opsview-results-exporter
 
# RPM
sudo yum install opsview-results-exporter

Next, ensure that your component configuration (see Configuration section below) is updated to contain the correct:

• Message Queue encoder key and password in the results_queue and command_queue sections.
• Database credentials in the runtime_database section.
• Registry password in the registry section.

These will be the values set when deploying your Opsview system, as specified in /opt/opsview/deploy/etc/user_secrets.yml.

# Once the dependencies are installed, configured and running, 
# run the following command to allow opsview-monit to monitor
# the new component
sudo -iu root /opt/opsview/watchdog/bin/opsview-monit reload

Configuration

The opsview-results-exporter component ships with a default configuration file (/opt/opsview/resultsexporter/etc/resultsexporter.defaults.yaml). Any changes to this file will be overwritten on package updates so you should not edit this file if you want to keep your changes across package updates.

Create /opt/opsview/resultsexporter/etc/resultsexporter.yaml for any custom settings if it does not already exist. resultsexporter.yaml is not overwritten across package updates and overrides settings in resultsexporter.defaults.yaml.

With resultsexporter.yaml permissions, ensure /opt/opsview/resultsexporter/etc/resultsexporter.yaml is accessible by the opsview group with read permissions. For example:

chown root:opsview /opt/opsview/resultsexporter/etc/resultsexporter.yaml
chmod 640 /opt/opsview/resultsexporter/etc/resultsexporter.yaml

Example contents of /opt/opsview/resultsexporter/etc/:

drwxr-x--- 2 root opsview 4096 Jan 21 11:53 .
drwxr-x--- 7 root opsview 4096 Jan 21 11:31 ..
-rw-r----- 1 root opsview  625 Jan 18 16:33 resultsexporter.defaults.yaml
-rw-r----- 1 root opsview 2343 Jan 21 17:04 resultsexporter.yaml                   <----- make your changes here

The following options can be overridden in resultsexporter.yaml:

• results_queue — results message queue configuration.
• command_queue — command message queue configuration.
• runtime_database — runtime database connection configuration.
• outputs — list of outputs to export the results to. Results can be exported via syslog, file, and HTTP.
• registry — connection configuration for the registry.
• default_fields — default message fields to export. See Field Mapping.
• default_filter — default filter that is applied to messages. See Filtering.

Credentials

Ensure that your component configuration has the correct Message Queue encoder key, password, database password, and registry password during installation. These values were generated when you deployed Opsview and can be found in /opt/opsview/deploy/etc/user_secrets.yml.

When overriding settings in resultsexporter.yaml, refer to the structure used in resultsexporter.defaults.yaml. For example, to override the results Message Queue or registry settings within resultsexporter.yaml:

resultsexporter:
    ...
    results_queue:
        messagequeue:
            encoder_key: ZW5jb2RlcmR1bW15a2V5MTIzNA==
            host: 127.0.0.1
            port: 35672
            management_port: 45672
            user: opsview
            password: opsview
 
    registry:
        host: 127.0.0.1
        port: 12379
        user: opsview
        password: opsview

Refer to the examples as described on Exporting Results on how to add your own outputs.

Advanced Configuration

Worker settings

The number of parallel component workers, and the timeout before component worker restart (if encountering errors) can be set as follows:

resultsexporter:
  ...
  worker_timeout: <secs>        # default 30
  initial_worker_count: <num>   # default to 0, which will use the number of CPU cores

Note the following considerations:

• On a multi-core system, the more workers in use, the lower the load and elapsed time will be, with an upper bound on the number of cores available in your system.
• The more workers in use, the greater the overall component memory usage will be used.
• The memory usage of workers is multiplicative with the number of outputs configured.

Data Cache settings

The Results Exporter can cache Opsview data if the dal_fetchall or dal_fetchone field operations are in use (see Exporting Results).

Override the data_access_cache section in the configuration file for full control over the cache settings:

resultsexporter:
  ...
  data_access_cache:
    - <data model type>:        # e.g. RuntimeHosts
        cache_items: <yes|no>   # whether to cache data for the model
        max_items: <num>        # if caching, the maximum number of items to cache (if field unset, defaults to no limit)
        max_item_size: <num>    # if caching, the maximum size of any cached item (if field unset, defaults to no limit)
        pre_load: <yes|no>      # whether to pre-load data on component start up (default no)
        refresh_on: reload      # set if the cache should be cleared for the model when Apply Changes completes

Note the following considerations:

• The maximum number of items refers to the number of individual queries (for example, individual field operation executions after message fields are populated) that can be cached at once. For example, if a query was configured to run using a message field and it was ran against 2 messages with different values for that field, it would count as 2 separate cache items.
• The maximum size of each item refers to the number of data model results the cache will accept. For example, if a query matches 2 RuntimeHosts in the system, it would be a cache item of size 2.
• Setting a maximum number or size on items will reduce the memory usage of the component, but it will increase the load and elapsed time to process individual messages.
• Enabling preloading will increase load and elapsed time on component startup (and when Apply Changes occurs, if refreshing on reload is set), but it will decrease load and elapsed time during processing.
• Using message filters to decrease the number of messages processed will decrease the overall load and elapsed time to process all messages.

By default, the component uses the following cache configuration:

opsview_results_exporter_data_access_cache:
  - RuntimeHashtags:
      cache_items: no
  - RuntimeHosts:
      cache_items: yes
      max_items: 8_000
      max_item_size: 1
      pre_load: yes
      refresh_on: reload
  - RuntimeServicechecks:
      cache_items: yes
      max_items: 32_000
      max_item_size: 1
      pre_load: yes
      refresh_on: reload

The component will generate a warning if it encounters an unexpected model type.

Data from unsupported model ... could not be queried from cache

If intentionally not caching a model type, then add a cache_items: no entry for the model to the cache settings to suppress the logging. This will require overriding the entire section and then adding your new entries.

Logging Settings

The Results Exporter logging settings can be set by overriding the following section:

resultsexporter:
  ...
  logging:
    sensitive_trace: <yes|no>   # enable low-level trace logging (default no)
    data_access_cache:
      collisions: <yes|no>      # whether to log data cache collisions (default no)
    loggers:
      opsview:
        level: NOTICE           # overall component log level

The sensitive_trace will only have an effect if the component is also set at the DEBUG log level.

Monitoring

To apply the self monitoring Host Template for this component, run the following:

host1:~# cd /opt/opsview/deploy/
# configure monitoring of the Opsview Results Exporter component
host1:/opt/opsview/deploy# ./bin/opsview-deploy lib/playbooks/setup-monitoring.yml

Management

Configuration

DPKGs

Watchdog service files are now managed by the package, doing a remove would leave the watchdog service file behind with a .save extension. Purging the package will remove it. The package managed config files are as follows:

/opt/opsview/watchdog/etc/services/opsview-resultsexporter.conf

RPMs

Watchdog service files are now managed by the package. Any modifications will be saved at upgrade and remove processes with the .rpmnew and .rpmsave extensions correspondingly.

/opt/opsview/watchdog/etc/services/opsview-resultsexporter.conf

Service administration

As root, start, stop and restart the service using:

/opt/opsview/watchdog/bin/opsview-monit <stop|start|restart> opsview-resultsexporter

Troubleshooting

Credentials

If you see this error message in the log, please check that the component configuration has the correct password for the registry. The Results Exporter will need to be restarted if any changes are made.

[ERR] RegistryError<status_code=401, error_code=110, index=0, message=u'The request requires user authentication', raw_message='{"errorCode":110,"message":"The request requires user authentication","cause":"Insufficient credentials","index":0}\n'>

A message containing the following error will be seen on start-up if the Message Queue password is incorrect:

[ERR] Opsview Results-Exporter launcher failed to start: (libev) epoll_wait: Invalid argument

Messages containing the following errors will be seen when processing messages if the Message Queue encoder key is incorrect:

[ERR] Cannot verify HMAC signature! ...
[ERR] Dumping unparsable message: ...

If the database credentials are incorrect, the following error will be displayed on start-up:

[ERR] Failed to connect to Data Access Layer: (1045, "Access denied for user ...

Component restarting too frequently

The error message below indicates that the component is encountering a recurring error that is causing it to restart too frequently:

[ERR] Opsview Results-Exporter launcher failed to start: (libev) epoll_wait: Bad file descriptor

If this is the case, look for an earlier error message in the logs that matches one of the examples on this page.

Configuration file formatting

If you receive this error message, with the /var/log/opsview/opsview, please check the /opt/opsview/resultsexporter/etc/resultsexporter.yaml filter syntax is correct.

For the error below, an extra parenthesis was present. The opsview-resultsexporter will need to be restarted if any change is made.

Jul 22 12:02:24 orchestrator-main opsview : [ERR] Opsview Results-Exporter launcher failed to start: Output 'syslog.test': Filter Syntax Error - char: ')', line: '1,113'.