Configuration reference

Important

This information refers to the previous helm install method for the ITRS Analytics Platform. If you are looking to install using the more streamlined Kubernetes Off-the-Shelf (KOTS) method, see the updated deployment options.

ITRS provides sample configuration files for installing an ITRS Analytics instance. You can download sample scenarios in Sample configuration files.

Advanced configuration settings are all optional.

You are not required to change the settings described here for a default installation of ITRS Analytics.

Resource settings Copied

For each ITRS Analytics workload there is a resources parameter that defines the resource requests and limits for the pods in that workload. We recommend setting these parameters for all applications, otherwise some pods may consume all available resources on your Kubernetes cluster.

The provided configuration examples have baseline resources and limits set for all workloads. They can be altered as needed, for example:

kafka:
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "2Gi"
      cpu: "1"

ClickHouse volume sizing Copied

The current sample configuration files for fresh installs use clickhouse rather than timescale. Each ClickHouse workload has its own disk settings under clickhouse.platform, clickhouse.metrics, clickhouse.logs, and clickhouse.traces.

clickhouse:
  traces:
    diskSize: "80Gi"
  metrics:
    diskSize: "500Gi"
    diskCount: 4
    storageClass: "gp3-clickhouse"
  platform:
    diskSize: "100Gi"
    storageClass: "gp3-clickhouse"
  logs:
    diskSize: "500Gi"
    storageClass: "gp3-clickhouse"
  keeper:
    replicas: 3

Use diskSize to set the size of the persistent volume for a ClickHouse workload. Each ClickHouse workload also supports diskCount, which can be used to distribute data across multiple disks. In practice, this is most likely to be used for traces, metrics, and logs, but it is available for all ClickHouse workloads. You can also set storageClass per workload when ClickHouse data needs a different storage class from defaultStorageClass.

When clickhouse.metrics.diskCount is greater than 1, calculate the total metrics disk capacity per replica using the following formula:

clickhouse.metrics.diskSize * clickhouse.metrics.diskCount

The clickhouse.keeper.replicas setting controls the size of the ClickHouse Keeper cluster used by replicated ClickHouse deployments.

Note

The nginx sample configuration files still include commented timescale and loki sections for upgrades from pre-2.18 deployments. These are upgrade-only examples and are not used for fresh installs.

Kafka retention settings Copied

The kafka.defaultRetentionMillis parameter controls how long data is retained in Kafka topics, which defaults to 21600000 (6 hours).

ClickHouse on reserved nodes Copied

For larger deployments, you may want to run ClickHouse workloads on reserved Kubernetes nodes. The current medium and large sample configurations do this by using Kubernetes labels and taints on the nodes and nodeSelector and tolerations on the ClickHouse workloads.

  1. Add a label to the ClickHouse nodes in the manifest or by using kubectl:

    instancegroup: clickhouse-nodes

  2. Add a taint to the ClickHouse nodes in the manifest or by using kubectl:

    dedicated=clickhouse-nodes:NoSchedule

  3. Set the following for each ClickHouse workload that should run on the reserved nodes:

    clickhouse:
  metrics:
    nodeSelector:
      instancegroup: clickhouse-nodes
    tolerations:
      - key: dedicated
        operator: Equal
        value: clickhouse-nodes
        effect: NoSchedule

Apply the same pattern to clickhouse.platform, clickhouse.logs, and clickhouse.traces if those workloads should also run on the reserved nodes.

Default workload placement Copied

Use defaultNodeSelector and defaultTolerations when most operator-managed workloads should run on the same Kubernetes nodes.

For example:

defaultNodeSelector:
  role: iax
defaultTolerations:
  - key: dedicated
    operator: Equal
    value: iax
    effect: NoSchedule

platformd:
  nodeSelector:
    role: iax-control

Use workload-specific settings only when a workload must run on a different node pool from the shared default.

Warning

When you use nodeSelector or defaultNodeSelector with high-availability workloads such as Kafka, etcd, pgplatform, and ClickHouse, make sure the selector matches at least as many nodes as the replica count.

For zone-level redundancy, make sure the matching nodes also span multiple availability zones.

nodeSelector can neutralize topology spread constraints. When nodeSelector limits a workload to fewer eligible nodes than the replica count, Kubernetes evaluates topology spread only across those eligible nodes. If only one node matches the selector, all replicas can be scheduled onto that node without raising a scheduling error.

For example, Kafka can be configured with three replicas and a topology spread constraint on kubernetes.io/hostname. If nodeSelector: { role: iax } is set but only one node has the role=iax label, all three replicas can be scheduled onto that single node. If that node fails, all three replicas are lost at the same time.

Warning

If you add or change nodeSelector for stateful workloads such as Kafka, etcd, pgplatform, or ClickHouse, during an upgrade, make sure the selected nodes are in the same availability zone as the existing persistent volumes. Otherwise, the pods can remain in Pending state with a volume node affinity conflict.

To resolve this, update the nodeSelector so the workload is scheduled onto nodes in the same availability zone as the existing PVC. Deleting and recreating the PVC can also remove the conflict, but doing so deletes the existing persistent data and results in data loss.

Ingestion services Copied

ITRS Analytics supports ingesting data via two gRPC endpoints, which are both serviced from the same external host name (as defined by ingestion.externalHostname) and port (443).

The current sample configuration files tune ingestion by setting ingestion and ingestion.traces resources and producer settings, for example:

ingestion:
  externalHostname: "iax-ingestion.mydomain.internal"
  ingress:
    className: "nginx"
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  traces:
    sampler:
      expectedSpanRate: 20000
    producerProperties:
      buffer.memory: 67108864

If your deployment ingests OpenTelemetry traces, review the ingestion.traces settings in the sample configuration that most closely matches your expected load. The larger sample configurations also override ingestion.traces.resources to account for higher span rates.

OpenTelemetry ingestion service Copied

While raw traces are currently not stored, the OpenTelemetry ingestion service in the ITRS Analytics platform captures the following metrics (where <span> is the span name):

When pointing your instrumentation to the ITRS Analytics OpenTelemetry API, make sure to:

For more information, refer to the instrumentation guide from Open Telemetry.

Kubernetes log collection Copied

ITRS Analytics is configured to collect logs for Kubernetes pods and containers in the configured namespaces, or the ITRS Analytics namespace by default. This behavior is controlled by the following setting:

collection:
  logs:
    enabled: true
    # Restrict collection to specific namespaces. If empty, only the ITRS Analytics installation namespace
    # will be collected. To collect all, specify "*".
    namespaces: []

Note

For the log collection process to function properly, the obcerv-operator Kubernetes service account must have appropriate permissions to create resources with read and write access to hostPath volumes. The process for granting these permissions may vary depending on your Kubernetes distribution.
["ITRS Analytics"] ["User Guide", "Technical Reference"]

Was this topic helpful?