What can cause pods to become Evicted

Related to Copied

ITRS Analytics, Evicted, Eviction

Problem Copied

When user looks at the ITRS Analytics web console, some pods are not running under Ingestion => Workloads.

User may find the pods in Evicted status when checking the ITRS Analytics debug shell.

sudo ./itrs-analytics shell
kubectl get pods -A

Possible causes Copied

The current versions of ITRS Analytics are built on top of Kubernetes. If the node is under resource pressure, some pods may be terminated to reclaim resources and prevent starvation. The resources that are generally involved could be memory, disk space, and inodes.

Possible solutions Copied

User should review the default eviction thresholds and resources available on the nodes where ITRS Analytics is installed.

Start a proxy server using kubectl proxy in your terminal.

kubectl proxy

Open another terminal window and use curl to fetch the kubelet configuration. Obtain the node name and replace in the second command.

kubectl get nodes
curl -s -X GET http://127.0.0.1:8001/api/v1/nodes/<my_node_name>/proxy/configz | jq .kubeletconfig.evictionHard

The command returns the hard eviction thresholds in JSON format, which should look like this:

In particular, please note the disk space thresholds which could often be overlooked by users. We have provided link to Kubernetes documentation below for your reference.

Related articles Copied

• ITRS Analytics Resource and Hardware Requirements
• Node-pressure Eviction (linked to Kubernetes documentation)