Multiple pods failed to start or in CrashLoopBackOff state due to DNS errors
Related to Copied
CoreDNS
Problem Copied
The ITRS Analytics web console may become inaccessible. When checking the ITRS Analytics debug shell, multiple pods failed to start or in CrashLoopBackOff state. The log files for pods may show error symptoms on resolving DNS hostnames.
User may want to check the “platformd” pod status and its log file.
kubectl get pods -n kotsadm | grep platformd
kubectl logs platformd-<pod-id> -n kotsadm
This is an excerpt of the DNS error that may return in the log.
java.lang.IllegalArgumentException: Environment variable STATSD_SERVER value 'statsd-internal.kotsadm.svc.cluster.local' is not a valid address or hostname
Possible causes Copied
- Root Cause 1:
The CoreDNS pod in Kubernetes may have failed or does not respond properly.
- Root Cause 2:
The node may have antivirus or security tools running that interfere Kubernetes operations.
- Root Cause 3:
The node may have SELinux enabled in enforcing mode.
- Root Cause 4:
Certain Linux kernel parameters may be changed unexpectedly that affects Kubernetes operations.
Possible solutions Copied
- Solution Root Cause 1:
User may check the coredns pod status and its log file.
kubectl get pods -n kube-system | grep coredns
kubectl logs coredns-<pod-id> -n kube-system
If it fails to start or the log contains unexpected timeouts or errors, you may consider restarting the pod.
kubectl rollout restart deployment/coredns -n kube-system
- Solution Root Cause 2:
User may check the node for security software or tool installed or running. Please contact our support teams and provide a support bundle (details in link provided under below section) for further investigation.
- Solution Root Cause 3:
User may execute the below commands to verify SELinux status.
sestatus
getenforce
cat /etc/selinux/config
Please liaise with the IT security or related teams in your organization if SELinux settings need to be updated.
- Solution Root Cause 4:
User may verify the net.ipv4.conf.default.rp_filter and net.ipv4.conf.all.rp_filter settings. The values should be set to 2. These kernel parameters affect routing of packets between network interfaces created by Kubernetes.
Please note that Embedded Cluster already sets rp_filter sysctl parameters in /etc/sysctl.d/99-embedded-cluster.conf file. The parameters may have been overridden in a different conf file in /etc/sysctl.d/ or in /etc/sysctl.conf. You may look for clues with the below command.
grep rp_filter -R /etc/sysctl*