Why do ITRS Analytics apps get stuck in CrashLoopBackOff after deployment?

When deploying multiple applications packaged together (for example, as part of the ITRS Analytics suite), all apps depend on a common Spring framework version. If one or more apps are built or deployed with a different Spring version, internal incompatibilities can occur.

These mismatches cause serialization failures within Spring’s internal state handling, leading to runtime errors during startup or inter-app communication. In such cases, you may observe multiple apps entering a CrashLoopBackOff state or see general failures where none of the dependent apps can initialize successfully. The logs of the app pods display exceptions similar to the following:

This issue can appear in both KOTS and Helm-based deployments.

What causes this issue? Copied

Different apps packaged with mismatched Spring framework versions can fail to serialize or deserialize shared internal state. This mismatch breaks the inter-app communication layer, preventing all apps from running correctly.

• One or more apps (for example, Alerting, IAM, Capacity Planner, and Dashboards) stuck in CrashLoopBackOff.
• Admin Console showing apps in Failed or Unready state.
• Log messages indicating Spring serialization or deserialization errors.
• Inconsistent startup behavior after deployment or upgrade.

How can I fix this in a KOTS deployment? Copied

If you’re using KOTS Admin Console, follow the steps below to redeploy with a clean state.

1. Delete all app deployments.

   kubectl delete -n itrs deployments -l app.kubernetes.io/part-of=obcerv
   

2. Delete all app persistent volume claims (PVCs):

   kubectl delete -n itrs pvcs -l app.kubernetes.io/part-of=obcerv
   

3. In the KOTS Admin Console, open your Currently Deployed version and click Redeploy.

4. Wait for the app to reach the Ready state (typically within 10 minutes).

How can I fix this in a Helm deployment? Copied

If you deployed using Helm and have several apps in CrashLoopBackOff, use these steps to clean up and redeploy.

1. List all pods for the affected label.

   kubectl get pods -l app.kubernetes.io/part-of=obcerv -n itrs
   

2. Delete all app deployments.

   kubectl delete deployments -l app.kubernetes.io/part-of=obcerv -n itrs
   

3. Delete all app persistent volume claims (PVCs):

   kubectl get pvc -l app.kubernetes.io/part-of=obcerv -n itrs
   kubectl delete pvc -l app.kubernetes.io/part-of=obcerv -n itrs
   

4. Redeploy each app using Helm.

   helm upgrade <app-name> itrs-snapshots/<app-name> \
     -f default-apps-nginx.yaml \
     --devel \
     -n itrs \
     --wait
   

   This should bring all apps back up successfully and resolve the CrashLoopBackOff issues.