Recovery steps for Embedded Cluster and BYO upgrade

If you performed an upgrade without enabling Dynamic Thresholds and your version displays a Deploy Failed status, please follow these recovery steps carefully.

1. Check if the clear-dpd-tasks-job is stuck or incomplete.

2. Run the following command to list all jobs in the kotsadm namespace.

   kubectl get jobs -n kotsadm
   

   If you see clear-dpd-tasks-job in the output with a status of Running, it is stuck or has not completed.

   NAME                                      STATUS     COMPLETIONS   DURATION   AGE
   clear-dpd-tasks-job                       Running    0/1           5m25s      5m25s
   embedded-cluster-upgrade-<id>  Complete   1/1           30s        7m47s
   

3. Use the following command to check the pod created by the clear-dpd-tasks-job job in the kotsadm namespace.

   kubectl get pods -n kotsadm | grep clear-dpd-tasks-job
   

   Example output:

   clear-dpd-tasks-job-cq4z6            0/1     ContainerCreating   0         12s
   

4. Delete the stuck clear-dpd-tasks-job.

   kubectl delete job clear-dpd-tasks-job -n kotsadm
   

   Example output:

   job.batch "clear-dpd-tasks-job" deleted
   

5. Verify that the job has been removed. You should no longer see clear-dpd-tasks-job listed.

   kubectl get jobs -n kotsadm
   NAME                                      STATUS     COMPLETIONS   DURATION   AGE
   embedded-cluster-upgrade-<id>   Complete   1/1           30s        8m11s
   

6. Go to the KOTS Admin Console and click the Redeploy button.

7. Wait for the deployment to complete.

8. Confirm that the deployment was successful and the system is operating as expected.