Upgrading Kafka Controller Replica for HA installations

Warning

Before upgrading ITRS Analytics to version 2.12.0+1, this prerequisite upgrade must be completed. These instructions specifically apply to high availability (HA) installations that currently have only one Kafka Controller replica running. The process requires manual intervention and write-level access to the Kubernetes cluster. It is relevant for any earlier version where Kafka Controller replicas are present.

The upgrade may fail if the Kafka controller replica count was modified in the ITRS Analytics configuration file without first completing the upgrade steps. In such cases, do not attempt to uninstall or manually re-run the ITRS Analytics upgrade.

Please contact ITRS Support for assistance.

Manual upgrade instructions Copied

Note

The recommended replica count is three, as this is the default setting specified in the HA configuration files.

1. Modify the NODE_COUNT and CONTROLLER_NODE_COUNT variables to three by running this command.

   kubectl edit cm -n itrs kafka
   

2. Verify the changes by running the command kubectl describe cm -n <namespace> kafka | grep NODE_COUNT=.

   kubectl describe cm -n itrs kafka | grep NODE_COUNT=
   CONTROLLER_NODE_COUNT=3
   NODE_COUNT=3
   

3. Manually scale the Kafka controller StatefulSet to match the updated replica count by running the following command:

   kubectl scale sts kafka-controller --replicas <N>
   

   Example output of this command:

   kubectl scale sts -n itrs kafka-controller --replicas 3                                                                        
   statefulset.apps/kafka-controller scaled
   

4. To check the scaled kafka-controller, run this command kubectl get pods -n <namespace> | grep kafka-controller.

   Example output of this command:

   kubectl get pods -n itrs | grep kafka-controller
   kafka-controller-0                                            2/2     Running     0          102m
   kafka-controller-1                                            2/2     Running     0          5m
   kafka-controller-2                                            2/2     Running     0          5m
   

5. Wait for the new controller replicas to synchronize. To verify synchronization, run the following command on each Kafka controller replica (replace with the appropriate pod index):

   kubectl exec -it kafka-controller-<N> -n <namespace> -- kafka-metadata-quorum.sh --bootstrap-controller localhost:9094 describe --status
   

   Ensure that the CurrentObservers list includes the new controller replicas and that the MaxFollowerLag value is close to 0 by running the command kubectl exec -it kafka-controller-2 -n <namespace> -- kafka-metadata-quorum.sh --bootstrap-controller localhost:9094 describe --status.

   When synchronization is successful, the output should resemble the following:

   kubectl exec -it kafka-controller-2 -n itrs -- kafka-metadata-quorum.sh --bootstrap-controller localhost:9094 describe --status
       ClusterId:              obcerv
       LeaderId:               1000
       LeaderEpoch:            35
   

6. Delete the kafka-controller-0 pod to trigger a restart with the updated ConfigMap settings.

   kubectl delete pod -n <namespace> kafka-controller-0
   

   Once restarted, verify that it is synchronized by rerunning the previous verification commands.

7. Continue with the standard upgrade procedure for ITRS Analytics to version 2.12.0.