Upgrading Kafka Controller Replica for HA installations

Note

These upgrade instructions apply only to high availability (HA) installations that currently have only one Kafka Controller replica running. Manual intervention is required and will necessitate write-level access to the Kubernetes cluster. This upgrade must be completed before upgrading ITRS Analytics to version 2.12.0. It is applicable to any earlier version where Kafka Controller replicas are present.

Please consult ITRS Support before proceeding with the upgrade.

Manual upgrade instructions Copied

1. Modify the Kafka ConfigMap by updating the NODE_COUNT and CONTROLLER_NODE_COUNT variables to reflect the new, increased number of controller replicas. Once updated, save the ConfigMap.

2. Manually scale the Kafka controller StatefulSet to match the updated replica count by running the following command:

   kubectl scale sts kafka-controller --replicas <N>
   

3. Wait for the new controller replicas to synchronize. To verify synchronization, run the following command on each Kafka controller replica (replace with the appropriate pod index):

   kubectl exec -it kafka-controller-<N> -- kafka-metadata-quorum.sh --bootstrap-controller localhost:9094 describe --status
   

   Ensure that the CurrentObservers list includes the new controller replicas and that the MaxFollowerLag value is close to 0. When synchronization is successful, the output should resemble the following:

   % kc exec -it kafka-controller-2 -- kafka-metadata-quorum.sh --bootstrap-controller localhost:9094 describe --status
   ClusterId:              obcerv
   LeaderId:               1000
   LeaderEpoch:            35
   HighWatermark:          35353
   MaxFollowerLag:         0
   MaxFollowerLagTimeMs:   0
   CurrentVoters:          [{"id": 1000, "directoryId": null, "endpoints": ["CONTROLLER://kafka-controller-0.kafka-controller.itrs.svc.cluster.local:9094"]}]
   CurrentObservers:       [{"id": 2, "directoryId": "Cg6j4VsI_lZqGRaGD2GxYQ"}, {"id": 1002, "directoryId": "KKq2KljOYYRYnN7JXfuu1w"}, {"id": 1, "directoryId": "lEdp0iqMgPy8vanF2qYOHg"}, {"id": 0, "directoryId": "_LBgdIa06mIFYPNxCMAPMA"}, {"id": 1001, "directoryId": "pfbcNY1K9Z0TAsM8gUiiFg"}]
   

4. Delete the kafka-controller-0 pod to trigger a restart with the updated ConfigMap settings. Once restarted, verify that it is synchronized by rerunning the previous verification commands.

5. Update and save the ITRS Analytics configuration file to reflect the new Kafka controller replica count by setting the kafka.controller.replicas parameter accordingly.

6. Continue with the standard upgrade procedure for ITRS Analytics to version 2.12.0.

Warning

If the Kafka controller replica count was updated in the ITRS Analytics configuration file without completing the prerequisite steps outlined above, and the upgrade to version 2.12.0 was initiated, the upgrade may fail. In this scenario, please contact ITRS Support for guidance.

Do not attempt to uninstall or manually re-run the ITRS Analytics upgrade.