SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

• 19.3 Hotfix Package C
• 19.2 Hotfix Package E

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Run the following commands to restart the Kafka broker:

1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka 
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Repeat the above steps on each of the servers that are hosting Kafka, first Replicas and then Controller. You can check which server is the 'Controller' from the output of the status script as mentioned in the Symptoms.

Affected Versions

This article applies to Appian 19.3 and earlier.