SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

• 19.3 Hotfix Package C
• 19.2 Hotfix Package E

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Ensure to Stop and Start Kafka on Controller last 

1. /usr/local/appian/ae/aed stopKafka (Cloud)
2. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka (On-Premise)
3. /usr/local/appian/ae/aed startKafka (Cloud)
4. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka (On-Premise)

Repeat the above step to restart the Kafka broker on each of the other servers that are hosting Kafka, one server at a time.

Affected Versions

This article applies to Appian 19.3 and earlier.