KB-1589 Kafka fails to start after an unexpected shutdown in a high availability environment

Symptoms

After an unexpected shutdown of the entire cluster which hosts Appian high availabilty servers, Kafka fails to start up on a particular node with the following type of error seen in the <APPIAN_HOME>/logs/service-manager/kafka/server.log :

Exiting because log truncation is not allowed for partition serviceManager.transaction.process-design-0, 
Current leader 1's latest offset 6656337 is less than replica 2's latest offset 6656340 (kafka.server.ReplicaFetcherThread) 

Cause

This can happen due to the mismatch in the transaction ids logged across the Kafka nodes.

Action

1. Make sure all Appian components are shut down.
2. Delete the <APPIAN_HOME>/services/data/zookeeper folder on all servers which host Zookeeper.
3. Start Appian.

Affected Versions

This article applies to Appian 17.3 and later.