Kafka fails to start after an unexpected shutdown in a High Availability environment

Symptoms

After an unexpected shutdown of Appian high availabilty servers, Kafka fails to start up on a particular node with the following type of error seen in the <APPIAN_HOME>\logs\service-manager\kafka\server.log :

Exiting because log truncation is not allowed for partition serviceManager.transaction.process-design-0, 
Current leader 1's latest offset 6656337 is less than replica 2's latest offset 6656340 (kafka.server.ReplicaFetcherThread) 

Cause

This can happen due to the mismatch in the transaction ids logged across the Kafka nodes.

Action

1. Make sure all Appian components (Service manager, Enginer, Kakfa, Zookeeper, Jboss, Search-server) are shut down.
2. Delete the <APPIAN_HOME>/services/data/zookeeper folder on the server where Kafka is unable to start.
3. Start Appian

Affected Versions

This article applies to Appian 17.3 and later.