KB-1589 Kafka fails to start after an unexpected shutdown in a high availability environment

Symptoms

After an unexpected shutdown of the entire cluster which hosts Appian high availabilty servers, Kafka fails to start up on a particular node with the following type of error seen in the <APPIAN_HOME>/logs/service-manager/kafka/server.log :

Exiting because log truncation is not allowed for partition serviceManager.transaction.process-design-0, 
Current leader 1's latest offset 6656337 is less than replica 2's latest offset 6656340 (kafka.server.ReplicaFetcherThread)

Cause

This can happen due to the mismatch in the transaction ids logged across the Kafka nodes.

Action

DISCLAIMER:

This KB article is currently under review given its age and the relevant updates in the product since its last review. At this time, it is advised to not following the steps for resolution.

If you believe that your environment is currently experiencing the below behavior, please open a case with Appian Support for further review.

Make sure all Appian components are shut down.
Delete the <APPIAN_HOME>/services/data/zookeeper folder on all servers which host Zookeeper.
Start Appian.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: May 2018