KB-1824 Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Symptoms

When starting Appian services after a Kafka failure, Service Manager starts successfully, but the following message is is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Some transactions had not been written to disk at the point of the Kafka failure. Kafka must re-analyze the transaction log and rebuild any missing or corrupt index files. The amount of time this takes is based on the size of the unwritten transactions. 

Action

Allow Kafka to finish rebuilding the index files, after which Service Manager should be able to successfully connect to Kafka and complete the startup of Appian services.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: December 2018

Related
Recommended