KB-1824 Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Jordan Horwat — Fri, 28 Dec 2018 20:24:42 GMT

Current Revision posted to Appian Knowledge Base by Jordan Horwat on 12/28/2018 8:24:42 PM

Symptoms

When starting Appian services after a Kafka failure, Service Manager starts successfully, but the following message is is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Some transactions had not been written to disk at the point of the Kafka failure. Kafka must re-analyze the transaction log and rebuild any missing or corrupt index files. The amount of time this takes is based on the size of the unwritten transactions.

Action

Allow Kafka to finish rebuilding the index files, after which Service Manager should be able to successfully connect to Kafka and complete the startup of Appian services.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: December 2018

Tags: kafka, engines, infrastructure, service manager

DRAFT-XXXX Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Daniel DeVeau — Fri, 28 Dec 2018 14:34:22 GMT

Revision 4 posted to Appian Knowledge Base by Daniel DeVeau on 12/28/2018 2:34:22 PM

Symptoms

When starting Appian services after a Kafka failure, Service Manager starts successfully, but the following message is is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Action

Allow Kafka to finish rebuilding the index files, after which Service Manager should be able to successfully connect to Kafka and complete the startup of Appian services.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: December 2018

Tags: kafka, engines, infrastructure, service manager

DRAFT-XXXX Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Jordan Horwat — Mon, 17 Dec 2018 16:57:05 GMT

Revision 3 posted to Appian Knowledge Base by Jordan Horwat on 12/17/2018 4:57:05 PM

Symptoms

When starting Appian services after a Kafka failure, Service Manager starts successfully, but the following message is is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Action

Allow Kafka to finish rebuilding the index files, after which Service Manager should be able to successfully connect to Kafka and complete the startup of Appian services.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.4 and later.

Last Reviewed: December 2018

Tags: kafka, engines, infrastructure, service manager

DRAFT-XXXX Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Daniel DeVeau — Tue, 04 Dec 2018 17:07:32 GMT

Revision 1 posted to Appian Knowledge Base by Daniel DeVeau on 12/4/2018 5:07:32 PM

Symptoms

When starting Appian services after a Kafka failure, service manager starts successfully. but the following is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Action

Allow Kafka to finish rebuilding the index files, after which point service manager should be able to successfully connect to Kafka.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.4 and later.

Last Reviewed: December 2018

Tags: kafka, engines, infrastructure, service manager

DRAFT-XXXX Service manager logs "Cannot connect to Kafka broker" when Kafka is running

Daniel DeVeau — Tue, 04 Dec 2018 12:08:37 GMT

Revision 2 posted to Appian Knowledge Base by Daniel DeVeau on 12/4/2018 12:08:37 PM

Symptoms

When starting Appian services after a Kafka failure, service manager starts successfully. but the following is logged in the console and in <APPIAN_HOME>/logs/service_manger.log:

com.appian.komodo.log.KafkaUtils - Cannot connect to Kafka broker <SERVER_NAME>:9092

However, Kafka is running, and the following messages are logged in <APPIAN_HOME>/logs/service-manager/kafka/server.log:

WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>/services/data/kafka-logs/<filename>.index) has non-zero size but the last offset is x which is no larger than the base offset y.}. deleting <APPIAN_HOME>/services/data/kafka-logs/<filename>.timeindex, <APPIAN_HOME>/services/data/kafka-logs/<filename>.index and rebuilding index... (kafka.log.Log)

Cause

Action

Allow Kafka to finish rebuilding the index files, after which point service manager should be able to successfully connect to Kafka and complete the startup of Appian services.

Continue starting Appian normally according to the startup guide.

Affected Versions

This article applies to Appian 17.4 and later.

Last Reviewed: December 2018

Tags: kafka, engines, infrastructure, service manager