KB-1642 Kafka: WARN Found a corrupted index file due to requirement failed: Corrupt index found

Symptoms

When running Appian 18.1 and later on Windows the following output may occur during the start up of Appian in the Kafka logs located at: <APPIAN_HOME>\logs\service-manager\kafka\server.log

[YYYY-MM-DD HH:MM:SS.mmm] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting <APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.timeindex, <APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.index and rebuilding index... (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Recovering unflushed segment 0 in log ds_txEffects-0. (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Completed load of log ds_txEffects-0 with 1 log segments and log end offset 1 in X ms (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.exec.idblocks-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting <APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.engine.idblocks-0\00000000000000000000.timeindex, <APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.engine.idblocks-0\00000000000000000000.index and rebuilding index... (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Recovering unflushed segment 0 in log serviceManager.engine.idblocks-0. (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Completed load of log serviceManager.engine.idblocks-0 with 1 log segments and log end offset 0 in X ms (kafka.log.Log)

Cause

This default warning output in Kafka is merely highlighting a limitation of Kafka when running on Windows which forces a rebuild of the Kafka transaction index on each start up of Appian. The default warning output details data corruption but Appian is handling this internally on shutdown and start up of Appian, so there is no risk to the data. The Kafka index is rebuilt and start up moves forward as normal each time.

Action

The warning can be safely ignored.

Affected Versions

This article applies to Appian 18.1 and later.

Last Reviewed: August 2018