You are currently reviewing an older revision of this page.

DRAFT Kafka: WARN Found a corrupted index file due to requirement failed: Corrupt index found

Symptoms

Customers running Appian 18.1+ on Windows may see the following output during the start up of Appian in the Kafka logs located at: <APPIAN_HOME>\logs\service-manager\kafka\server.log

[YYYY-MM-DD HH:MM:SS.mmm] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting <APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.timeindex, <APPIAN_HOME>\services\bin\..\data\kafka-logs\ds_txEffects-0\00000000000000000000.index and rebuilding index... (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Recovering unflushed segment 0 in log ds_txEffects-0. (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Completed load of log ds_txEffects-0 with 1 log segments and log end offset 1 in X ms (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (<APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.exec.idblocks-0\00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting <APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.engine.idblocks-0\00000000000000000000.timeindex, <APPIAN_HOME>\services\bin\..\data\kafka-logs\serviceManager.engine.idblocks-0\00000000000000000000.index and rebuilding index... (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Recovering unflushed segment 0 in log serviceManager.engine.idblocks-0. (kafka.log.Log)
[YYYY-MM-DD HH:MM:SS.mmm] INFO Completed load of log serviceManager.engine.idblocks-0 with 1 log segments and log end offset 0 in X ms (kafka.log.Log)

Cause

This default warning output in Kafka is merely highlighting a limitation of Kafka when running on Windows which forces a rebuild of the Kafka transaction index on each start up of Appian.

Action

The warning can be safely ignored. The default warning output details data corruption but Appian is handling this internally on shutdown and start up of Appian so there is no risk to your data. The Kafka index is rebuilt and start up moves forward as normal each time.

Affected Versions

Appian 18.1 and 18.2

Last Reviewed

08-Aug-2018