KB-2068 How to understand checkpointing in service_manager.log

Parmida Borhani — Thu, 30 Jan 2020 00:21:09 GMT

Current Revision posted to Appian Knowledge Base by Parmida Borhani on 1/30/2020 12:21:09 AM

Purpose

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically. This article illustrates and explains the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk. These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. These files are stored in the <APPIAN_HOME>/services/data/temporary folder.
<path_to_archived>: This is the archived folder where historical KDB files are stored. These files are stored in the <APPIAN_HOME>/services/data/archived folder. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

When a checkpoint is first requested or triggered, it is placed into a queue within the Service Manager component. Checkpoints occur serially, in the order that they are placed into the queue. In the logs, this looks like:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing KDB is created in the temporary folder associated with the engine undergoing a checkpoint. This is usually located at <APPIAN_HOME>/services/data/temporary/<engine_id>.

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

Once the checkpoint is complete Service Manager will perform cleanup of the prior "live" KDB, and will also instruct the Kafka component to mark any transactions that were persisted to disk in the checkpoint as ready for cleanup. In the logs, this is associated with:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, infrastructure, service manager, checkpointing

KB-2068 How to understand checkpointing in service_manager.log

Parmida Borhani — Thu, 30 Jan 2020 00:20:39 GMT

Revision 18 posted to Appian Knowledge Base by Parmida Borhani on 1/30/2020 12:20:39 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk. These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. These files are stored in the <APPIAN_HOME>/services/data/temporary folder.
<path_to_archived>: This is the archived folder where historical KDB files are stored. These files are stored in the <APPIAN_HOME>/services/data/archived folder. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Parmida Borhani — Wed, 29 Jan 2020 02:23:09 GMT

Revision 17 posted to Appian Knowledge Base by Parmida Borhani on 1/29/2020 2:23:09 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk. These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. These files are stored in the <APPIAN_HOME>/services/data/temporary folder.
<path_to_archived>: This is the archived folder where historical KDB files are stored. These files are stored in the <APPIAN_HOME>/services/data/archived folder. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Parmida Borhani — Wed, 29 Jan 2020 02:22:32 GMT

Revision 16 posted to Appian Knowledge Base by Parmida Borhani on 1/29/2020 2:22:32 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk. These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. These files are stored in the <APPIAN_HOME>/services/data/temporary folder.
<path_to_archived>: This is the archived folder where historical KDB files are stored. These files are stored in the <APPIAN_HOME>/services/data/archived folder. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified unless advised by Appian Support. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Parmida Borhani — Wed, 29 Jan 2020 02:22:05 GMT

Revision 15 posted to Appian Knowledge Base by Parmida Borhani on 1/29/2020 2:22:05 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk. These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. These files are stored in the <APPIAN_HOME>/services/data/temporary folder.
<path_to_archived>: This is the archived folder where historical KDB files are stored. These files are stored in the <APPIAN_HOME>/services/data/archived folder. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified unless advised by Appian Support. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Tue, 28 Jan 2020 23:31:01 GMT

Revision 14 posted to Appian Knowledge Base by Tristan Newmann on 1/28/2020 11:31:01 PM

Purpose

Instructions

Note: The remainder of the document uses the following notations and concepts:

Appian Engines: Appian engines are real-time in-memory (RAM) databases that also persist all data in a file on disk.These database files have the extension .kdb.
<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support.
<path_to_archived>: This is the archived folder where historical KDB files are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified unless advised by Appian Support. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Parmida Borhani — Tue, 28 Jan 2020 06:59:13 GMT

Revision 13 posted to Appian Knowledge Base by Parmida Borhani on 1/28/2020 6:59:13 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations:

<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support.
<path_to_archived>: This is the archived folder where historical KDB files are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified unless advised by Appian Support. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint_id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB_name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.kdb to <path_to_archived>/archived/<engine_id>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Parmida Borhani — Tue, 28 Jan 2020 06:52:04 GMT

Revision 12 posted to Appian Knowledge Base by Parmida Borhani on 1/28/2020 6:52:04 AM

Purpose

Instructions

Note: The remainder of the document uses the following notations:

<engine_id>: This is the name of the engine associated with the checkpoint. e.g. groups, execution00.
<path_to_temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support.
<path_to_archived>: This is the archived folder where historical KDB files are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised by Appian Support. This directory can be cleaned and maintained using the cleanupArchives script.
<path_to_engine_gateway>: This is the directory that stores the live KDB that an engine will boot from if it is shutdown. This file should never be modified unless advised by Appian Support. The exact path varies from engine to engine. e.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase 1: Queuing

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine_id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine_id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine_id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, KDBFileIndex=X,retentionMetadata=serviceManager.transaction.<engine_id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2: De-queuing

Service Manager will de-queue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine_id>

Phase 3: Checkpoint

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing KDB from Phase 2 will be renamed to *.written.
The *.written KDB will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live KDB.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path_to_temporary>/temporary/<engine_id>/<engine_id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/<KDB name>.KDB
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine_id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine_id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine_id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4: Cleanup

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving KDB image from <APPIAN_HOME>/server/<path_to_engine_gateway>/gw1/*.KDB to <path_to_archived>/archived/<engine_id>/*.KDB
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine_id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine_id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on different versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 04:15:45 GMT

Revision 11 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 4:15:45 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine undergoing a checkpoint. This is usually located at <APPIAN_HOME>/services/data/temporary/<engine id>.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to *.written.
The *.written kdb will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

Once the checkpoint is complete Service Manager will perform cleanup of the prior "live" kdb, and will also instruct the Kafka component to mark any transactions that were persisted to disk in the checkpoint as ready for cleanup. In the logs, this is associated with:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/*.kdb to <path to archived>/archived/<engineid>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 04:14:24 GMT

Revision 10 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 4:14:24 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine undergoing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to *.written.
The *.written kdb will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/*.kdb to <path to archived>/archived/<engineid>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 04:13:22 GMT

Revision 9 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 4:13:22 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g. <APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine undergoing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to *.written.
The *.written kdb will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/*.kdb to <path to archived>/archived/<engineid>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 04:02:47 GMT

Revision 8 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 4:02:47 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine undergoing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to *.written.
The *.written kdb will be moved into the <APPIAN_HOME>/server/**/gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/*.kdb to <path to archived>/archived/<engineid>/*.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offset for serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 03:59:55 GMT

Revision 7 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 3:59:55 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00/gw1 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine experiencing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to <kdb name>.written
The *.written kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/peX.kdb to <path to archived>/archived/<engineid>/peX.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 03:58:45 GMT

Revision 6 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 3:58:45 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=X,retentionMetadata=serviceManager.transaction.<engine id>-X@X,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine experiencing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to <kdb name>.written
The *.written kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/peX.kdb to <path to archived>/archived/<engineid>/peX.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 03:56:36 GMT

Revision 5 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 3:56:36 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=316957, kdbFileIndex=437,retentionMetadata=serviceManager.transaction.<engine id>-0@656290,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine experiencing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to <kdb name>.written
The *.written kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/peX.kdb to <path to archived>/archived/<engineid>/peX.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Fri, 24 Jan 2020 03:55:37 GMT

Revision 4 posted to Appian Knowledge Base by Tristan Newmann on 1/24/2020 3:55:37 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf.Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with the execution00 engine gateway.

Phase1:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>>,
 requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId
=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=XXXX, kdbFileIndex=XXX,retentionMetadata=serviceManager.transaction.<engine id>-X@XXXX,initiatingHost=<host>, executionHost=<host>}}

User Requested (using the checkpoint script):

YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false, checkpointContext=null}
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=YYYY-MM-DDTHH:MM:SS.mmmZ,engineId=<engine id>, requestId=<checkpoint id>,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=<checkpoint id>,minTxId=316957, kdbFileIndex=437,retentionMetadata=serviceManager.transaction.<engine id>-0@656290,initiatingHost=<host>, executionHost=<host>}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint <checkpoint id> dequeued for: <engineid>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary *.writing kdb is created in the temporary folder associated with the engine experiencing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The *.writing kdb from Phase 2 will be renamed to <kdb name>.written
The *.written kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint <checkpoint id> file<path to temporary>/temporary/<engine id>/<engineid>-<checkpoint id>.written at YYYY-MM-DD HH:MM:SS.mmmZ
YYYY-MM-DD HH:MM:SS.mmm[checkpoint-watcher-<engine id>-XXX-X] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engine id>/<engine id>-<checkpoint id>.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdb name>.kdb
YYYY-MM-DD HH:MM:SS.mmm [checkpoint-watcher-<engineid>-XXX-X] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint <checkpoint id> for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint<checkpoint id> completed for <engine id>
YYYY-MM-DD HH:MM:SS.mmm [CheckpointManager-<engine id>-X] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpoint <checkpoint id> execution complete

Phase 4:

YYYY-MM-DD HH:MM:SS.mmm[checkpointArchiver-XXX-X]INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to engine gateway>/gw1/peX.kdb to <path to archived>/archived/<engineid>/peX.kdb
...
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-X from XXXX to XXXX
YYYY-MM-DD HH:MM:SS.mmm [KomodoEventBus-XXXX] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entries for <engine id> that are an earlier offset than XXXX for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Thu, 23 Jan 2020 07:21:10 GMT

Revision 3 posted to Appian Knowledge Base by Tristan Newmann on 1/23/2020 7:21:10 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary.However, this can be changed in<APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf.Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with execution00.

Phase1:

When a checkpoint is first requested or triggered, it is placed into a queue within the Service Manager component. Checkpoints occur serially, in the orderthat they are placed into the queue. In the logs, this looks like:

2020-01-23 06:07:06,297 [KomodoEventBus-3139] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

2020-01-2211:47:53,682 [CheckpointManager-<engine id>-0] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>>,
 requestId=b6049c4c-af87-4594-88c2-354c1c6c632b,engineShuttingDown=false, checkpointContext=null}
 2020-01-2211:47:53,689 [CheckpointManager-groups-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>, requestId
=6c2e9683-7149-4003-b810-a4e9f075d792,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=6c2e9683-7149-4003-b810-a4e9f075d792,minTxId=259326, kdbFileIndex=436,retentionMetadata=serviceManager.transaction.<engine id>-0@51882
2,initiatingHost=dev194, executionHost=dev194}}

User Requested (using the checkpoint script):

2020-01-23 06:07:06,298[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false, checkpointContext=null}
2020-01-23 06:07:06,307[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,minTxId=316957, kdbFileIndex=437,retentionMetadata=serviceManager.transaction.<engine id>-0@656290,initiatingHost=dev194, executionHost=dev194}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

2020-01-23 06:07:06,307 [KomodoEventBus-3115]INFO  com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc dequeued for: <engineid>
2020-01-23 06:07:06,307 [KomodoEventBus-3142] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:06,322 [KomodoEventBus-3115] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc started for <engine id>

After this output, Service Manager will be actively checkpointing the specific engine that the request was associated with. While an engine is being checkpointed, a temporary "writing" kdb is created in the temporary folder associated with the engine experiencing a checkpoint.

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The "writing" kdb from Phase 2 will be renamed to <kdb name>.written
The ".written" kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

2020-01-2306:07:16,828 [checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc file<path to temporary>/temporary/<engine id>/<engineid>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written at2020-01-23T06:07:16.828Z
2020-01-23 06:07:16,828[checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engineid>/<engine id>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdbname>.kdb
2020-01-23 06:07:19,065 [checkpoint-watcher-<engineid>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:19,066 [KomodoEventBus-3136] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc completed for <engine id>
2020-01-2306:07:19,066 [CheckpointManager-<engine id>-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc execution complete

Phase 4:

Once the checkpoint is complete Service Manager will perform cleanup of the prior "live" kdb, and will also instruct the Kafka component to mark anytransactions that were persisted to disk in the checkpoint as ready for cleanup. In the logs, this is associated with:

2020-01-23 06:07:19,067[checkpointArchiver-<path to archived>/archived/<engine id>-171-1]INFO  com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to enginegateway>/gw1/pe436.kdb to <path to archived>/archived/<engineid>/pe436.kdb
...
2020-01-23 06:07:19,089 [KomodoEventBus-3128]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-0 from 630299 to 638385
2020-01-2306:07:19,091 [KomodoEventBus-3128] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entriesfor <engine id> that are an earlier offset than 638385 for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Thu, 23 Jan 2020 07:20:21 GMT

Revision 2 posted to Appian Knowledge Base by Tristan Newmann on 1/23/2020 7:20:21 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the <APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary.However, this can be changed in<APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf.Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with execution00.

Phase1:

When a checkpoint is first requested or triggered, it is placed into a queue within the Service Manager component. Checkpoints occur serially, in the orderthat they are placed into the queue. In the logs, this looks like:

2020-01-23 06:07:06,297 [KomodoEventBus-3139] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

2020-01-2211:47:53,682 [CheckpointManager-<engine id>-0] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>>,
 requestId=b6049c4c-af87-4594-88c2-354c1c6c632b,engineShuttingDown=false, checkpointContext=null}
 2020-01-2211:47:53,689 [CheckpointManager-groups-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>, requestId
=6c2e9683-7149-4003-b810-a4e9f075d792,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=6c2e9683-7149-4003-b810-a4e9f075d792,minTxId=259326, kdbFileIndex=436,retentionMetadata=serviceManager.transaction.<engine id>-0@51882
2,initiatingHost=dev194, executionHost=dev194}}

User Requested (using the checkpoint script):

2020-01-23 06:07:06,298[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false, checkpointContext=null}
2020-01-23 06:07:06,307[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,minTxId=316957, kdbFileIndex=437,retentionMetadata=serviceManager.transaction.<engine id>-0@656290,initiatingHost=dev194, executionHost=dev194}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

2020-01-23 06:07:06,307 [KomodoEventBus-3115]INFO  com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc dequeued for: <engineid>
2020-01-23 06:07:06,307 [KomodoEventBus-3142] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:06,322 [KomodoEventBus-3115] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc started for <engine id>

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The "writing" kdb from Phase 2 will be renamed to <kdb name>.written
The ".written" kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

2020-01-2306:07:16,828 [checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc file<path to temporary>/temporary/<engine id>/<engineid>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written at2020-01-23T06:07:16.828Z
2020-01-23 06:07:16,828[checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engineid>/<engine id>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdbname>.kdb
2020-01-23 06:07:19,065 [checkpoint-watcher-<engineid>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:19,066 [KomodoEventBus-3136] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc completed for <engine id>
2020-01-2306:07:19,066 [CheckpointManager-<engine id>-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc execution complete

Phase 4:

Once the checkpoint is complete Service Manager will perform cleanup of theprior "live" kdb, and will also instruct the Kafka component to mark anytransactions that were persisted to disk in the checkpoint as ready forcleanup. In the logs, this is associated with:

2020-01-23 06:07:19,067[checkpointArchiver-<path to archived>/archived/<engine id>-171-1]INFO  com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to enginegateway>/gw1/pe436.kdb to <path to archived>/archived/<engineid>/pe436.kdb
...
2020-01-23 06:07:19,089 [KomodoEventBus-3128]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-0 from 630299 to 638385
2020-01-2306:07:19,091 [KomodoEventBus-3128] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entriesfor <engine id> that are an earlier offset than 638385 for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing

KB-XXXX How to understand checkpointing in service_manager.log

Tristan Newmann — Thu, 23 Jan 2020 07:19:02 GMT

Revision 1 posted to Appian Knowledge Base by Tristan Newmann on 1/23/2020 7:19:02 AM

Purpose

Illustrate and explain the process of a single engine checkpointing, and how the associated output appears in the<APPIAN_HOME>/logs/service_manager.log.

Context

The Appian Engines use a process called "checkpointing" to persist data stored in the in-memory engines to disk. This is triggered either manually or automatically.

Instructions

Note: The remainder of the document will use the following notation:

<engine id>: This is the name of the engine associated with the checkpoint. E.g. groups, execution00.
<path to temporary>: This is the temporary folder where files related to an in-progress checkpoint are stored. This is typically <APPIAN_HOME>/services/data/temporary.However, this can be changed in<APPIAN_HOME>/services/conf/service_manager.conf. Changing this file is not supported unless advised to by Appian Support.
<path to archived>: This is the archived folder where historical kdbs are stored.This is typically <APPIAN_HOME>/services/data/archived. However, this can be changed in <APPIAN_HOME>/services/conf/service_manager.conf.Changing this file is not supported unless advised to by Appian Support. This directly can be cleaned and maintained using the cleanupArchives script.
<path to engine gateway>: This is the directory that stores the live kdb that an engine will boot from if it is shutdown. This file should never be modified unless advised to by Appian Support. The exact path varies from engine to engine. E.g.<APPIAN_HOME>/server/process/exec/00 is associated with execution00.

Phase1:

When a checkpoint is first requested or triggered, it is placed into a queue within the Service Manager component. Checkpoints occur serially, in the orderthat they are placed into the queue. In the logs, this looks like:

2020-01-23 06:07:06,297 [KomodoEventBus-3139] INFO  com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpoint <checkpoint id> requested for: <engine id>

Service Manager will also log output indicating the type of checkpoint that was requested, e.g. automatic or user triggered.

Automatic:

2020-01-2211:47:53,682 [CheckpointManager-<engine id>-0] INFO  com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>>,
 requestId=b6049c4c-af87-4594-88c2-354c1c6c632b,engineShuttingDown=false, checkpointContext=null}
 2020-01-2211:47:53,689 [CheckpointManager-groups-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventAutoCheckpointRequestedEvent{createdInstant=2020-01-22T11:47:53.681Z,engineId=<engine id>, requestId
=6c2e9683-7149-4003-b810-a4e9f075d792,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=6c2e9683-7149-4003-b810-a4e9f075d792,minTxId=259326, kdbFileIndex=436,retentionMetadata=serviceManager.transaction.<engine id>-0@51882
2,initiatingHost=dev194, executionHost=dev194}}

User Requested (using the checkpoint script):

2020-01-23 06:07:06,298[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - HandlingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false, checkpointContext=null}
2020-01-23 06:07:06,307[CheckpointManager-<engine id-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - QueueingCheckpointRequestedEventUserCheckpointRequestedEvent{createdInstant=2020-01-23T06:07:06.296Z,engineId=<engine id>, requestId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,engineShuttingDown=false,checkpointContext=CheckpointContext{checkpointId=b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc,minTxId=316957, kdbFileIndex=437,retentionMetadata=serviceManager.transaction.<engine id>-0@656290,initiatingHost=dev194, executionHost=dev194}}

As can be seen, in both of the above outputs, the CheckpointRequestedEvent is handled by putting it on the checkpointing queue, to be processed.

Phase 2:

Service Manager will dequeue a checkpoint request in order to process it. This is associated with:

2020-01-23 06:07:06,307 [KomodoEventBus-3115]INFO  com.appian.komodo.engine.status.CheckpointingStateProvider -Checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc dequeued for: <engineid>
2020-01-23 06:07:06,307 [KomodoEventBus-3142] INFO com.appian.komodo.engine.checkpoint.CheckpointWatcher - Waiting for checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:06,322 [KomodoEventBus-3115] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc started for <engine id>

Phase 3:

Service Manager will complete the checkpoint operation and perform some operations on the file system:

The "writing" kdb from Phase 2 will be renamed to <kdb name>.written
The ".written" kdb will be moved into the gw1 folder associated with the engine that underwent a checkpoint. This is the new live kdb.

These appear in the logs as:

2020-01-2306:07:16,828 [checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Notified of Kimage created for checkpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc file<path to temporary>/temporary/<engine id>/<engineid>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written at2020-01-23T06:07:16.828Z
2020-01-23 06:07:16,828[checkpoint-watcher-<engine id>-172-1] INFO com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier - Moving KCheckpoint to next live: <path to temporary>/temporary/<engineid>/<engine id>-b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc.written to/<APPIAN_HOME>/server/<path to engine gateway>/gw1/<kdbname>.kdb
2020-01-23 06:07:19,065 [checkpoint-watcher-<engineid>-172-1] INFO com.appian.komodo.engine.checkpoint.LocalCheckpointWatcherTask - Finishedcheckpoint b2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc for <engine id>
2020-01-2306:07:19,066 [KomodoEventBus-3136] INFO com.appian.komodo.engine.status.CheckpointingStateProvider - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc completed for <engine id>
2020-01-2306:07:19,066 [CheckpointManager-<engine id>-0] INFO com.appian.komodo.engine.checkpoint.CheckpointManager - Checkpointb2d6c4d0-8be9-42fd-aeaa-4e35a63da3dc execution complete

Phase 4:

Once the checkpoint is complete Service Manager will perform cleanup of theprior "live" kdb, and will also instruct the Kafka component to mark anytransactions that were persisted to disk in the checkpoint as ready forcleanup. In the logs, this is associated with:

2020-01-23 06:07:19,067[checkpointArchiver-<path to archived>/archived/<engine id>-171-1]INFO  com.appian.komodo.engine.checkpoint.SingleGatewayCheckpointCopier -Archiving kdb image from <APPIAN_HOME>/server/<path to enginegateway>/gw1/pe436.kdb to <path to archived>/archived/<engineid>/pe436.kdb
...
2020-01-23 06:07:19,089 [KomodoEventBus-3128]INFO  com.appian.komodo.log.KafkaLogCleaner - Updating committed offsetfor serviceManager.transaction.<engine id>-0 from 630299 to 638385
2020-01-2306:07:19,091 [KomodoEventBus-3128] INFO com.appian.komodo.log.KafkaLogCleaner - Marking all transaction log entriesfor <engine id> that are an earlier offset than 638385 for deletion.

Affected Versions

This article applies to Appian 17.3 and later. The output was captured on Appian 19.4 and may be slightly different on older / newer versions of Appian.

Last Reviewed: January 2020

Tags: kafka, engines, how-to, Architecture, service manager, checkpointing