KB-1467 Transaction logs FAQ

Jordan Horwat — Wed, 12 Dec 2018 15:06:35 GMT

Current Revision posted to Appian Knowledge Base by Jordan Horwat on 12/12/2018 3:06:35 PM

Table of Contents:

Where is the transaction log located?
How many files are there and do I have to manually clean it up?
Is the cleanup process configurable?
Would checkpointing more frequently influence the maintenance process of the transaction log?

Where is the transaction log located?

Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). These files store data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

How many files are there and do I have to manually clean it up?

The number of files will vary. Kafka and Zookeeper will automatically clean up (delete) older files which they no longer need. Appian automatically removes transactions from Kafka once they are older than the third most recent checkpoint for the relevant engine.

Is the cleanup process configurable?

How frequently Kafka and Zookeeper data is cleaned up is configurable based on how frequently the engines checkpoint. The more frequently the engines checkpoint, the less data will be kept in the transaction logs. Refer to Configuring Application Checkpointing for more information.

Would checkpointing more frequently influence the maintenance process of the transaction log?

By default, old checkpoints do not get deleted. They are moved to an archive location (<APPIAN_HOME>/services/data/archived). Checkpointing more frequently will use up more disk space for the checkpoint images. The CleanupArchives Script should be used to clean up these old checkpoint images.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, engines, zookeeper, infrastructure, service manager, FAQ

KB-1467 Transaction logs FAQ

Jordan Horwat — Fri, 07 Dec 2018 15:13:58 GMT

Revision 18 posted to Appian Knowledge Base by Jordan Horwat on 12/7/2018 3:13:58 PM

Where is the transaction log located?

How many files are there and do I have to manually clean it up?

Is the cleanup process configurable?

Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, engines, zookeeper, infrastructure, service manager, FAQ

KB-1467 Transaction logs FAQ

Jordan Horwat — Mon, 23 Jul 2018 12:54:50 GMT

Revision 17 posted to Appian Knowledge Base by Jordan Horwat on 7/23/2018 12:54:50 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). These files store data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually clean it up?

A: The number of files will vary. Kafka and Zookeeper will automatically clean up (delete) older files which they no longer need. Appian automatically removes transactions from Kafka once they are older than the third most recent checkpoint for the relevant engine.

Q: Is the cleanup process configurable?

A: How frequently Kafka and Zookeeper data is cleaned up is configurable based on how frequently the engines checkpoint. The more frequently the engines checkpoint, the less data will be kept in the transaction logs. Refer to Configuring Application Checkpointing for more information.

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted. They are moved to an archive location (<APPIAN_HOME>/services/data/archived). Checkpointing more frequently will use up more disk space for the checkpoint images. The CleanupArchives Script should be used to clean up these old checkpoint images.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, engines, zookeeper, infrastructure, service manager, FAQ

KB-1467 Transaction logs FAQ

Parmida Borhani — Mon, 29 Jan 2018 01:07:07 GMT

Revision 16 posted to Appian Knowledge Base by Parmida Borhani on 1/29/2018 1:07:07 AM

Q: Where is the transaction log located?

Q: How many files are there and do I have to manually clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Tom Ryan — Sun, 28 Jan 2018 17:49:46 GMT

Revision 15 posted to Appian Knowledge Base by Tom Ryan on 1/28/2018 5:49:46 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted, they are moved to an archive location (<APPIAN_HOME>/services/data/archived). Checkpointing more frequently will use up more disk space for the checkpoint images. The CleanupArchives Script should be used to clean up these old checkpoint images.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Tom Ryan — Sun, 28 Jan 2018 17:49:10 GMT

Revision 14 posted to Appian Knowledge Base by Tom Ryan on 1/28/2018 5:49:10 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Tom Ryan — Sun, 28 Jan 2018 17:48:28 GMT

Revision 13 posted to Appian Knowledge Base by Tom Ryan on 1/28/2018 5:48:28 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store the transaction data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Tom Ryan — Sun, 28 Jan 2018 17:46:48 GMT

Revision 12 posted to Appian Knowledge Base by Tom Ryan on 1/28/2018 5:46:48 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store the transaction data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually go in and clean it up?

A: The number of files will vary. Kafka and Zookeeper will automatically clean up (delete) older files which they no longer need. Appian automatically removes transactions out of Kafka once they are older than the third most recent checkpoint for the relevant engine.

Q: Is the cleanup process configurable?

A: How Kafka and Zookeeper clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently the engines checkpoint, the less data will be kept in the transaction logs. Refer to Configuring Application Checkpointing for more information

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted. They are just moved to an archive location (<APPIAN_HOME>/services/data/archived) so checkpointing more frequently will use up more disk space for the checkpoint images. The CleanupArchives Script should be used to clean up these old checkpoint images.

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Tom Ryan — Sun, 28 Jan 2018 17:45:40 GMT

Revision 11 posted to Appian Knowledge Base by Tom Ryan on 1/28/2018 5:45:40 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and Zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store the transaction data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually go in and clean it up?

A: The number of files will vary. Kafka and Zookeeper will automatically clean up (delete) older files which they no longer need. Appian automatically deletes transactions out of Kafka once they are older than the third most recent checkpoint for the relevant engine.

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Khanh Tran — Thu, 25 Jan 2018 08:50:19 GMT

Revision 10 posted to Appian Knowledge Base by Khanh Tran on 1/25/2018 8:50:19 AM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files store the transaction data for the Appian environment. Touching these files beyond documented procedures may result in data loss or the environment being in an unsupported state.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Parmida Borhani — Wed, 24 Jan 2018 17:47:43 GMT

Revision 9 posted to Appian Knowledge Base by Parmida Borhani on 1/24/2018 5:47:43 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Khanh Tran — Wed, 24 Jan 2018 16:23:48 GMT

Revision 8 posted to Appian Knowledge Base by Khanh Tran on 1/24/2018 4:23:48 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

A: The number of files will vary and is not really important to worry about because Kafka and Zookeeper will automatically clean up (delete) older files which they no longer need. Appian automatically delete transactions out of kafka once they are older than the 3rd most recent checkpoint for the relevant engine.

Q: Is the cleanup process configurable?

A: How they clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently they checkpoint, the less data will be kept in the transaction logs. Refer to Configuring Application Checkpointing for more information

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted. They are just moved to an archive location (<APPIAN_HOME>/services/data/archived by default) so checkpointing more frequently will use up more disk space for the checkpoint images. The cleanupArchives script should be used to clean up these old checkpoint images (which is mentioned in the checkpoint documentation).

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2018

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Khanh Tran — Wed, 24 Jan 2018 16:23:19 GMT

Revision 7 posted to Appian Knowledge Base by Khanh Tran on 1/24/2018 4:23:19 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2017

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Khanh Tran — Wed, 24 Jan 2018 16:22:34 GMT

Revision 6 posted to Appian Knowledge Base by Khanh Tran on 1/24/2018 4:22:34 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted. They are just moved to an archive location (<Appian_Home>/services/data/archived by default) so checkpointing more frequently will use up more disk space for the checkpoint images. The cleanupArchives script should be used to clean up these old checkpoint images (which is mentioned in the checkpoint documentation).

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2017

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX Transaction logs FAQ

Khanh Tran — Tue, 23 Jan 2018 13:46:06 GMT

Revision 5 posted to Appian Knowledge Base by Khanh Tran on 1/23/2018 1:46:06 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <APPIAN_HOME>/services/data/kafka-logs and zookeeper's data is in <APPIAN_HOME>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

A: By default, old checkpoints do not get deleted. They are just moved to an archive location (<Appian Home>/services/data/archived by default) so checkpointing more frequently will use up more disk space for the checkpoint images. The cleanupArchives script should be used to clean up these old checkpoint images (which is mentioned in the checkpoint documentation).

Affected Versions

This article applies to Appian 17.3 and later.

Last Reviewed: January 2017

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX How the Transaction Log is Maintained in New Appian Architecture FAQ

Parmida Borhani — Mon, 22 Jan 2018 18:28:36 GMT

Revision 4 posted to Appian Knowledge Base by Parmida Borhani on 1/22/2018 6:28:36 PM

Q: Where is the transaction log located?

A: Kafka and Zookeeper each have a directory where they write their data files (by default Kafka's data is in <appian home>/services/data/kafka-logs and zookeeper's data is in <appian home>/services/data/zookeeper). Those files make up the actual transaction logs for all K engines so they should not be touched.

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

A: How they clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently they checkpoint, the less data will be kept in the transaction logs. Checkpoint frequency configuration docs are here: Configuring Application Checkpointing

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Tags: kafka, logging, zookeeper, service manager, FAQ

DRAFT KB-XXXX How the Transaction Log is Maintained in New Appian Architecture FAQ

Khanh Tran — Mon, 22 Jan 2018 16:31:17 GMT

Revision 1 posted to Appian Knowledge Base by Khanh Tran on 1/22/2018 4:31:17 PM

Q: Where is the transaction log located?

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

A: How they clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently they checkpoint, the less data will be kept in the transaction logs. Checkpoint frequency configuration docs are here: Configuring Application Checkpointing

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

DRAFT KB-XXXX How the Transaction Log is Maintained in New Appian Architecture FAQ

Khanh Tran — Mon, 22 Jan 2018 11:35:17 GMT

Revision 3 posted to Appian Knowledge Base by Khanh Tran on 1/22/2018 11:35:17 AM

Q: Where is the transaction log located?

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

A: How they clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently they checkpoint, the less data will be kept in the transaction logs. Checkpoint frequency configuration docs are here: Configuring Application Checkpointing

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Tags: logging, service manager

DRAFT KB-XXXX How the Transaction Log is Maintained in New Appian Architecture FAQ

Khanh Tran — Mon, 22 Jan 2018 11:34:38 GMT

Revision 2 posted to Appian Knowledge Base by Khanh Tran on 1/22/2018 11:34:38 AM

Q: Where is the transaction log located?

Q: How many files are there and do I have to manually go in and clean it up?

Q: Is the cleanup process configurable?

A: How they clean up is somewhat configurable based on how frequently the engines checkpoint. The more frequently they checkpoint, the less data will be kept in the transaction logs. Checkpoint frequency configuration docs are here: Configuring Application Checkpointing

Q: Would checkpointing more frequently influence the maintenance process of the transaction log?

Affected Versions

This article applies to Appian 17.3 and later.

Tags: logging