KB-2096 Kafka brokers out of sync in a high availability environment

James Lee — Sun, 15 Mar 2020 22:34:21 GMT

Current Revision posted to Appian Knowledge Base by James Lee on 3/15/2020 10:34:21 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

It will be necessary to perform a rolling restart of the Kafka brokers on the site. The Kafka brokers that must be restarted one at a time are listed in <APPIAN_HOME>/conf/appian-topology.xml file.

Make sure all ZooKeeper servers configured in appian-topology.xml are running. To start ZooKeeper run the following on each machine hosting ZooKeeper:
1. <APPIAN_HOME>/services/bin/start.sh -p <password> -s zookeeper
For one machine, restart the Kafka broker using the below commands:
1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Repeat the above steps on each of the servers that are hosting Kafka, starting with the Replicas first, and the Controller last. The Controller can be identified from the output of the status script used in the Symptoms section.

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: March 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

James Lee — Sun, 15 Mar 2020 22:33:46 GMT

Revision 15 posted to Appian Knowledge Base by James Lee on 3/15/2020 10:33:46 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

It will be necessary to perform a rolling restart of the Kafka brokers on the site. The Kafka brokers that must be restarted one at a time are listed in <APPIAN_HOME>/conf/appian-topology.xml file.

Make sure all ZooKeeper servers configured in appian-topology.xml are running. To start ZooKeeper run the following on each machine hosting ZooKeeper:
1. <APPIAN_HOME>/services/bin/start.sh -p <password> -s zookeeper
For one machine, restart the Kafka broker using the below commands:
1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: March 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

James Lee — Tue, 10 Mar 2020 21:28:05 GMT

Revision 14 posted to Appian Knowledge Base by James Lee on 3/10/2020 9:28:05 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

It will be necessary to perform a rolling restart of the Kafka brokers on the site. The Kafka brokers that must be restarted one at a time are listed in <APPIAN_HOME>/conf/appian-topology.xml file.

Make sure all ZooKeeper servers configured in appian-topology.xml are running. To start ZooKeeper run the following on each machine hosting ZooKeeper:
1. <APPIAN_HOME>/services/bin/start.sh -p <password> -s zookeeper
For one machine, restart the Kafka broker using the below commands:
1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: March 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

James Lee — Tue, 10 Mar 2020 21:27:57 GMT

Revision 13 posted to Appian Knowledge Base by James Lee on 3/10/2020 9:27:57 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

It will be necessary to perform a rolling restart of the Kafka brokers on the site. The Kafka brokers that must be restarted one at a time are listed in <APPIAN_HOME>/conf/appian-topology.xml file.

Make sure all ZooKeeper servers configured in appian-topology.xml are running. To start ZooKeeper run the following on each machine hosting ZooKeeper:
1. <APPIAN_HOME>/services/bin/start.sh -p <password> -s zookeeper
For one machine, restart the Kafka broker using the below commands:
1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: March 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

nishant.shah — Mon, 09 Mar 2020 22:47:47 GMT

Revision 12 posted to Appian Knowledge Base by nishant.shah on 3/9/2020 10:47:47 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

It will be necessary to perform a rolling restart of the Kafka brokers on the site. The Kafka brokers that must be restarted one at a time are listed in <APPIAN_HOME>/conf/appian-topology.xml file.

Make sure all ZooKeeper servers configured in appian-topology.xml are running. To start ZooKeeper run the following on each machine hosting ZooKeeper:
1. <APPIAN_HOME>/services/bin/start.sh -p <password> -s zookeeper
For one machine, restart the Kafka broker using the below commands:
1. <APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
2. <APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

nishant.shah — Mon, 09 Mar 2020 22:42:48 GMT

Revision 11 posted to Appian Knowledge Base by nishant.shah on 3/9/2020 10:42:48 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Run the following commands to restart the Kafka broker:

<APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
<APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Repeat the above steps on each of the servers that are hosting Kafka, first Replicas and then Controller. You can check which server is the 'Controller' from the output of the status script as mentioned in the Symptoms.

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

nishant.shah — Thu, 27 Feb 2020 22:48:22 GMT

Revision 10 posted to Appian Knowledge Base by nishant.shah on 2/27/2020 10:48:22 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Run the following commands to restart the Kafka broker:

<APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka
<APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka brokers out of sync in a high availability environment

nishant.shah — Fri, 21 Feb 2020 22:14:50 GMT

Revision 9 posted to Appian Knowledge Base by nishant.shah on 2/21/2020 10:14:50 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Ensure to Stop and Start Kafka on Controller last

/usr/local/appian/ae/aed stopKafka (Cloud)
<APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka (On-Premise)
/usr/local/appian/ae/aed startKafka (Cloud)
<APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka (On-Premise)

Repeat the above step to restart the Kafka broker on each of the other servers that are hosting Kafka, one server at a time.

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka replicas out of sync in a high availability environment

nishant.shah — Fri, 21 Feb 2020 22:13:42 GMT

Revision 8 posted to Appian Knowledge Base by nishant.shah on 2/21/2020 10:13:42 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Running <APPIAN_HOME>/services/bin/status.bat (.sh) -p <password> -c in a high availability environment shows one of the Kafka brokers to be not in sync:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Ensure to Stop and Start Kafka on Controller last

/usr/local/appian/ae/aed stopKafka (Cloud)
<APPIAN_HOME>/services/bin/stop.sh -p <password> -s kafka (On-Premise)
/usr/local/appian/ae/aed startKafka (Cloud)
<APPIAN_HOME>/services/bin/start.sh -p <password> -s kafka (On-Premise)

Repeat the above step to restart the Kafka broker on each of the other servers that are hosting Kafka, one server at a time.

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

James Lee — Tue, 11 Feb 2020 19:08:33 GMT

Revision 7 posted to Appian Knowledge Base by James Lee on 2/11/2020 7:08:33 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on example1.com
Stop and Start Kafka broker on example2.com
Stop and Start Kafka broker on example3.com (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

SP-3079 DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

Parmida Borhani — Tue, 11 Feb 2020 06:16:04 GMT

Revision 6 posted to Appian Knowledge Base by Parmida Borhani on 2/11/2020 6:16:04 AM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on example1.com
Stop and Start Kafka broker on example2.com
Stop and Start Kafka broker on example3.com (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

nishant.shah — Fri, 07 Feb 2020 21:53:02 GMT

Revision 5 posted to Appian Knowledge Base by nishant.shah on 2/7/2020 9:53:02 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on example1.com
Stop and Start Kafka broker on example2.com
Stop and Start Kafka broker on example3.com (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

nishant.shah — Fri, 07 Feb 2020 21:52:10 GMT

Revision 4 posted to Appian Knowledge Base by nishant.shah on 2/7/2020 9:52:10 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on example1.com
Stop and Start Kafka broker on example2.com
Stop and Start Kafka broker on example3.com (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, high availibility, fixed issues, infrastructure

DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

nishant.shah — Fri, 07 Feb 2020 21:50:52 GMT

Revision 3 posted to Appian Knowledge Base by nishant.shah on 2/7/2020 9:50:52 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
example1.com:9092 Reachable ISR: All replicas in sync 
example2.com:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
example3.com:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on example1.com
Stop and Start Kafka broker on example2.com
Stop and Start Kafka broker on example3.com (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, fixed issues, infrastructure

DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

nishant.shah — Fri, 07 Feb 2020 21:47:35 GMT

Revision 2 posted to Appian Knowledge Base by nishant.shah on 2/7/2020 9:47:35 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
localhost1:9092 Reachable ISR: All replicas in sync 
localhost2:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
localhost3:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on localhost1
Stop and Start Kafka broker on localhost2
Stop and Start Kafka broker on localhost3 (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Last Reviewed: February 2020

Tags: kafka, known issues, fixed issues, infrastructure

DRAFT KB-XXXX Kafka Replicas Out of Sync in High Availability

nishant.shah — Fri, 07 Feb 2020 21:42:23 GMT

Revision 1 posted to Appian Knowledge Base by nishant.shah on 2/7/2020 9:42:23 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

One of the Kafka Brokers goes out of sync in the High Availability setup. The output of the Status script contains the following:

Kafka Broker Connectivity ----------------------------------------------------
localhost1:9092 Reachable ISR: All replicas in sync 
localhost2:9092 Reachable controller ISR: Replicas not in sync
 not in sync: __consumer_offsets-0
 not in sync: __consumer_offsets-12
 not in sync: __consumer_offsets-15
 not in sync: __consumer_offsets-18
 not in sync: __consumer_offsets-21 
 not in sync: serviceManager.transaction.download-stats-0
 not in sync: serviceManager.transaction.execution01-0
 not in sync: serviceManager.transaction.execution02-0
 not in sync: serviceManager.transaction.portal-0 
localhost3:9092 Reachable ISR: All replicas in sync

Cause

This issue has been addressed via AN-141918 in the following hotfixes/versions:

Action

Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Workaround

Stop and Start Kafka broker on localhost1
Stop and Start Kafka broker on localhost2
Stop and Start Kafka broker on localhost3 (Controller)

Please ensure to Stop and Start Kafka on Controller last

Affected Versions

This article applies to Appian 19.3 and earlier.

Tags: kafka, known issues, fixed issues, infrastructure