KB-2355 One or more Zookeeper node(s) fail to start after applying hotfix

pauline.delacruz — Mon, 03 Nov 2025 16:52:50 GMT

Current Revision posted to Appian Knowledge Base by pauline.delacruz on 11/3/2025 4:52:50 PM

Symptoms

After deploying a hotfix released after September 18, 2025 where Zookeeper upgrades from 3.9.3 to 3.9.4 (found in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on one or two nodes.

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

For Appian on Kubernetes, the affected Zookeeper pod(s) are stuck on CRASHLOOPBACKOFF.
For legacy self-managed Appian, starting engines show the following on affected node(s).
```
Waiting for Zookeeper to be available 
```

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in an earlier version.

Action

Appian on Kubernetes

Set the impacted Zookeeper pod to sleep.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod.
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
Exec into the Zookeeper pod.
- kubectl -n <namespace> exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the sleep command.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
If there is another affected pod, repeat the steps above to remediate.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on the affected node(s).
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian applying a hotfix released on September 18, 2025 and later.

Last Reviewed: November 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

KB-2355 One or more Zookeeper node(s) fail to Start after applying hotfix

pauline.delacruz — Mon, 03 Nov 2025 16:50:21 GMT

Revision 25 posted to Appian Knowledge Base by pauline.delacruz on 11/3/2025 4:50:21 PM

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

For Appian on Kubernetes, the affected Zookeeper pod(s) are stuck on CRASHLOOPBACKOFF.
For legacy self-managed Appian, starting engines show the following on affected node(s).
```
Waiting for Zookeeper to be available 
```

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in an earlier version.

Action

Appian on Kubernetes

Set the impacted Zookeeper pod to sleep.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod.
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
Exec into the Zookeeper pod.
- kubectl -n <namespace> exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the sleep command.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
If there is another affected pod, repeat the steps above to remediate.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on the affected node(s).
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian applying a hotfix released on September 18, 2025 and later.

Last Reviewed: November 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

pauline.delacruz — Mon, 03 Nov 2025 16:39:40 GMT

Revision 24 posted to Appian Knowledge Base by pauline.delacruz on 11/3/2025 4:39:40 PM

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

For Appian on Kubernetes, the affected Zookeeper pod(s) are stuck on CRASHLOOPBACKOFF.
For legacy self-managed Appian, starting engines show the following on affected node(s).
```
Waiting for Zookeeper to be available 
```

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in an earlier version.

Action

Appian on Kubernetes

Set the impacted Zookeeper pod to sleep.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod.
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
Exec into the Zookeeper pod.
- kubectl -n <namespace> exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the sleep command.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
If there is another affected pod, repeat the steps above to remediate.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on the affected node(s).
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian applying a hotfix released on September 18, 2025 and later.

Last Reviewed: November 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

pauline.delacruz — Wed, 22 Oct 2025 22:00:26 GMT

Revision 23 posted to Appian Knowledge Base by pauline.delacruz on 10/22/2025 10:00:26 PM

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

For Appian on Kubernetes, the affected Zookeeper pod(s) are stuck on CRASHLOOPBACKOFF.
For legacy self-managed Appian, starting engines show "Waiting for Zookeeper to be available" on affected node(s).

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Set the impacted Zookeeper pod to sleep.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod.
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
Exec into the Zookeeper pod.
- kubectl -n <namespace> exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the sleep command.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
If there is another affected pod, repeat the steps above to remediate.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on the affected node(s).
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian applying a hotfix released on September 18, 2025 and later.

Last Reviewed: October 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

pauline.delacruz — Wed, 22 Oct 2025 21:57:43 GMT

Revision 22 posted to Appian Knowledge Base by pauline.delacruz on 10/22/2025 9:57:43 PM

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

For Appian on Kubernetes, the affected Zookeeper pod(s) are stuck on CRASHLOOPBACKOFF.
For legacy self-managed Appian, starting engines show "Waiting for Zookeeper to be available" on affected node/s.

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Set the Zookeeper pod to sleep.
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod(s).
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
Go into the affected Zookeeper pod
- kubectl -n <namespace> exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the sleep command
- kubectl -n <namespace> patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl -n <namespace> delete pod <ZOOKEEPER POD>
If there is another affected pod, repeat the steps above to remediate.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on the affected node(s).
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian applying a hotfix released on September 18, 2025 and later.

Last Reviewed: October 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Mon, 13 Oct 2025 21:40:39 GMT

Revision 21 posted to Appian Knowledge Base by kaitlyn.wee on 10/13/2025 9:40:39 PM

KB-XXXX Zookeeper Fails to Start

Symptoms

After deploying a hotfix or upgrading Appian where Zookeeper upgrades from 3.9.3 to 3.9.4 (version of Zookeeper located in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on one or two nodes. For Appian on Kubernetes, affected Zookeeper is stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, starting engines show "Waiting for Zookeeper to be available" on affected node/s.

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER POD>
If there is another affected node, repeat the steps above on the respective node.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node/s
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian deploying a hotfix on or after September 18, 2025 (e.g., 25.2.605.0 and 25.3.335.0) from a hotfix released before that date. Specifically, if the Zookeeper is being upgraded from 3.9.3 to 3.9.4.

Last Reviewed: October 2025

Tags: known issues, zookeeper, fails, upgrade, hotfix, restart

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Mon, 13 Oct 2025 21:39:40 GMT

Revision 20 posted to Appian Knowledge Base by kaitlyn.wee on 10/13/2025 9:39:40 PM

KB-XXXX Zookeeper Fails to Start

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER POD>
If there is another affected node, repeat the steps above on the respective node.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node/s
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Mon, 13 Oct 2025 21:39:18 GMT

Revision 19 posted to Appian Knowledge Base by kaitlyn.wee on 10/13/2025 9:39:18 PM

KB-XXXX Zookeeper Fails to Start

Symptoms

On the affected node, the zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <AFFECTED ZOOKEEPER POD>
Go into the affected Zookeeper pod
- kubectl exec -it <AFFECTED ZOOKEEPER POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <AFFECTED ZOOKEEPER POD>
If there is another affected node, repeat the steps above on the respective node.

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node/s
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 21:38:38 GMT

Revision 18 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 9:38:38 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After deploying a hotfix or upgrading Appian where Zookeeper upgrades from 3.9.3 to 3.9.4 (version of Zookeeper located in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on node 1/3. For Appian on Kubernetes, Zookeeper on node 1/3 is stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, starting engines show "Waiting for Zookeeper to be available" on node 1/3.

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects a product bug that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 16:10:01 GMT

Revision 17 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 4:10:01 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After deploying a hotfix or upgrading Appian where Zookeeper upgrades from 3.9.3 to 3.9.4 (version of Zookeeper located in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on node 1/3. For Appian on Kubernetes, Zookeeper on node 1/3 stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, Zookeeper on node 1/3 shows "Waiting for Zookeeper to be available".

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:57:31 GMT

Revision 16 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:57:31 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After deploying a hotfix or upgrading Appian where Zookeeper upgrades from 3.9.3 to 3.9.4 (version of Zookeeper located in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on node 1/3. For Appian on Kubernetes, Zookeeper on node 1/3 stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, Zookeeper on node 1/3 shows "Waiting for Zookeeper to be available".

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:39:32 GMT

Revision 15 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:39:32 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After upgrading from Zookeeper 3.9.3 to Zookeeper 3.9.4 (version located in <APPIAN_HOME>/services/) on a High Availability (HA) site, Zookeeper fails to start on node 1/3. For Appian on Kubernetes, Zookeeper on node 1/3 stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, Zookeeper on node 1/3 shows "Waiting for Zookeeper to be available".

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:38:42 GMT

Revision 14 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:38:42 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:38:10 GMT

Revision 13 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:38:10 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data only on affected node
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:36:54 GMT

Revision 12 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:36:54 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:36:35 GMT

Revision 11 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:36:35 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Fails to Start

kaitlyn.wee — Fri, 10 Oct 2025 15:35:55 GMT

Revision 10 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:35:55 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After upgrading from Zookeeper 3.9.3 to Zookeeper 3.9.4 (version located in <APPIAN_HOME>/services/) on High Availability (HA) site, Zookeeper fails to start on node 1/3. For Appian on Kubernetes, Zookeeper on node 1/3 stuck on CRASHLOOPBACKOFF. For legacy self-managed Appian, Zookeeper on node 1/3 shows "Waiting for Zookeeper to be available"

The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data
- rm -rf zookeeper/data/*
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node
- rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

kaitlyn.wee — Fri, 10 Oct 2025 15:29:43 GMT

Revision 9 posted to Appian Knowledge Base by kaitlyn.wee on 10/10/2025 3:29:43 PM

KB-XXXX Zookeeper on Node 1/3 Fails to Start

Symptoms

After upgrading from Zookeeper 3.9.3 to Zookeeper 3.9.4 on a High Availability (HA) site, Zookeeper fails to start on node 1/3. The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is unable to startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4, which detects previously silent data corruption that had gone unnoticed in earlier version.

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace
- kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod
- kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data
- cd zookeeper
- rm -rf data
Remove the command field
- kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod
- kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components gracefully
Delete the contents of Zookeeper's data only on affected node rm -rf <APPIAN HOME>/services/data/zookeeper/*
Start all Appian components

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

kaitlyn.wee — Thu, 09 Oct 2025 20:36:35 GMT

Revision 8 posted to Appian Knowledge Base by kaitlyn.wee on 10/9/2025 8:36:35 PM

This article serves as a template that you can copy/paste into a new article for use later on.

The title of the article must be under 256 characters (less depending on the types of characters used in the title as this is a limitation of Telligent). It must describe as concisely as possible the issue faced. If there is a short enough error message, it should be included in the title for readability. Those using the Knowledge Base should get a good idea of what the article is about just by reading the title.

This description is for informational purposes only. There should be no text above the "Symptoms" header when using this template. The only exception to this is if the article you are writing is an issue that has been resolved, include the following green box above the Symptoms section:

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

To include this box, add the following in the source code of your article:

<table style='background-color: #bcedaf;'>
<tbody>
<tr>
<td>This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.</td>
</tr>
</tbody>
</table>

KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

Symptoms

After upgrading from Zookeeper 3.9.3 to Zookeeper 3.9.4 on an High Availability (HA) site, Zookeeper is stuck in a CRASHLOOPBACKOFF on node 1/3. The zookeeper-XXX.log (located in <APPIAN HOME>/logs/service-manager/zookeeper) indicates that the node is crashing up startup due to a corrupted or inconsistent state.

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

The zookeeper logs could also indicate inability to connect.


YYYY-MM-DD HH:MM:SS,mmm [main-SendThread(appian-zookeeper-0.appian-zookeeper-headless.12335.svc.cluster.local:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181
java.net.UnknownHostException: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local
 at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1197) ~[zookeeper-3.9.4.jar:3.9.4]
YYYY-MM-DD HH:MM:SS,mmm [main-SendThread(appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x200001199b30009 for server appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
java.lang.IllegalArgumentException: Unable to canonicalize address appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181 because it's not resolvable
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1150) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1200) ~[zookeeper-3.9.4.jar:3.9.4]

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace kubectl patch statefulset <ZOOKEEPER STS> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the affected Zookeeper pod kubectl delete pod <ZOOKEEPER-0 POD>
Go into the affected Zookeeper pod kubectl exec -it <ZOOKEEPER-0 POD> -- bash
Delete the contents of zookeeper/data
1. cd zookeeper
2. rm -rf data
Remove the command field kubectl patch statefulset <ZOOKEEPER-0 POD> --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod kubectl delete pod <ZOOKEEPER-0 POD>

Legacy Self-Managed Appian

Stop all Appian components
Delete the contents of Zookeeper's data only on affected node rm -rf <APPIAN HOME>/services/data/zookeeper/*
Restart engines

Workaround

Restore the site to the previous version.

Affected Versions

This article applies to all High Availability (HA) versions of Appian upgrading Zookeeper 3.9.3 to Zookeeper 3.9.4.

Last Reviewed: October 2025

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

kaitlyn.wee — Thu, 09 Oct 2025 20:17:43 GMT

Revision 7 posted to Appian Knowledge Base by kaitlyn.wee on 10/9/2025 8:17:43 PM

This article serves as a template that you can copy/paste into a new article for use later on.

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

To include this box, add the following in the source code of your article:

KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

Symptoms

YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
YYYY-MM-DD HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

The zookeeper logs could also indicate inability to connect.


YYYY-MM-DD HH:MM:SS,mmm [main-SendThread(appian-zookeeper-0.appian-zookeeper-headless.12335.svc.cluster.local:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181
java.net.UnknownHostException: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local
 at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1197) ~[zookeeper-3.9.4.jar:3.9.4]
YYYY-MM-DD HH:MM:SS,mmm [main-SendThread(appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x200001199b30009 for server appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
java.lang.IllegalArgumentException: Unable to canonicalize address appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181 because it's not resolvable
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1150) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1200) ~[zookeeper-3.9.4.jar:3.9.4]

Cause

These errors are caused by the upgrade from Zookeeper 3.9.3 to Zookeeper 3.9.4

Action

Appian on Kubernetes

Request cluster admin kubectl access to the site namespace kubectl patch statefulset appian-zookeeper --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["sleep", "infinity"]}]'
Restart the Zookeeper pod kubectl delete pod appian-zookeeper-0
Go into the Zookeeper pod kubectl delete pod appian-zookeeper-0
Delete the contents of zookeeper/data
1. cd zookeeper
2. rm -rf data
Remove the command field kubectl patch statefulset appian-zookeeper --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/command"}]'
Restart the Zookeeper pod kubectl delete pod appian-zookeeper-0

Legacy Self-Managed Appian

Workaround

Restore the site to the previous version.

Affected Versions

This section includes any relevant version information for Appian or other third/party configurations. Some examples of valid affected versions are as follows:

This article applies to all versions of Appian.
This article applies to all self-managed versions of Appian.
This article applies to all versions of Appian Cloud.
This article applies to Appian 7.11 and later.
This article applies to Appian 16.2 and earlier.
This article applies to Appian 7.11 and 16.1.
This article applies to Appian 16.1, 16.2, and 16.3.
This article applies to self-managed Appian 19.3 and earlier.
This article applies to all versions of Appian from Appian 7.10 to Appian 16.2.
This article applies to all versions of Appian using JBoss EAP 6.4.9 as an application server and Internet Explorer 9 as a web browser.

Last Reviewed: Month YYYY

[DRAFT SUPP-1707] KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

kaitlyn.wee — Thu, 09 Oct 2025 19:04:05 GMT

Revision 6 posted to Appian Knowledge Base by kaitlyn.wee on 10/9/2025 7:04:05 PM

This article serves as a template that you can copy/paste into a new article for use later on.

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

To include this box, add the following in the source code of your article:

KB-XXXX Zookeeper on Node 1/3 Stuck in CrashLoopBackOff State Upon Site Startup

Symptoms

After upgrading from Zookeeper 3.9.3 to Zookeeper 3.9.4 on an High Availability (HA) site, Zookeeper is stuck in a CRASHLOOPBACKOFF.

HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, exiting abnormally
java.lang.IllegalStateException: Committed proposal cached out of order: 0x5000000xxx is not the next proposal of 0x5000000xxx
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:341)
 at org.apache.zookeeper.server.ZKDatabase.addCommittedProposal(ZKDatabase.java:313)
 at org.apache.zookeeper.server.ZKDatabase.access$000(ZKDatabase.java:73)
 at org.apache.zookeeper.server.ZKDatabase$1.onTxnLoaded(ZKDatabase.java:278)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.fastForwardFromEdits(FileTxnSnapLog.java:360)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.lambda$restore$0(FileTxnSnapLog.java:267)
 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:312)
 at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:290)
 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:1149)
 at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:1135)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:229)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137)
 at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91)
HH:MM:SS.mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1

YYYY-MM-DD HH:MM:SS,mmm [main-SendThread(appian-zookeeper-0.appian-zookeeper-headless.12335.svc.cluster.local:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181
java.net.UnknownHostException: appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local
 at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1197) ~[zookeeper-3.9.4.jar:3.9.4]
2025-09-27 08:06:45,886 [main-SendThread(appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x200001199b30009 for server appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
java.lang.IllegalArgumentException: Unable to canonicalize address appian-zookeeper-1.appian-zookeeper-headless.12335.svc.cluster.local/<unresolved>:2181 because it's not resolvable
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1150) ~[zookeeper-3.9.4.jar:3.9.4]
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1200) ~[zookeeper-3.9.4.jar:3.9.4]

This section is required for this template. When writing a cause for a particular article, ensure the following is taken into consideration:

The cause is concise and does not disclose more information than absolutely necessary.
The cause does not share any Appian source code. Source code for plugins/shared components are fine.
If this is an issue outside of the Appian product and the third-party vendor included documentation referencing the bug, provide their documentation for reference.

For any enhancements/defects, use one of the following as applicable:

This issue has been reported to the Appian Product Team. The reference number for this issue is AN-XXXXX.
This product use case has been logged to the Appian Product Team for consideration to add this functionality in the product. Kindly note it is not Appian Support’s policy to disclose how or when a product use case will be implemented.
This issue has been hotfixed via AN-XXXXX in Appian X.X.
This issue has been hotfixed via AN-XXXXX in Appian X.X to X.X.
This use case has been incorporated in Appian X.X.
This use case has been incorporated in Appian X.X to X.X.

When logging a product defect, ensure you use the known issues tag as well as the open issues or fixed issues tag, depending if the issue is fixed or not.

If no cause is known and this is not something Engineering can reproduce/needs to know about, then say “The root cause of this issue is currently unknown.” If you use that as your cause, then a valid action must be present.

Action

This section is required for this template and contains the steps to resolve the issue. If an issue in an article is solved by a hotfix and/or major version of Appian, use the following language to indicate that:

Apply the latest hotfix to your Appian installation.
Upgrade to the latest version of Appian.
Apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

If there is a product issue with Engineering that is open and a workaround for that issue exists, include it in the Action section until the fix is available. After the fix is made available, use that as a workaround instead.

Otherwise, the content in this section should wholly solve a customer’s issue without producing any undesirable side effects. If there is at least one valid action that doesn’t cause undesirable side effects, all other actions with negative side effects should go in the Workaround section.

If there is no action or workaround that resolves the issue, use the following language:

There is no action or workaround currently available for this issue. If you are facing this issue, please open a support case with Appian.

Workaround

This section is optional and should only be used as an alternative action if there is a massive technical debt associated with performing the action in the above section for customers (such as upgrading). It can also be used if there is a solution to the issue in the article but is more inconvenient to implement or it causes other undesirable behavior and wouldn’t warrant being an action.

Affected Versions

This section includes any relevant version information for Appian or other third/party configurations. Some examples of valid affected versions are as follows:

This article applies to all versions of Appian.
This article applies to all self-managed versions of Appian.
This article applies to all versions of Appian Cloud.
This article applies to Appian 7.11 and later.
This article applies to Appian 16.2 and earlier.
This article applies to Appian 7.11 and 16.1.
This article applies to Appian 16.1, 16.2, and 16.3.
This article applies to self-managed Appian 19.3 and earlier.
This article applies to all versions of Appian from Appian 7.10 to Appian 16.2.
This article applies to all versions of Appian using JBoss EAP 6.4.9 as an application server and Internet Explorer 9 as a web browser.

Last Reviewed: Month YYYY