KB-2294 Tomcat startup hangs indefinitely after "Search Server is Healthy" message

pauline.delacruz — Mon, 15 Apr 2024 21:27:25 GMT

Current Revision posted to Appian Knowledge Base by pauline.delacruz on 4/15/2024 9:27:25 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log. The message "Appian component Engines is healthy" does not print at any time.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

If the root cause is related to a mismatch in appian-topology.xml entries, this can be reflected in the logging for the affected components.

In this example, references to analytics02 at node1.hostname.com:5022 are missing from the logs on Node 2 because the engine does not have an entry within the local appian-topology.xml file.

Node 1: engine-client.log

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node1.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node1.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from NEW to CONNECTING

Node 2: engine-client.log

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from NEW to CONNECTING

Cause

This behavior can occur when there are issues with the connection between Tomcat and the engines. This is most commonly caused by mismatched <APPIAN_HOME>/conf/appian.sec or <APPIAN_HOME>/conf/appian-topology.xml files in Highly Available (HA) environments, blocked ports, or other network issues.

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If the environment is HA, please ensure that the appian-topology.xml files match between all nodes in all locations.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: March 2024

Tags: Tomcat, engines, engine, search server

KB-2294 Tomcat startup hangs indefinitely after "Search Server is Healthy" message

Maggie Deppe-Walker — Wed, 21 Feb 2024 05:59:48 GMT

Revision 14 posted to Appian Knowledge Base by Maggie Deppe-Walker on 2/21/2024 5:59:48 AM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log, and the message "Appian component Engines is healthy" does not print at any time.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when there are issues with the connection between Tomcat and the engines. This is most commonly caused by mismatched <APPIAN_HOME>/conf/appian.sec files in Highly Available (HA) environments, blocked ports, or other network issues.

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

[DRAFT SP-9556] KB-XXXX Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 11 Jan 2024 19:51:00 GMT

Revision 13 posted to Appian Knowledge Base by rachel.gillespie on 1/11/2024 7:51:00 PM

Symptoms

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

[DRAFT SP-9556] KB-XXXX Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 11 Jan 2024 19:46:45 GMT

Revision 12 posted to Appian Knowledge Base by rachel.gillespie on 1/11/2024 7:46:45 PM

Symptoms

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

In Highly Available (HA) environments, this issue can occur when <APPIAN_HOME>/conf/appian.sec is not shared across all servers.

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

[DRAFT SP-9556] KB-XXXX Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 11 Jan 2024 19:46:26 GMT

Revision 11 posted to Appian Knowledge Base by rachel.gillespie on 1/11/2024 7:46:26 PM

Symptoms

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

In Highly Available (HA) environments, this issue can occur when <APPIAN_HOME>/conf/appian.sec is not shared across all servers.

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

[DRAFT SP-9556] KB-XXXX Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 11 Jan 2024 19:45:57 GMT

Revision 10 posted to Appian Knowledge Base by rachel.gillespie on 1/11/2024 7:45:57 PM

Symptoms

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

In Highly Available (HA) environments, this issue can occur when appian.sec is not shared across all servers.

Action

If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

[DRAFT SP-9556] KB-XXXX Tomcat startup hangs indefinitely after "Search Server is Healthy" message

Maggie Deppe-Walker — Tue, 09 Jan 2024 06:04:17 GMT

Revision 9 posted to Appian Knowledge Base by Maggie Deppe-Walker on 1/9/2024 6:04:17 AM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

If the environment is Highly Available (HA), please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

Maggie Deppe-Walker — Tue, 09 Jan 2024 06:02:00 GMT

Revision 8 posted to Appian Knowledge Base by Maggie Deppe-Walker on 1/9/2024 6:02:00 AM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

If the environment is Highly Available (HA), please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: Jan 2024

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Mon, 08 Jan 2024 20:59:17 GMT

Revision 7 posted to Appian Knowledge Base by rachel.gillespie on 1/8/2024 8:59:17 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

If the environment is Highly Available (HA), please ensure that all files in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
If all files have been shared properly, or if your environment is not HA, please create a ticket with Appian Support and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Mon, 08 Jan 2024 20:56:07 GMT

Revision 6 posted to Appian Knowledge Base by rachel.gillespie on 1/8/2024 8:56:07 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state and your environment is Highly Available (HA), please ensure that all files in <APPIAN_HOME>/conf/ have been shared as described in the documentation.
1. If Tomcat has detected that engines are in a healthy state, please open a Support Case with Appian Support and attach the following logs:
  1. <APPIAN_HOME>/logs/tomcat-stdOut.log
  2. <APPIAN_HOME>/logs/service_manager.log
  3. <APPIAN_HOME>/logs/engine-client.log
If all files have been shared properly, or if your environment is not HA, please create a ticket with Appian Support and attach the following logs:
1. <APPIAN_HOME>/logs/tomcat-stdOut.log
2. <APPIAN_HOME>/logs/service_manager.log
3. <APPIAN_HOME>/logs/engine-client.log

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 04 Jan 2024 18:43:28 GMT

Revision 5 posted to Appian Knowledge Base by rachel.gillespie on 1/4/2024 6:43:28 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state, review service_manager.log and engine-client.log to begin troubleshooting.

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 04 Jan 2024 18:41:02 GMT

Revision 4 posted to Appian Knowledge Base by rachel.gillespie on 1/4/2024 6:41:02 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state, review service_manager.log and engine-client.log to begin troubleshooting.

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 04 Jan 2024 18:40:00 GMT

Revision 3 posted to Appian Knowledge Base by rachel.gillespie on 1/4/2024 6:40:00 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state, review service_manager.log and engine-client.log to begin troubleshooting the engines.

Tags: Tomcat, engines, engine, search server

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 04 Jan 2024 18:36:33 GMT

Revision 2 posted to Appian Knowledge Base by rachel.gillespie on 1/4/2024 6:36:33 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state, review service_manager.log and engine-client.log to begin troubleshooting the engines.

Tomcat startup hangs indefinitely after "Search Server is Healthy" message

rachel.gillespie — Thu, 04 Jan 2024 18:35:54 GMT

Revision 1 posted to Appian Knowledge Base by rachel.gillespie on 1/4/2024 6:35:54 PM

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

Cause

This issue can occur when the engines are in an unhealthy state. This can be true even if <APPIAN_HOME>/services/bin/status.sh indicates that all engines are running.

Action

Check tomcat-stdOut.log to see if Tomcat has detected that the engines are healthy

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy

If Tomcat has not yet detected that the engines are in a healthy state, review service_manager.log and engine-client.log to begin troubleshooting the engines.