KB-2294 Tomcat startup hangs indefinitely after "Search Server is Healthy" message

Symptoms

After starting Tomcat, component startup halts indefinitely after the message "Appian component Search Server is healthy" is printed to tomcat-stdOut.log. The message "Appian component Engines is healthy" does not print at any time.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Search Server is healthy

If the root cause is related to a mismatch in appian-topology.xml entries, this can be reflected in the logging for the affected components.

In this example, references to analytics02 at node1.hostname.com:5022 are missing from the logs on Node 2 because the engine does not have an entry within the local appian-topology.xml file.

Node 1: engine-client.log

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node1.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node1.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-11] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from NEW to CONNECTING

Node 2: engine-client.log

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from null to NEW

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node2.hostname.com:5022 state changed from NEW to CONNECTING

YYYY-MM-DD HH:MM:SS,mmm [engine-channel-thread-20] INFO  - Connection for analytics02 at node3.hostname.com:5022 state changed from NEW to CONNECTING

Cause

This behavior can occur when there are issues with the connection between Tomcat and the engines. This is most commonly caused by mismatched <APPIAN_HOME>/conf/appian.sec or <APPIAN_HOME>/conf/appian-topology.xml files in Highly Available (HA) environments, blocked ports, or other network issues.  

Action

1. If the environment is HA, please ensure that the necessary files and folders in <APPIAN_HOME>/conf/ have been shared as described in the documentation. 
2. If the environment is HA, please ensure that the appian-topology.xml files match between all nodes in all locations.
3. If all files and folders have been shared as described in the documentation, or if the environment is not HA, please create a Technical Support Case and attach the following logs:
   1. <APPIAN_HOME>/logs/tomcat-stdOut.log 
   2. <APPIAN_HOME>/logs/service_manager.log 
   3. <APPIAN_HOME>/logs/engine-client.log

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: March 2024