HA Server won't start - waiting for healthy searchserver

After a nightly restart, the following logs were generated.  After several attempts waiting for a "healthy search service", the Appian Application on tomcat is stopped.

2020-06-01 18:15:40,219 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300]
2020-06-01 18:15:40,529 [wait-for-component] INFO  com.appiancorp.tools.WaitForServers - Waiting for analytics to be ready...
2020-06-01 18:15:41,730 [wait-for-component] INFO  com.appiancorp.tools.WaitForServers - ... done.
2020-06-01 18:15:41,730 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 7C0DB711","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"}
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{UYnfk3BISlOiqBSxeOl1dQ}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{FxYO0pv_QueunHXDfTbJpw}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{oe82IEx_QJ6JqjAXPu_Zdg}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}]
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes:  [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}]
2020-06-01 18:15:42,760 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8
2020-06-01 18:16:12,769 [wait-for-component] INFO  com.appian.dl.repo.es.client.TransportClientManager - Shutting down client.
2020-06-01 18:16:12,779 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...
2020-06-01 18:16:17,779 [wait-for-component] INFO  com.appiancorp.common.persistence.search.SearchServerClientConfiguration - Using the Search Server client configuration specified in /datadisk/apps/hc_appian/conf/appian-topology.xml.
2020-06-01 18:16:17,785 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300]
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 45707BF8","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"}
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{vVwyUmf4Qo2PlDGHhnZiog}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{WGhsSaMMTvCIGRepoJtTRg}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{0J600F3uTbGRxm-b-BtNSw}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}]
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes:  [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}]
2020-06-01 18:16:17,865 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8
2020-06-01 18:16:47,869 [wait-for-component] INFO  com.appian.dl.repo.es.client.TransportClientManager - Shutting down client.
...skipping...
2020-06-01 18:18:33,124 [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - MasterNotDiscoveredException[null]
2020-06-01 18:18:33,124 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Has anyone seen this before and what did you do to recover the node?

  Discussion posts and replies are publicly visible