HA Server won't start - waiting for healthy searchserver

After a nightly restart, the following logs were generated.  After several attempts waiting for a "healthy search service", the Appian Application on tomcat is stopped.

2020-06-01 18:15:40,219 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300]
2020-06-01 18:15:40,529 [wait-for-component] INFO  com.appiancorp.tools.WaitForServers - Waiting for analytics to be ready...
2020-06-01 18:15:41,730 [wait-for-component] INFO  com.appiancorp.tools.WaitForServers - ... done.
2020-06-01 18:15:41,730 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 7C0DB711","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"}
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{UYnfk3BISlOiqBSxeOl1dQ}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{FxYO0pv_QueunHXDfTbJpw}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{oe82IEx_QJ6JqjAXPu_Zdg}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}]
2020-06-01 18:15:42,728 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes:  [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}]
2020-06-01 18:15:42,760 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8
2020-06-01 18:16:12,769 [wait-for-component] INFO  com.appian.dl.repo.es.client.TransportClientManager - Shutting down client.
2020-06-01 18:16:12,779 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...
2020-06-01 18:16:17,779 [wait-for-component] INFO  com.appiancorp.common.persistence.search.SearchServerClientConfiguration - Using the Search Server client configuration specified in /datadisk/apps/hc_appian/conf/appian-topology.xml.
2020-06-01 18:16:17,785 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300]
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 45707BF8","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"}
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{vVwyUmf4Qo2PlDGHhnZiog}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{WGhsSaMMTvCIGRepoJtTRg}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{0J600F3uTbGRxm-b-BtNSw}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}]
2020-06-01 18:16:17,862 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes:  [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}]
2020-06-01 18:16:17,865 [wait-for-component] INFO  com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8
2020-06-01 18:16:47,869 [wait-for-component] INFO  com.appian.dl.repo.es.client.TransportClientManager - Shutting down client.
...skipping...
2020-06-01 18:18:33,124 [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - MasterNotDiscoveredException[null]
2020-06-01 18:18:33,124 [wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Has anyone seen this before and what did you do to recover the node?

  Discussion posts and replies are publicly visible

Parents Reply Children
No Data