After a nightly restart, the following logs were generated. After several attempts waiting for a "healthy search service", the Appian Application on tomcat is stopped.
2020-06-01 18:15:40,219 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300] 2020-06-01 18:15:40,529 [wait-for-component] INFO com.appiancorp.tools.WaitForServers - Waiting for analytics to be ready... 2020-06-01 18:15:41,730 [wait-for-component] INFO com.appiancorp.tools.WaitForServers - ... done. 2020-06-01 18:15:41,730 [wait-for-component] INFO com.appiancorp.common.startup.WaitForStatefulComponents - Appian component Engines is healthy 2020-06-01 18:15:42,728 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 7C0DB711","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"} 2020-06-01 18:15:42,728 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{UYnfk3BISlOiqBSxeOl1dQ}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{FxYO0pv_QueunHXDfTbJpw}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{oe82IEx_QJ6JqjAXPu_Zdg}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}] 2020-06-01 18:15:42,728 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes: [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}] 2020-06-01 18:15:42,760 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8 2020-06-01 18:16:12,769 [wait-for-component] INFO com.appian.dl.repo.es.client.TransportClientManager - Shutting down client. 2020-06-01 18:16:12,779 [wait-for-component] INFO com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy... 2020-06-01 18:16:17,779 [wait-for-component] INFO com.appiancorp.common.persistence.search.SearchServerClientConfiguration - Using the Search Server client configuration specified in /datadisk/apps/hc_appian/conf/appian-topology.xml. 2020-06-01 18:16:17,785 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - initializing client with server addresses: [appian-stg-web-01.hc-sc.gc.ca:9300, appian-stg-wb-02.hc-sc.gc.ca:9300, appian-stg-wb-03.hc-sc.gc.ca:9300] 2020-06-01 18:16:17,862 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client settings: {"client.transport.ping_timeout":"20s","client.transport.sniff":"true","client.type":"transport","cluster.name":"appian-search-cluster","http.type.default":"netty4","network.server":"false","node.name":"Client 45707BF8","path.logs":"/datadisk/apps/hc_appian/logs","transport.features.transport_client":"true","transport.ping_schedule":"5s","transport.type.default":"netty4"} 2020-06-01 18:16:17,862 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client configured nodes: [{#transport#-1}{vVwyUmf4Qo2PlDGHhnZiog}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}, {#transport#-2}{WGhsSaMMTvCIGRepoJtTRg}{appian-stg-wb-03.hc-sc.gc.ca}{10.170.0.66:9300}, {#transport#-3}{0J600F3uTbGRxm-b-BtNSw}{appian-stg-wb-02.hc-sc.gc.ca}{10.170.0.65:9300}] 2020-06-01 18:16:17,862 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - client connected nodes: [{Node appian-stg-web-01.hc-sc.gc.ca:9300}{EQ5OKj0eQJ-7JqnH_hVGfw}{G0dxq3kWTcScQmZt_h6kaw}{appian-stg-web-01.hc-sc.gc.ca}{10.170.0.62:9300}] 2020-06-01 18:16:17,865 [wait-for-component] INFO com.appiancorp.common.persistence.search.AppianSearchSpringConfig - search server: productName=ElasticSearch, productVersion=6.8.8, driverName=org.elasticsearch.transport.client.PreBuiltTransportClient, driverVersion=6.8.8 2020-06-01 18:16:47,869 [wait-for-component] INFO com.appian.dl.repo.es.client.TransportClientManager - Shutting down client. ...skipping... 2020-06-01 18:18:33,124 [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - MasterNotDiscoveredException[null] 2020-06-01 18:18:33,124 [wait-for-component] INFO com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...Has anyone seen this before and what did you do to recover the node?
Discussion posts and replies are publicly visible
Was this issue resolved?
This problem happened a long time ago, so I think it has been solved pou
I resolved this issue be re-configuring my firewall rule to allow the ports used by the search across all nodes in my distributed installation.