Home » Support » Appian Knowledge Base » KB-2322 Application server fails to start due to improperly configured search server credentials

KB-2322 Application server fails to start due to improperly configured search server credentials

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

The cited errors indicate that the application server is unable to verify the health of the search server. The references to HTTP/1.1 403 Forbidden indicate that this is due to failure to properly authenticate using the provided credentials.

The most common cause of this error messaging is a misconfiguration of the password associated with the search server. This is a newly required property and was introduced in Appian 24.3 as part of the upgrade to Elasticsearch 8.

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

  1. Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Self-Managed Appian

  1. Shutdown the application server and search server in the event they are already running.
  2. Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
  3. Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
  4. Save changes.
  5. Navigate to <APPIAN_HOME>/conf/.
  6. Make a copy of the passwords.properties.examples file and rename it passwords.properties.
  7. Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
  8. Save changes.
  9. Proceed to start up the search server then application server.

Appian on Kubernetes

  1. Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
  2. Edit the Appian CR file to contain the following entry under customProperties underneath spec:
    spec:
   customProperties:
      conf.search-server.user.appian.password: "<VALUE>"
  3. Create a file named passwords.properties that has the following contents:
    conf.password.SearchServer=<VALUE>
  4. Create the following secret within the same namespace as the appian object:
    kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties
  5. Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:
    spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties
  6. Apply the changes made to the Appian CR file.
  7. To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Related
Recommended
© 2024 Appian. All rights reserved.
We want to hear from you! Submit a review on Gartner or G2.