KB-2322 Application server fails to start due to improperly configured search server credentials

Note for Appian on Kubernetes: In operator versions v0.181.0 and later, the Appian Operator can handle the search server password, so you no longer need to manually specify one. If you had previously set a search server password, you can refer to the steps listed under Operator v0.181.0 and Later if you wish to have the Appian Operator manage the password instead.

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

The cited errors indicate that the application server is unable to verify the health of the search server. The references to HTTP/1.1 403 Forbidden indicate that this is due to failure to properly authenticate using the provided credentials.

The most common cause of this error messaging is a misconfiguration of the password associated with the search server. This is a newly required property and was introduced in Appian 24.3 as part of the upgrade to Elasticsearch 8.

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Appian on Kubernetes

Operator v0.176.0 and Earlier

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
 customProperties:
   conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Operator v0.181.0 and Later

The following assumes your Appian Operator is v0.181.0 or later, and your Appian major version includes the July 03, 2025 hotfix release or later.

Manually specifying a search server password in the CR or passwords-properties secret is no longer needed. If a password had been previously set one and would like to be removed, follow the steps below.

Edit the Appian CR file to remove the .spec.customProperties.conf.search-server.user.appian.password field:
```
spec:
   customProperties:
```
Recreate the passwords-properties secret. If the passwords-properties secret stored other credentials apart from conf.password.SearchServer, take note of them as they will need to added back.
1. Delete the secret:
  kubectl -n <NAMESPACE> delete secret passwords-properties
2. Edit the passwords.properties file to remove the conf.password.SearchServer entry. If the file does not exist, create a new one and add back any other credentials that the secret previously stored. If passwords-properties did not have any other credentials previously, a blank passwords.properties file is fine.
3. Re-create the passwords-properties secret:
  kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties
Start Appian. If the above steps were done on a live site, restart the search-server and webapp pods by deleting them for the changes to be applied.

Legacy Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 3.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: July 2025