KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Wed, 23 Jul 2025 20:29:57 GMT

Current Revision posted to Appian Knowledge Base by Ryan Good on 7/23/2025 8:29:57 PM

Note for Appian on Kubernetes: In operator versions v0.181.0 and later, the Appian Operator can handle the search server password, so you no longer need to manually specify one. If you had previously set a search server password, you can refer to the steps listed under Operator v0.181.0 and Later if you wish to have the Appian Operator manage the password instead.

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

The cited errors indicate that the application server is unable to verify the health of the search server. The references to HTTP/1.1 403 Forbidden indicate that this is due to failure to properly authenticate using the provided credentials.

The most common cause of this error messaging is a misconfiguration of the password associated with the search server. This is a newly required property and was introduced in Appian 24.3 as part of the upgrade to Elasticsearch 8.

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Appian on Kubernetes

Operator v0.176.0 and Earlier

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
 customProperties:
   conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Operator v0.181.0 and Later

The following assumes your Appian Operator is v0.181.0 or later, and your Appian major version includes the July 03, 2025 hotfix release or later.

Manually specifying a search server password in the CR or passwords-properties secret is no longer needed. If a password had been previously set one and would like to be removed, follow the steps below.

Edit the Appian CR file to remove the .spec.customProperties.conf.search-server.user.appian.password field:
```
spec:
   customProperties:
```
Recreate the passwords-properties secret. If the passwords-properties secret stored other credentials apart from conf.password.SearchServer, take note of them as they will need to added back.
1. Delete the secret:
  kubectl -n <NAMESPACE> delete secret passwords-properties
2. Edit the passwords.properties file to remove the conf.password.SearchServer entry. If the file does not exist, create a new one and add back any other credentials that the secret previously stored. If passwords-properties did not have any other credentials previously, a blank passwords.properties file is fine.
3. Re-create the passwords-properties secret:
  kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties
Start Appian. If the above steps were done on a live site, restart the search-server and webapp pods by deleting them for the changes to be applied.

Legacy Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 3.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: July 2025

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

pauline.delacruz — Wed, 23 Jul 2025 14:38:45 GMT

Revision 20 posted to Appian Knowledge Base by pauline.delacruz on 7/23/2025 2:38:45 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Appian on Kubernetes

Operator v0.176.0 and Earlier

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
 customProperties:
   conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Operator v0.181.0 and Later

The following assumes your Appian Operator is v0.181.0 or later, and your Appian major version includes the July 03, 2025 hotfix release or later.

Edit the Appian CR file to remove the .spec.customProperties.conf.search-server.user.appian.password field:
```
spec:
   customProperties:
```
Recreate the passwords-properties secret. If the passwords-properties secret stored other credentials apart from conf.password.SearchServer, take note of them as they will need to added back.
1. Delete the secret:
  kubectl -n <NAMESPACE> delete secret passwords-properties
2. Edit the passwords.properties file to remove the conf.password.SearchServer entry. If the file does not exist, create a new one and add back any other credentials that the secret previously stored. If passwords-properties did not have any other credentials previously, a blank passwords.properties file is fine.
3. Re-create the passwords-properties secret:
  kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties
Start Appian. If the above steps were done on a live site, restart the search-server and webapp pods by deleting them for the changes to be applied.

Legacy Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 3.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: July 2025

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

pauline.delacruz — Wed, 23 Jul 2025 14:38:26 GMT

Revision 19 posted to Appian Knowledge Base by pauline.delacruz on 7/23/2025 2:38:26 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Appian on Kubernetes

Operator v0.176.0 and Earlier

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
 customProperties:
   conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Operator v0.181.0 and Later

The following assumes your Appian Operator is v0.181.0 or later, and your Appian major version includes the July 03, 2025 hotfix release or later.

Edit the Appian CR file to remove the .spec.customProperties.conf.search-server.user.appian.password field:
```
spec:
   customProperties:
```
Recreate the passwords-properties secret. If the passwords-properties secret stored other credentials apart from conf.password.SearchServer, take note of them as they will need to added back.
1. Delete the secret:
  kubectl -n <NAMESPACE> delete secret passwords-properties
2. Edit the passwords.properties file to remove the conf.password.SearchServer entry. If the file does not exist, create a new one and add back any other credentials that the secret previously stored. If passwords-properties did not have any other credentials previously, a blank passwords.properties file is fine.
3. Re-create the passwords-properties secret:
  kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties
Start Appian. If the above steps were done on a live site, restart the search-server and webapp pods by deleting them for the changes to be applied.

Legacy Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 3.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: July 2025

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Fri, 07 Feb 2025 17:08:30 GMT

Revision 18 posted to Appian Knowledge Base by Ryan Good on 2/7/2025 5:08:30 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 3.
Save changes.
Proceed to start up the search server then application server.

Appian on Kubernetes

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
   customProperties:
      conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Wed, 25 Sep 2024 20:30:23 GMT

Revision 17 posted to Appian Knowledge Base by Ryan Good on 9/25/2024 8:30:23 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Self-Managed Appian

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Appian on Kubernetes

Scale down the search server and webapp stateful sets to have 0 replicas so that neither have running pods.
Edit the Appian CR file to contain the following entry under customProperties underneath spec:
```
spec:
   customProperties:
      conf.search-server.user.appian.password: "<VALUE>"
```
Create a file named passwords.properties that has the following contents:
conf.password.SearchServer=<VALUE>
Create the following secret within the same namespace as the appian object:
kubectl -n <NAMESPACE> create secret generic passwords-properties --from-file=passwords.properties

Add the following value to the passwordsPropertiesSecretName entry under webapp in the Appian CR file:

spec:
   customProperties:
      ...
   webapp:
      passwordsPropertiesSecretName: passwords-properties

Apply the changes made to the Appian CR file.
To bring up the pods for the components, scale up the search server and webapp stateful sets to their previous number of replicas.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Mon, 23 Sep 2024 19:53:05 GMT

Revision 16 posted to Appian Knowledge Base by Ryan Good on 9/23/2024 7:53:05 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 6 characters long.

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 10 Sep 2024 21:58:11 GMT

Revision 15 posted to Appian Knowledge Base by Ryan Good on 9/10/2024 9:58:11 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameter(s):

Must be at least 7 characters long.

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 10 Sep 2024 21:57:06 GMT

Revision 14 posted to Appian Knowledge Base by Ryan Good on 9/10/2024 9:57:06 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

pauline.delacruz — Thu, 05 Sep 2024 22:08:10 GMT

Revision 13 posted to Appian Knowledge Base by pauline.delacruz on 9/5/2024 10:08:10 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: Tomcat, application server, search server, infrastructure

KB-2322 Application server fails to start due to improperly configured search server credentials

pauline.delacruz — Thu, 05 Sep 2024 22:07:09 GMT

Revision 12 posted to Appian Knowledge Base by pauline.delacruz on 9/5/2024 10:07:09 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

The cited errors indicate that the application server is unable to verify the health of the search server. The references to HTPP/1.1 403 Forbidden indicate that this is due to failure to properly authenticate using the provided credentials.

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

Tags: search server, infrastructure

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:40:41 GMT

Revision 11 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:40:41 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:38:37 GMT

Revision 10 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:38:37 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above and save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the new file and fill out the conf.password.SearchServer entry with the password from step 2 and save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:37:01 GMT

Revision 9 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:37:01 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password fit the following parameters, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above and save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the file and fill out the conf.password.SearchServer entry with the new password from step 2 and save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:37:01 GMT

Revision 8 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:37:01 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password fit the following parameters, the following steps can be performed.

Shutdown the application server and search server in the event they are already running.
Navigate to <APPIAN_HOME>/search-server/conf/custom.properties.
Modify the property conf.search-server.user.appian.password such that its value fits the requirements outlined above and save changes.
Navigate to <APPIAN_HOME>/conf/.
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the file and fill out the conf.password.SearchServer entry with the new password from step 2 and save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:30:59 GMT

Revision 7 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:30:59 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password fit the following parameters, the following steps can be performed.

Shutdown the application server and search server, if they are already running.
Modify the password provided in <APPIAN_HOME>/search-server/conf/custom.properties to fit the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the file and fill out the conf.password.SearchServer entry with the new password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:29:14 GMT

Revision 6 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:29:14 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password fit the following parameters, the following steps can be performed.

Shutdown the application server and search server, if they are already running.
Modify the password provided in <APPIAN_HOME>/search-server/conf/custom.properties to fit the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the file and fill out the conf.password.SearchServer entry with the new password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:28:21 GMT

Revision 5 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:28:21 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS.mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
		at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
		at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) ~[?:?]
YYYY-MM-DD HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck - ElasticsearchStatusException[Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/health] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]]

Cause

Action

A password used to satisfy the requirements of Elasticsearch 8 must fit the following parameters:

Must be at least 7 characters long.
Must not contain the following symbol: (=)
Must not be preceded in any form by the following symbol: (#)

To apply a new password fit the following parameters, the following steps can be performed.

Shutdown the application server and search server, if they are already running.
Modify the password provided in <APPIAN_HOME>/search-server/conf/custom.properties to fit the requirements outlined above.
Save changes.
Navigate to <APPIAN_HOME>/conf/
Make a copy of the passwords.properties.examples file and rename it passwords.properties.
Navigate to the bottom of the file and fill out the conf.password.SearchServer entry with the new password from step 2.
Save changes.
Proceed to start up the search server then application server.

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:10:32 GMT

Revision 4 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:10:32 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above excerpts are the following traces referencing the Search Server.

HH:MM:SS,mmm [wait-for-component] ERROR com.appiancorp.common.persistence.search.RestClientManagerUtil - error initializing client
com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException: The search server cannot be reached. Failed to connect to server at [localhost:9200]. Check that the search server is started. If running multiple application servers, check that appian-topology.xml is properly configured with the search cluster details. The appian-topology.xml file must be distributed to each /conf/ and /search-server/conf/ directory. See documentation for details.
	at com.appiancorp.suiteapi.common.exceptions.AppianRuntimeException.<init>(AppianRuntimeException.java:45) ~[appian-suiteapi-24.3.205.0.jar:?]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:69) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.healthcheck.SearchServerHealthCheck.isComponentHealthy(SearchServerHealthCheck.java:34) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.waitUntilComponentIsHealthy(WaitForStatefulComponents.java:105) ~[appian-ae-24.3.205.0.jar:?]
	at com.appiancorp.common.startup.WaitForStatefulComponents.lambda$startHealthChecks$0(WaitForStatefulComponents.java:117) ~[appian-ae-24.3.205.0.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:178) ~[elasticsearch-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969) ~[elasticsearch-rest-high-level-client-7.17.19.jar:7.17.19]
	at com.appiancorp.common.persistence.search.RestClientManagerUtil.getRestClientManagerNoSpring(RestClientManagerUtil.java:52) ~[appian-ae-24.3.205.0.jar:?]
	... 7 more
	Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://localhost:9200], URI [/], status line [HTTP/1.1 403 Forbidden]
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"}],"type":"security_exception","reason":"action [cluster:monitor/main] is unauthorized for user [health_check_user] with effective roles [] (assigned roles [health_check_role] were not found), this action is granted by the cluster privileges [monitor,manage,all]"},"status":403}

Cause

Action

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:08:23 GMT

Revision 3 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:08:23 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Interspersed between the above statements will be the following traces referencing the Search Server.

Cause

Action

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 22:06:57 GMT

Revision 2 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 10:06:57 PM

Symptoms

After upgrading to Appian 24.3 or beyond, the application server is unable to start while continuously citing the following within tomcat-stdOut.log.

[wait-for-component] INFO  com.appiancorp.common.startup.WaitForStatefulComponents - Waiting for Appian component Search Server to be healthy...

Cause

Action

Affected Versions

This article applies to Appian versions 24.3 and later.

Last Reviewed: September 2024

KB-XXXX Application server fails to start due to improperly configured search server credentials

Ryan Good — Tue, 03 Sep 2024 21:57:54 GMT

Revision 1 posted to Appian Knowledge Base by Ryan Good on 9/3/2024 9:57:54 PM

test