KB-2041 Application server unresponsive following startup in a high availability and/or distributed environment with multiple application servers

Rebecca Jonas — Thu, 18 Jun 2020 20:10:57 GMT

Current Revision posted to Appian Knowledge Base by Rebecca Jonas on 6/18/2020 8:10:57 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

Following a restart of the search servers and application servers in a high availability (HA) and/or distributed environment with multiple application servers, some subset of users experience an HTTP Status 404 - Not Found error when attempting to access the environment.

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

For the same application server node where the above error message is printed, the following message is printed repeatedly in the search-server.log located in the <APPIAN_HOME>/logs/search-server directory:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

The Appian web application fails to successfully deploy into Tomcat when the search server that is hosted on the same server is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a primary node with only a single search server running. This issue has been addressed via AN-146092 in the following hotfixes/versions:

Action

Upgrade to the latest version of Appian.

Workaround

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search-server directory. A healthy output is indicated by the cluster status turning GREEN:

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: June 2020

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

KB-2041 Application server unresponsive following startup in a high availability and/or distributed environment with multiple application servers

Rebecca Jonas — Thu, 18 Jun 2020 20:09:05 GMT

Revision 34 posted to Appian Knowledge Base by Rebecca Jonas on 6/18/2020 8:09:05 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Upgrade to the latest version of Appian.

Workaround

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

KB-2041 Application server unresponsive following startup in a high availability and/or distributed environment with multiple application servers

Parmida Borhani — Wed, 18 Dec 2019 22:45:58 GMT

Revision 33 posted to Appian Knowledge Base by Parmida Borhani on 12/18/2019 10:45:58 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

The Appian web application fails to successfully deploy into Tomcat when the search server that is hosted on the same server is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a master node with only a single search server running. This issue has been addressed via AN-146092 in the following hotfixes/versions:

Action

Upgrade to the latest version of Appian.

Workaround

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Wed, 18 Dec 2019 13:11:06 GMT

Revision 32 posted to Appian Knowledge Base by James Lee on 12/18/2019 1:11:06 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

The Appian web application fails to successfully deploy into Tomcat when the search server that is hosted on the same server is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a master node with only a single search server running. This issue has been addressed via AN-146092 in the following hotfixes/versions:

Action

Upgrade to the latest version of Appian.

Workaround

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Fri, 13 Dec 2019 15:56:49 GMT

Revision 31 posted to Appian Knowledge Base by James Lee on 12/13/2019 3:56:49 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

The Appian web application fails to successfully deploy into Tomcat when the search server that is hosted on the same server is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a master node with only a single search server running. This issue has been addressed via AN-136158 in the following hotfixes/versions:

Action

Upgrade to the latest version of Appian.

Workaround

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

Parmida Borhani — Thu, 12 Dec 2019 04:58:46 GMT

Revision 30 posted to Appian Knowledge Base by Parmida Borhani on 12/12/2019 4:58:46 AM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

The application server fails to start up successfully when the search server that is hosted on the same server is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a master node with only a single search server running. This issue has been addressed via AN-136158 in Appian 19.4 GA.

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

Parmida Borhani — Thu, 12 Dec 2019 04:57:37 GMT

Revision 29 posted to Appian Knowledge Base by Parmida Borhani on 12/12/2019 4:57:37 AM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

Parmida Borhani — Thu, 12 Dec 2019 04:56:34 GMT

Revision 28 posted to Appian Knowledge Base by Parmida Borhani on 12/12/2019 4:56:34 AM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

The application server logs also show that the application server has started. No application server logging will occur following the startup:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output is indicated by the cluster status turning GREEN::

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Wed, 11 Dec 2019 14:34:43 GMT

Revision 27 posted to Appian Knowledge Base by James Lee on 12/11/2019 2:34:43 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output is indicated by the cluster status turning GREEN::

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Wed, 11 Dec 2019 14:33:12 GMT

Revision 26 posted to Appian Knowledge Base by James Lee on 12/11/2019 2:33:12 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output is indicated by the cluster status turning GREEN::

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, fixed issues, search server, distributed, infrastructure

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Wed, 11 Dec 2019 14:29:14 GMT

Revision 25 posted to Appian Knowledge Base by James Lee on 12/11/2019 2:29:14 PM

This issue has been resolved in an Appian hotfix/new Appian version. Please apply the latest hotfix to your Appian installation or upgrade to the latest version of Appian.

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

If a hotfix is required for Appian 19.2 or 19.3, open a case with Appian Support.

Action

Upgrade to the latest version of Appian.

Workaround

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output is indicated by the cluster status turning GREEN::

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Wed, 11 Dec 2019 14:13:22 GMT

Revision 24 posted to Appian Knowledge Base by James Lee on 12/11/2019 2:13:22 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
...
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Cloud

Open a case with Appian Support to report the behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. Confirm the status of the search server cluster in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output is indicated by the cluster status turning GREEN::

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Tue, 10 Dec 2019 21:58:18 GMT

Revision 23 posted to Appian Knowledge Base by James Lee on 12/10/2019 9:58:18 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

For the same application server node where the above error message is printed, the below message is printed repeatedly in the search-server.log located in the <APPIAN_HOME>/logs/search server directory:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Cloud

Open a case with Appian Support and note to the case description that you are experiencing behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. You can confirm the status of the search server cluster in in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output will be indicated by the below line from this log:

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Tue, 10 Dec 2019 21:58:02 GMT

Revision 22 posted to Appian Knowledge Base by James Lee on 12/10/2019 9:58:02 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Cloud

Open a case with Appian Support and note to the case description that you are experiencing behavior in line with this article.

On-Premise

The search server cluster will automatically stabilize once all search servers have joined the cluster. Once all search servers have joined the cluster, restarting the unresponsive application server will resolve this issue. You can confirm the status of the search server cluster in in the search-server.log located in the <APPIAN_HOME>/logs/search server directory. A healthy output will be indicated by the below line from this log:

[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [Node <node_subdomain>:XXXX] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [RED] to [GREEN]. Current cluster information: [status=GREEN], [timed out=false], [nodes=X], [data nodes=X], [active primary shards=X], [active shards=X], [relocating shards=X], [initializing shards=X], [unassigned shards=X]

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in a high availability and/or distributed environment running multiple application servers

James Lee — Mon, 09 Dec 2019 22:40:36 GMT

Revision 21 posted to Appian Knowledge Base by James Lee on 12/9/2019 10:40:36 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException
... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 23:18:25 GMT

Revision 20 posted to Appian Knowledge Base by James Lee on 12/4/2019 11:18:25 PM

Symptoms

Following a restart of the search servers and application servers in an HA/Distributed environment with multiple application servers, some subset of users experience an HTTP Status 404 - Not Found error when attempting to access the environment:

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

This behavior results from the Appian web application failing to successfully deploy into Tomcat. This deployment failure occurs during application server startup when the search server for the same application server node is running, but not in a completely healthy state. The search server falls into this unhealthy state when it has been started at least 4 minutes prior to any other search server nodes in the cluster. The unhealthy state is caused by the search server cluster being unable to elect a master node with only a single search server running. This issue has been reported to the Appian Product Team. The reference number is AN-136158.

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 23:11:06 GMT

Revision 19 posted to Appian Knowledge Base by James Lee on 12/4/2019 11:11:06 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

For the same application server node where the above error message is printed, the below message is printed repeatedly in the search server.log located in the <APPIAN_HOME>/logs/search server directory:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 23:07:30 GMT

Revision 18 posted to Appian Knowledge Base by James Lee on 12/4/2019 11:07:30 PM

Symptoms

Following a restart of the search servers and application servers in an HA/Distributed environment with multiple application servers, some subset of users experience an HTTP Status 404 - Not Found error when attempting to access the environment:

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 23:06:38 GMT

Revision 17 posted to Appian Knowledge Base by James Lee on 12/4/2019 11:06:38 PM

Symptoms

In the tomcat-stdOut.log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 18:41:59 GMT

Revision 16 posted to Appian Knowledge Base by James Lee on 12/4/2019 6:41:59 PM

Symptoms

In the application server log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

For the same application server node where the above error message is printed, the below message is printed repeatedly in the search server.log located in the <APPIAN_HOME>/logs/search server directory:

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues

DRAFT KB-XXXX Application server unresponsive following startup in an HA/Distributed environment running multiple application servers

James Lee — Wed, 04 Dec 2019 17:28:44 GMT

Revision 15 posted to Appian Knowledge Base by James Lee on 12/4/2019 5:28:44 PM

Symptoms

In the application server log for at least one of the application servers, the below error will be displayed during the application server's startup:

YYYY-MM-DD HH:MM:SS SEVERE [localhost-startStop-X] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.appiancorp.common.startup.WaitForStatefulComponentsListener]
 java.lang.reflect.UndeclaredThrowableException

... 26 more
 Caused by: MasterNotDiscoveredException[null]

No application server logging will occur following the startup.

[WARN ][org.elasticsearch.discovery.zen.ZenDiscovery] [unknown] not enough master nodes discovered during pinging...

Followed by the below messages:

[DEBUG][org.elasticsearch.action.admin.cluster.health.TransportClusterHealthAction] [unknown] no known master node, scheduling a retry...
[INFO ][com.appian.es.logging.HealthStatusLoggingListener] [unknown] Cluster health changed for [cluster name=appian-search-cluster]. Status changed from [none] to [RED]. Current cluster information: [status=RED]...

Cause

Action

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: December 2019

Tags: high availability, known issues, application server, search server, distributed, infrastructure, open issues