KB-2163 High heap usage of search server component

Nicholas Van Dyke — Fri, 06 Nov 2020 01:33:47 GMT

Current Revision posted to Appian Knowledge Base by Nicholas Van Dyke on 11/6/2020 1:33:47 AM

Symptoms

On an Appian 20.2 or 20.3 environment, functionality depending on the search server is negatively impacted. Symptoms are varied but include degraded performance, inability to upload documents, and/or issues deploying packages.

To validate whether the symptoms are a result of this issue, navigate to the search-server.log file in <APPIAN_HOME>/logs/search-server and confirm whether output similar to the following is observed:

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Note: The values (e.g. [1036786612/988.7mb]) from the log output above are simply an example and may differ from the output observed in the logs for your environment. However, the values will likely be close to 1 GB (>900 MB) if using the default configurations.

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

If you are actively experiencing this issue and unable to take an outage to apply the hotfix at this time, the search server component may be restarted as a workaround. There is no need to shutdown the application server or take an outage to apply this workaround. On highly available environments that have the search server running on three nodes, the search server can be restarted in a rolling fashion. Start with the node that has experienced the problem. Then, restart each node one by one after ensuring that the cluster status is green. The cluster status can be monitored via the search_server_cluster.csv log in <APPIAN_HOME>/logs/data-metrics.

If this issue is being encountered frequently despite the search server being restarted, increase the search server heap allocation from the default 1 GB. Instructions to increase the search server heap can be found here. Note that increasing the search server heap allocation can result in higher overall system memory usage. Ensure there is enough system memory available before increasing heap. If you are unsure how much to increase the heap usage by, a safe option is increasing it to 2048 MB and monitoring the situation.

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

KB-2163 High heap usage of search server component

Nicholas Van Dyke — Fri, 06 Nov 2020 01:30:02 GMT

Revision 9 posted to Appian Knowledge Base by Nicholas Van Dyke on 11/6/2020 1:30:02 AM

Symptoms

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Note: The values (e.g. [1036786612/988.7mb]) from the log output above are simply an example and may differ from the output observed in the logs for your environment. The values will likely be close to 1 GB (>900 MB) or whatever the ssMaxHeapSize site property is set to.

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Nicholas Van Dyke — Fri, 06 Nov 2020 01:28:05 GMT

Revision 8 posted to Appian Knowledge Base by Nicholas Van Dyke on 11/6/2020 1:28:05 AM

Symptoms

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Nicholas Van Dyke — Fri, 06 Nov 2020 01:26:25 GMT

Revision 7 posted to Appian Knowledge Base by Nicholas Van Dyke on 11/6/2020 1:26:25 AM

Symptoms

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Parmida Borhani — Thu, 05 Nov 2020 03:20:50 GMT

Revision 6 posted to Appian Knowledge Base by Parmida Borhani on 11/5/2020 3:20:50 AM

Symptoms

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Note: The values (e.g.[1036786612/988.7mb]) from the log output above are simply an example and may differ from the output observed in the logs for your environment.

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Nicholas Van Dyke — Tue, 03 Nov 2020 14:02:23 GMT

Revision 5 posted to Appian Knowledge Base by Nicholas Van Dyke on 11/3/2020 2:02:23 PM

Symptoms

On an Appian 20.2 or 20.3 environment, functionality depending on the search server is negatively impacted. Symptoms are varied but include things like degraded performance, inability to upload documents, and/or deploying packages.

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%<node_name>] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server>:<port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Parmida Borhani — Tue, 03 Nov 2020 03:53:28 GMT

Revision 4 posted to Appian Knowledge Base by Parmida Borhani on 11/3/2020 3:53:28 AM

Symptoms

On an Appian 20.2 or 20.3 environment, various functionality depending on the search server is negatively impacted. Symptoms are varied but include things like degraded performance, inability to upload documents, and/or deploy packages.

To validate if the symptoms are a result of this issue, navigate to the search-server.log file in <APPIAN_HOME>/logs/search-server and confirm if output similar to the following is observed:

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%node_name] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server:port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

If you are actively experiencing this issue and unable to take an outage to apply the hotfix at this time, the search server component may be restarted as a workaround. There is no need to shutdown the application server or take an outage to apply this workaround. On highly available environments that have the search server running on three nodes, the search server can be restarted in a rolling fashion. Start with the node that has experienced the problem. Then restart each node one by one after ensuring that the cluster status is green. The cluster status can be monitored via the search_server_cluster.csv log in <APPIAN_HOME>/logs/data-metrics.

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Parmida Borhani — Mon, 02 Nov 2020 06:37:05 GMT

Revision 3 posted to Appian Knowledge Base by Parmida Borhani on 11/2/2020 6:37:05 AM

Symptoms

To validate if the symptoms are a result of this issue, navigate to the search-server.log file in <APPIAN_HOME>/logs/search-server and confirm if output similar to the following is observed:

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%node_name] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server:port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

If you are actively experiencing this issue and unable to take an outage to apply the hotfix at this time, a workaround to increase the search server heap allocation is available. There is no need to shutdown the application server or take an outage to apply this workaround.

Instructions to increase the search server heap can be found here. Note that increasing the search server heap allocation can result in higher overall system memory usage. Ensure there is enough system memory available before increasing heap usage. If you are unsure how much to increase the heap usage by, a safe option is increasing it to 2048 MB and monitoring the situation. If required, more heap usage can be allocated depending on available system resources until the hotfix is applied.

If the heap allocation cannot be increased due to resource constraints, the search server can also be stopped and started to reset the heap memory. On highly available environments that have the search server running on three nodes, the search server can be restarted in a rolling fashion. Start with the node that has experienced the problem. Then restart each node one by one after ensuring that the cluster status is green. The cluster status can be monitored via the search_server_cluster.csv log in <APPIAN_HOME>/logs/data-metrics.

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Parmida Borhani — Mon, 02 Nov 2020 06:28:42 GMT

Revision 2 posted to Appian Knowledge Base by Parmida Borhani on 11/2/2020 6:28:42 AM

Symptoms

To validate if the symptoms are a result of this issue, navigate to the search-server.log file in <APPIAN_HOME>/logs/search-server and confirm if output similar to the following is observed:

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%node_name] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server:port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Instructions to increase the search server heap can be found here. Note that increasing the search server heap allocation can result in higher overall system memory usage. Ensure there is enough system memory available before increasing heap usage. If you are unsure how much to increase the heap usage by, a safe bet is to increase it to 2048 MB and monitor the situation. If required, more heap usage can be allocated depending on system resources until the hotfix is applied.

If the heap allocation cannot be increased, the search server can also be stopped and started to reset the heap memory. On highly available environments that have the search server on three nodes, the search server can be restarted in a rolling fashion. Start with the node that has experienced the problem. Then restart each node one by one after ensuring that the cluster status is green. The cluster status can be monitored via the search_server_cluster.csv log.

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure

[DRAFT] - KB-XXXX High heap usage of search server component

Parmida Borhani — Mon, 02 Nov 2020 06:09:57 GMT

Revision 1 posted to Appian Knowledge Base by Parmida Borhani on 11/2/2020 6:09:57 AM

Symptoms

To validate if the symptoms are a result of this issue, navigate to the search-server.log file in . There should be a message similar to the following:

[DEBUG][org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction] [%node_name] failed to execute on node [<ID>]
org.elasticsearch.transport.RemoteTransportException: [Node <server:port>][<IP>:<port>][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [cluster:monitor/nodes/stats[n]] would be [1036786612/988.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [1036785320/988.7mb], new bytes reserved: [1292/1.2kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1292/1.2kb, accounting=1887742/1.8mb]

The search_server.csv log shows the Used Heap Percentage nearing 100%.

Cause

This issue has been hotfixed via AN-170688 in Appian 20.3.205.0 and 20.2.235.0.

Action

Apply the latest hotfix to your Appian installation.

Workaround

Instructions to increase the search server heap can be found here. Note that increasing the search server heap allocation can result in higher overall system memory usage. Ensure there is enough system memory available before increasing heap usage. If you are unsure how much to increase the heap usage by, a safe bet is to increase it to 2048 MB and monitor the situation. If required, more heap usage can be allocated depending on system resources until the hotfix is applied.

Affected Versions

This article applies to Appian 20.2 and 20.3.

Last Reviewed: November 2020

Tags: known issues, heap, fixed issues, search server, infrastructure