KB-1256 How to address high CPU usage in self-managed Appian environments

Maggie Deppe-Walker — Fri, 06 Oct 2023 04:56:41 GMT

Current Revision posted to Appian Knowledge Base by Maggie Deppe-Walker on 10/6/2023 4:56:41 AM

This article details root causes as well as corrective actions to be taken in situations where an Appian environment is experiencing higher than normal CPU usage.

Managing high CPU usage

Identify one or more processes that show a high percentage of CPU as observed from the Task Manager (Windows) or the output of top (UNIX). Depending on what these processes are, it may be possible to investigate and resolve potential issues affecting the environment.

Application Server

The root cause investigation of the Application server’s high CPU usage is done in the following steps:

View the CPU usage per thread for the JVM process. On Linux, run top followed by Shift+H and on Windows, use Process Explorer. to see the individual threads in the JVM. Note the PID of the thread or threads consuming the highest CPU %
Convert the PIDs into Hex values using an online converter.
Generate a thread dump as described in KB-1177.
Search for the hex values obtained in Step 2 within the thread dump.

This allows you to find the stacktraces of those threads that are responsible for the most load. A review of the stacktrace(s) can help identify the relevant activities executed within Appian such as processes, reports etc that will need to be terminated to bring the CPU usage back to normal.

Search Server

The search server JVM’s high CPU usage can be investigated in the same way as the application server. Typical causes of search server’s high CPU usage include running large imports or impact analysis. In cases where there is significant performance degradation on mission-critical systems, a restart of environment can be done to alleviate this issue immediately. However it is recommended to consult with Appian Technical Support before considering this option.

Database

When a business database that Appian connects to is experiencing high load, this could indicate long running or resource intensive queries executing within the environment. Reviewing the active processes report and/or investigating the processes within the database can help pinpoint the exact query being executed through which the relevant activity within Appian can be identified and terminated if necessary.

Engines

Similar to KB-1248, identify the relevant engine corresponding to the OS process that is experiencing high load and follow the appropriate steps described in the sections below.

Process Execution Engines

High load on the execution engines directly relates to one or more processes launched within Appian. If the cause is not evident from the active processes on the system, then:

Using the perf_monitor_db_PX* logs, note the timestamp when the spike occurred or started
Filter the All Processes report showing processes that were initiated at the time

If any of the above processes are still active and are found to be resource intensive, they will either need to be terminated or allowed to complete in order to alleviate the high load.

If required, Appian Technical Support may be able to assist in further investigation by identifying long running nodes within the environment that are potentially responsible for such issues.

Process Analytics Engines

The load on Analytics is indicative of loading or exporting large reports within the environment at the time. Review the corresponding perf logs to identify if any processes are executing relevant smart services such as “Execute Process Report”.

In some cases, the Analytics engine logs might also have information on specific reports that are executed. For example:

[PA00XXXX] {paXXX.kdb XXX} (Default) ERROR .a.p.ANLYZE.i "xxxx" "16686" "id:2036" "Timeout: Timeout (Report Exceeded Maximum Time) (Report named 'My Report' (report document id 1556), maximum report time 30000ms) affected 0 rows"

Correlating this data with the Analytics perf logs can help identify the particular report(s) that is executed at that time.

Process Design Engine

As with the search server, the design engine is known to experience load mostly when running impact analysis or application imports where a large number of objects are updated.

A restart can help address performance degradation on production-like environments if necessary. That said, following the best practices for application design helps avoid such problems.

Personalization Engine

It is rare to observe high load on this engine, (adb ag) however this is typically seen during activities involving mass user/group creation or updates. Examples of such activities include the LDAP synchronization or excessive use of the User/Group smart services available in the base product or in shared components.

The recommendation is to review and improve the LDAP sync processes to work with a smaller set of users and also reducing the frequency of execution, if possible.

Recommendations to improve CPU usage

Configure relevant settings for sending/retrieving data: Query Limits, Web service timeouts, and Report optimizations.
Avoid excessive invocations of subprocesses or activity chaining that increases foreground load on the execution engines.
Follow application design best practices to minimize updates during application imports.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: self-managed, administration, how-to

KB-1256 How to address high CPU usage in self-managed Appian environments

Parmida Borhani — Tue, 29 Sep 2020 03:07:41 GMT

Revision 2 posted to Appian Knowledge Base by Parmida Borhani on 9/29/2020 3:07:41 AM

This article details root causes as well as corrective actions to be taken in situations where an Appian environment is experiencing higher than normal CPU usage.

Managing high CPU usage

Application Server

The root cause investigation of the Application server’s high CPU usage is done in the following steps:

View the CPU usage per thread for the JVM process. On Linux, run top followed by Shift+H and on Windows, use Process Explorer. to see the individual threads in the JVM. Note the PID of the thread or threads consuming the highest CPU %
Convert the PIDs into Hex values using an online converter.
Generate a thread dump as described in KB-1177.
Search for the hex values obtained in Step 2 within the thread dump.

Search Server

Database

Engines

Similar to KB-1248, identify the relevant engine corresponding to the OS process that is experiencing high load and follow the appropriate steps described in the sections below.

Process Execution Engines

High load on the execution engines directly relates to one or more processes launched within Appian. If the cause is not evident from the active processes on the system, then:

Using the perf_monitor_db_PX* logs, note the timestamp when the spike occurred or started
Filter the All Processes report showing processes that were initiated at the time

If any of the above processes are still active and are found to be resource intensive, they will either need to be terminated or allowed to complete in order to alleviate the high load.

If required, Appian Technical Support may be able to assist in further investigation by identifying long running nodes within the environment that are potentially responsible for such issues.

Process Analytics Engines

In some cases, the Analytics engine logs might also have information on specific reports that are executed. For example:

[PA00XXXX] {paXXX.kdb XXX} (Default) ERROR .a.p.ANLYZE.i "xxxx" "16686" "id:2036" "Timeout: Timeout (Report Exceeded Maximum Time) (Report named 'My Report' (report document id 1556), maximum report time 30000ms) affected 0 rows"

Correlating this data with the Analytics perf logs can help identify the particular report(s) that is executed at that time.

Process Design Engine

As with the search server, the design engine is known to experience load mostly when running impact analysis or application imports where a large number of objects are updated.

A restart can help address performance degradation on production-like environments if necessary. That said, following the best practices for application design helps avoid such problems.

Personalization Engine

The recommendation is to review and improve the LDAP sync processes to work with a smaller set of users and also reducing the frequency of execution, if possible.

Recommendations to improve CPU usage

Configure relevant settings for sending/retrieving data: Query Limits, Web service timeouts, and Report optimizations.
Avoid excessive invocations of subprocesses or activity chaining that increases foreground load on the execution engines.
Follow application design best practices to minimize updates during application imports.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: self-managed, administration, how-to

KB-1256 How to address high CPU usage in self-managed Appian environments

Nick Vigilante — Tue, 07 Mar 2017 14:52:43 GMT

Revision 1 posted to Appian Knowledge Base by Nick Vigilante on 3/7/2017 2:52:43 PM

This article details root causes as well as corrective actions to be taken in situations where an Appian environment is experiencing higher than normal CPU usage.

Managing high CPU usage

Application Server

The root cause investigation of the Application server’s high CPU usage is done in the following steps:

View the CPU usage per thread for the JVM process. On Linux, run top followed by Shift+H and on Windows, use Process Explorer. to see the individual threads in the JVM. Note the PID of the thread or threads consuming the highest CPU %
Convert the PIDs into Hex values using an online converter.
Generate a thread dump as described in KB-1177.
Search for the hex values obtained in Step 2 within the thread dump.

Search Server

Database

Engines

Similar to KB-1248, identify the relevant engine corresponding to the OS process that is experiencing high load and follow the appropriate steps described in the sections below.

Process Execution Engines

High load on the execution engines directly relates to one or more processes launched within Appian. If the cause is not evident from the active processes on the system, then:

Using the perf_monitor_db_PX* logs, note the timestamp when the spike occurred or started
Filter the All Processes report showing processes that were initiated at the time

If any of the above processes are still active and are found to be resource intensive, they will either need to be terminated or allowed to complete in order to alleviate the high load.

If required, Appian Technical Support may be able to assist in further investigation by identifying long running nodes within the environment that are potentially responsible for such issues.

Process Analytics Engines

In some cases, the Analytics engine logs might also have information on specific reports that are executed. For example:

[PA00XXXX] {paXXX.kdb XXX} (Default) ERROR .a.p.ANLYZE.i "xxxx" "16686" "id:2036" "Timeout: Timeout (Report Exceeded Maximum Time) (Report named 'My Report' (report document id 1556), maximum report time 30000ms) affected 0 rows"

Correlating this data with the Analytics perf logs can help identify the particular report(s) that is executed at that time.

Process Design Engine

As with the search server, the design engine is known to experience load mostly when running impact analysis or application imports where a large number of objects are updated.

A restart can help address performance degradation on production-like environments if necessary. That said, following the best practices for application design helps avoid such problems.

Personalization Engine

The recommendation is to review and improve the LDAP sync processes to work with a smaller set of users and also reducing the frequency of execution, if possible.

Recommendations to improve CPU usage

Configure relevant settings for sending/retrieving data: Query Limits, Web service timeouts, and Report optimizations.
Avoid excessive invocations of subprocesses or activity chaining that increases foreground load on the execution engines.
Follow application design best practices to minimize updates during application imports.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: administration, how-to