KB-1088 Execution engine load metric warnings in logs and engine status output

Symptoms

17.3 and later

When running status.sh|bat with the -c option, the following output is seen:

Engine [engine name] on [host] has a load metric of [load metric] and the configured limit is [load metric]

17.2 and earlier

checkengine.bat|sh shows one of the following warnings/errors for the execution engines:

Nearing load metric limit: XX.XXX%

Load metric limit exceeded: XXX.XXX%
The following error may also be observed in the application server logs:

Caused by: com.appiancorp.suiteapi.common.exceptions.StorageLimitException: All execution engines are at maximum load. Cannot start process/simulation. Archive existing processes to free up space.

Cause

The load metric on the execution engines has exceeded 75% or 100%, depending on the message displayed. New processes cannot be started on an execution engine where the calculated load metric limit is 100% or higher.

This issue is most likely due to processes currently in the system. Run sizing to help determine any large processes on that particular engine.

Another possible cause is the property server.conf.processcommon.MAX_EXEC_ENGINE_LOAD_METRIC was updated but has not yet been propagated through the environment.

Action

As per the MAX_EXEC_ENGINE_LOAD_METRIC documentation, there are two available options:

  1. Archive recent unused process instances in the environment.

  2. Increase the server.conf.processcommon.MAX_EXEC_ENGINE_LOAD_METRIC value in custom.properties by an amount that will clear the warning (if the current load metric percentage is at 130% and the value of the MAX_EXEC_ENGINE_LOAD_METRIC setting is at 250, then doubling that setting to 500 should bring the load metric down to 65% and correct the issue.

Note 1: When increasing the server.conf.processcommon.MAX_EXEC_ENGINE_LOAD_METRIC value:

  • For versions of Appian 7.10 and earlier, restart the engines for this change to take effect.

  • For versions of Appian 7.11 and later, this property is hot-reloadable, so a restart is not required for this property to be updated. The application server propagates this property though the environment. If the application server is not started yet, start the application server. If the application server is already started, check the application server log for a message indicating that the change has been detected:

    INFO com.appiancorp.common.config.LoggingFileChangedReloadingStrategy - Changes detected in custom.properties, reloading now. Not all properties take effect without a restart. See documentation for details.

Note 2: On production systems, the maximum metric value should be raised from its default value of 40 to a value of 120 or higher. It can be further raised (above 120) according to the amount of RAM available to the execution engines. As a general rule of thumb, if you have installed more RAM than the minimum system requirement of 4GB you can raise the value for this metric by a number that equals the GB of additional RAM on your system multiplied by the number 60. The maximum setting for this metric on a 64 bit machine is 2145.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: November 2019

Related
Recommended