KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

The Appian engines as a whole can obtain up to 30 threads at the same time from the application server to request it processes data on its behalf. Once the application server finishes the work, the response is sent back to the engines and the workpoller thread is released.

This issue occurs when the environment maxes out on these 30 threads for the engines, which suggests that the engines were sending long running work items to the application server in parallel.

Action

To resolve the issue, perform the following (the following steps are tailored for JBoss but the same principles apply to WebLogic):

  1. Identify the top threads by CPU.
    1. In Linux, run top, press the c key, then press Shift+H. This will show the threads sorted by highest to lowest CPU usage. Take a screenshot of the output.

    2. In Windows, use Process Explorer or Process Monitor to show the threads for the Java process and take a screenshot of the top threads by CPU.

  2. Take multiple thread dumps during the time of the issue
    1. Determine the process id of the JBoss process:
      ps -ef | grep standalone
    2. Take the thread dump:
      jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
    3. Repeat Step 2 every minute for 5 minutes.
  3. Restart the application server
  4. Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

  • Looping functions
  • Database server crash or restart
  • Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: February 2017

Related
Recommended