KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

Each engine can get up to 30 threads at the same time from the application server to request it processes data on its behalf. Once the application server finishes the work, the response is sent back to the engines and the workpoller thread is released.

This issue occurs when the environment maxes out on these 30 threads for the engines, which suggests that the engines were sending long running work items to the application server in parallel.

Action

To resolve the issue, perform the following (the following steps are tailored for JBoss but the same principles apply to WebLogic):

1. Take multiple thread dumps during the time of the issue
   1. Determine the process id of the JBoss process:

      ps -ef | grep standalone

   2. Take the thread dump:

      jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt

   3. Repeat Step 2 every minute for 5 minutes.
2. Restart the application server
3. Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

• Looping functions
• Database server crash or restart
• Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.