KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Parmida Borhani — Tue, 05 Jan 2021 04:24:24 GMT

Current Revision posted to Appian Knowledge Base by Parmida Borhani on 1/5/2021 4:24:24 AM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - x [<engine>] (originally: <thread name>)] ERROR com.appiancorp.process.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

The application server allocates up to 30 work poller threads to receive and process work from the Appian engines. This issue occurs when all 30 work poller threads are in use, preventing the engines from sending new work to the application server.

Action

To resolve the issue, perform the following:

Capture thread dumps while the behavior is ongoing:
1. Determine the process id of the application server process:
```
For JBoss: ps -ef | grep standalone
```
```
For Tomcat: ps -ef | grep bootstrap
```
2. Capture multiple thread dumps using the instructions in KB-1177.
Restart the application server.
Analyze the thread dumps to determine the cause of the long running Appian Work Item threads.

The usual culprits are:

Looping functions
Unresponsive or slow external servers (database, mail server, or other integration)
Long running calls from custom plugins

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: January 2021

Tags: java, engines, application server, application design, processes, work poller

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Parmida Borhani — Tue, 05 Jan 2021 04:22:24 GMT

Revision 6 posted to Appian Knowledge Base by Parmida Borhani on 1/5/2021 4:22:24 AM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - x [<engine>] (originally: <thread name>)] ERROR com.appiancorp.process.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

Action

To resolve the issue, perform the following:

Capture thread dumps while the behavior is ongoing:
1. Determine the process id of the application server process:
```
For JBoss: ps -ef | grep standalone
```
```
For Tomcat: ps -ef | grep bootstrap
```
2. Capture multiple thread dumps using the instructions in KB-1177.
Restart the application server.
Analyze the thread dumps to determine the cause of the long running Appian Work Item threads.

The usual culprits are:

Looping functions
Unresponsive or slow external servers (database, mail server, or other integration)
Long running calls from custom plugins

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: January 2021

Tags: engines, application server, application design, processes, work poller

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Parmida Borhani — Mon, 11 Nov 2019 03:58:37 GMT

Revision 5 posted to Appian Knowledge Base by Parmida Borhani on 11/11/2019 3:58:37 AM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

The application server allocates up to 30 workpoller threads to receive and process work from the Appian engines. This issue occurs when all 30 workpoller threads are in use, preventing the engines from sending new work to the application server.

Action

To resolve the issue, perform the following:

Identify the top threads by CPU.
1. In Linux, run top, press the c key, then press Shift+H. This will show the threads sorted by highest to lowest CPU usage. Take a screenshot of the output.
2. In Windows, use Process Explorer or Process Monitor to show the threads for the Java process and take a screenshot of the top threads by CPU.
Take multiple thread dumps during the time of the issue
1. Determine the process id of the application server process:
```
For JBoss: ps -ef | grep standalone
```
```
For Tomcat: ps -ef | grep bootstrap
```
2. Take the thread dump:
```
jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
```
3. Repeat Step 2 every minute for 5 minutes.
Restart the application server
Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

Looping functions
Unresponsive or slow external server (database, mail server, or other integration)
Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: November 2019

Tags: database, application server, application design

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Parmida Borhani — Mon, 11 Nov 2019 03:57:34 GMT

Revision 4 posted to Appian Knowledge Base by Parmida Borhani on 11/11/2019 3:57:34 AM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

The application server allocates up to 30 workpoller threads to receive and process work from the Appian engines. This issue occurs when all 30 workpoller threads are in use, preventing the engines from sending new work to the application server.

Action

To resolve the issue, perform the following:

Identify the top threads by CPU.
1. In Linux, run top, press the c key, then press Shift+H. This will show the threads sorted by highest to lowest CPU usage. Take a screenshot of the output.
2. In Windows, use Process Explorer or Process Monitor to show the threads for the Java process and take a screenshot of the top threads by CPU.
Take multiple thread dumps during the time of the issue
1. Determine the process id of the application server process:
```
For JBoss: ps -ef | grep standalone
```
```
For Tomcat: ps -ef | grep bootstrap
```
2. Take the thread dump:
```
jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
```
3. Repeat Step 2 every minute for 5 minutes.
Restart the application server
Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

Looping functions
Unresponsive or slow external server (database, mail server, or other integration)
Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: November 2019

Tags: database, application server, application design

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Parmida Borhani — Thu, 19 Apr 2018 22:45:25 GMT

Revision 3 posted to Appian Knowledge Base by Parmida Borhani on 4/19/2018 10:45:25 PM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

The Appian engines as a whole can obtain up to 30 threads at the same time from the application server to request it processes data on its behalf. Once the application server finishes the work, the response is sent back to the engines and the workpoller thread is released.

This issue occurs when the environment maxes out on these 30 threads for the engines, which suggests that the engines were sending long running work items to the application server in parallel.

Action

To resolve the issue, perform the following (the following steps are tailored for JBoss but the same principles apply to WebLogic):

Identify the top threads by CPU.
1. In Linux, run top, press the c key, then press Shift+H. This will show the threads sorted by highest to lowest CPU usage. Take a screenshot of the output.
2. In Windows, use Process Explorer or Process Monitor to show the threads for the Java process and take a screenshot of the top threads by CPU.
Take multiple thread dumps during the time of the issue
1. Determine the process id of the JBoss process:
```
ps -ef | grep standalone
```
2. Take the thread dump:
```
jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
```
3. Repeat Step 2 every minute for 5 minutes.
Restart the application server
Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

Looping functions
Database server crash or restart
Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: February 2017

Tags: database, application server, application design

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Nick Vigilante — Thu, 13 Apr 2017 13:28:14 GMT

Revision 2 posted to Appian Knowledge Base by Nick Vigilante on 4/13/2017 1:28:14 PM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

This issue occurs when the environment maxes out on these 30 threads for the engines, which suggests that the engines were sending long running work items to the application server in parallel.

Action

To resolve the issue, perform the following (the following steps are tailored for JBoss but the same principles apply to WebLogic):

Take multiple thread dumps during the time of the issue
1. Determine the process id of the JBoss process:
```
ps -ef | grep standalone
```
2. Take the thread dump:
```
jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
```
3. Repeat Step 2 every minute for 5 minutes.
Restart the application server
Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

Looping functions
Database server crash or restart
Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: February 2017

Tags: database, application server, application design

KB-1159 "Could not obtain 3 thread(s) after X attempts in work poller" error printed in application server log

Nick Vigilante — Tue, 28 Feb 2017 19:56:39 GMT

Revision 1 posted to Appian Knowledge Base by Nick Vigilante on 2/28/2017 7:56:39 PM

Symptoms

The following error is displayed in the application server log:

[Appian WorkPoller - 0 [Notifications] (originally: default-threads - 19)] ERROR com.appiancorp.ra.workpoller.WorkPoller - Could not obtain 3 thread(s) after X attempts in work poller

Cause

Each engine can get up to 30 threads at the same time from the application server to request it processes data on its behalf. Once the application server finishes the work, the response is sent back to the engines and the workpoller thread is released.

This issue occurs when the environment maxes out on these 30 threads for the engines, which suggests that the engines were sending long running work items to the application server in parallel.

Action

To resolve the issue, perform the following (the following steps are tailored for JBoss but the same principles apply to WebLogic):

Take multiple thread dumps during the time of the issue
1. Determine the process id of the JBoss process:
```
ps -ef | grep standalone
```
2. Take the thread dump:
```
jstack processid > /path/threaddump-YYYYMMDD-HHMM.txt
```
3. Repeat Step 2 every minute for 5 minutes.
Restart the application server
Analyze the thread dumps to determine the cause of the long running threads.

The usual culprits are:

Looping functions
Database server crash or restart
Custom plugin that contains a long running call

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: February 2017

Tags: database, application server, application design