Recently we have been noticing the following type of errors in the application-s

... (called by rule 'getsignaturedocid') at function 'document': Error evaluating function 'document' : com.appiancorp.kougar.driver.exceptions.SafeRetryException: Unable to acquire a Read connection. Safe to retry. ) (Data Outputs)
Recommended Action: Check the expression including any function requirements and resume.

This type of errors did not go away after we tried the following:
Cold rebooted the server - all services stopped and started smoothly w/o any issues. Turned off McAfee services. Confirmed Server resources are good. Our exec kdbs are around 4.5G.

Any information to help us troubleshoot and resolve this issue would be greatly appreciated.

Hi, Todd - have you tried checkengine.bat just to make absolutely sure all the engines are up and running?

Hi Stacey - yes we ran checkengine.bat quite often and found absolutely nothing wrong with any of the engines. I reviewed other related forum posts about "Unable to acquire a Read connection..." but could not locate any leads to the root cause.

I suggest you work with Appian Technical Support in investigating this issue further. Please let me know once you submit the support case with the application-server.log and the db_*.log's

Please post back what the answer is when you resolve it, Todd!

Eduardo, a support case opened - #413808

The Technical Support Team will work with you on this issue.

Todd!! Can you please share the resolution ...

Todd, did you have any luck solving the problem? How did you do so?

We have been experiencing similar issues with our production servers lately. I noticed that it seemed to happen most often when a data loading process runs and calls a subprocess a few hundreds times, with each subprocess running 1-2 query rules, updating the returned CDTs and writing the results back to the datastore. I am concerned that since this process runs as fast as Appian can create the hundreds of subprocesses (loops of 100 via MNI) and run the queries that we could be overloading either the engines or the database connections (though the DBAs don't think it is on their end).

My hoped for solution that I put in last Friday was to add in some random 10-30 second pauses between each 100 subprocess calls as well as brief random length pauses between some of the db queries. I am hoping this acts like throttling and slows the process down so that the engines & db aren't overloaded. It is too early to tell if it is helping however since it only happens 1-2 times a day (though 5-20 processes hang when it does).

As a side note, the normal timer nodes & script node with exception/timer event is limited to 1 minute+ increments when you select 'skip this node'. If you want to delay the execution of a node for fractional minutes you can use the interval() function and the 2nd radio button option: 'Skip this node at the date and time specified by this expression'.

Here are some code snippets, these have only had light testing and could be off. Also, the timer isn't precise and sometimes can be off a few seconds, which is fine for my needs:

random time delay of 1-10 seconds: =now()+intervalds(0,0,round(round(rand() ,2)*10,0))
ranom delay of 5-15 seconds: =now()+intervalds(0, 0, (round(round(rand(), 2)*10, 0)+5))
variable delay based on the count of the pv variable (5, 10 & 15 second pauses with a 20second one if the if statement is false):
=now()+intervalds(0, 0, if(pv!loopRowCount_Int<20,5,if(pv!loopRowCount_Int<100,10,if(pv!loopRowCount_Int<200,15,20))))

Ryanh, we had to open a support ticket to completely resolved the issue. We initially tried the following steps but they may help your env.
1) Remove the JBoss/standalone deployment data/temp folder
2) Add additional Exec engine
3) Redeploy suite.ear and appian-ds.xml