Recently we have been noticing the following type of errors in the application-s

Todd, did you have any luck solving the problem? How did you do so?

We have been experiencing similar issues with our production servers lately. I noticed that it seemed to happen most often when a data loading process runs and calls a subprocess a few hundreds times, with each subprocess running 1-2 query rules, updating the returned CDTs and writing the results back to the datastore. I am concerned that since this process runs as fast as Appian can create the hundreds of subprocesses (loops of 100 via MNI) and run the queries that we could be overloading either the engines or the database connections (though the DBAs don't think it is on their end).

My hoped for solution that I put in last Friday was to add in some random 10-30 second pauses between each 100 subprocess calls as well as brief random length pauses between some of the db queries. I am hoping this acts like throttling and slows the process down so that the engines & db aren't overloaded. It is too early to tell if it is helping however since it only happens 1-2 times a day (though 5-20 processes hang when it does).

As a side note, the normal timer nodes & script node with exception/timer event is limited to 1 minute+ increments when you select 'skip this node'. If you want to delay the execution of a node for fractional minutes you can use the interval() function and the 2nd radio button option: 'Skip this node at the date and time specified by this expression'.

Here are some code snippets, these have only had light testing and could be off. Also, the timer isn't precise and sometimes can be off a few seconds, which is fine for my needs:

random time delay of 1-10 seconds: =now()+intervalds(0,0,round(round(rand() ,2)*10,0))
ranom delay of 5-15 seconds: =now()+intervalds(0, 0, (round(round(rand(), 2)*10, 0)+5))
variable delay based on the count of the pv variable (5, 10 & 15 second pauses with a 20second one if the if statement is false):
=now()+intervalds(0, 0, if(pv!loopRowCount_Int<20,5,if(pv!loopRowCount_Int<100,10,if(pv!loopRowCount_Int<200,15,20))))

Todd, did you have any luck solving the problem? How did you do so?

We have been experiencing similar issues with our production servers lately. I noticed that it seemed to happen most often when a data loading process runs and calls a subprocess a few hundreds times, with each subprocess running 1-2 query rules, updating the returned CDTs and writing the results back to the datastore. I am concerned that since this process runs as fast as Appian can create the hundreds of subprocesses (loops of 100 via MNI) and run the queries that we could be overloading either the engines or the database connections (though the DBAs don't think it is on their end).

My hoped for solution that I put in last Friday was to add in some random 10-30 second pauses between each 100 subprocess calls as well as brief random length pauses between some of the db queries. I am hoping this acts like throttling and slows the process down so that the engines & db aren't overloaded. It is too early to tell if it is helping however since it only happens 1-2 times a day (though 5-20 processes hang when it does).

As a side note, the normal timer nodes & script node with exception/timer event is limited to 1 minute+ increments when you select 'skip this node'. If you want to delay the execution of a node for fractional minutes you can use the interval() function and the 2nd radio button option: 'Skip this node at the date and time specified by this expression'.

Here are some code snippets, these have only had light testing and could be off. Also, the timer isn't precise and sometimes can be off a few seconds, which is fine for my needs:

random time delay of 1-10 seconds: =now()+intervalds(0,0,round(round(rand() ,2)*10,0))
ranom delay of 5-15 seconds: =now()+intervalds(0, 0, (round(round(rand(), 2)*10, 0)+5))
variable delay based on the count of the pv variable (5, 10 & 15 second pauses with a 20second one if the if statement is false):
=now()+intervalds(0, 0, if(pv!loopRowCount_Int<20,5,if(pv!loopRowCount_Int<100,10,if(pv!loopRowCount_Int<200,15,20))))