I have this process model which loops continuously and checks if any new value is inserted. After 1000 instances i started getting above error in the node highlighted and the Process model gets paused. How can i control this issue and make the process model run continuously ?
Discussion posts and replies are publicly visible
@Siddharth,
The default limit on the number of times a node can execute is 1000.There are couple of options to overcome this.1. Select the option "delete previously completed/cancelled instances" in other tab.2. Create Sub process and send the data in batches(1000 per loop) to that sub process and put the spawning inside the sub process.
Did you try to use a scheduled start node instead of a looping process? Creating loops is critical in terms of best practices and should be avoided.
If i use scheduled start node, I can run only once in 5 mins, right ?
True. Depending on the required interval you might want to create short loops in process making sure that no single process instance runs longer than 5 mins.
There's a fun, handy dandy way to get faster time interval than that.
You have to use the intervalds() function and the now(function)
Basically, when you reach the timer node or node with a timer, you cause it to fire at a specified time, which is only 1 point in all of history that this node will run, and that time is now() + intervalds(0,0,30), which is 30 seconds after that expression is evaluated. (You can do 5 seconds or even 1 second, if you want to. My efforts to do faster than once a second so far have been stymied.)
Now it will run once, 30 seconds from now, and never again. So, when it runs, say it creates a subprocess that runs a batch of 1000. Then, when that's done, loop back to the timer which will calculate now() + intervalds(0,0,30) again. This will let you process up to a million records, a thousand at a time every 30 seconds.
You could also theoretically, instead of looping, recursively call the same process model as an asynchronous subprocess of itself.
The first one processes a record, then it hits that node, waits 30 seconds, then calls another copy of itself, then terminates and waits for archival.
The copy has already started processing the next record, then it hits that node and waits 30 seconds, then calls another copy of itself, then terminates and waits for archival....
You may want to consider setting archival time to 0 days depending on how many of these you want to do if you go the recursive route. You also need to make absolutely sure that there is a mechanism to get out of the recursion and process the LAST one and not call the subprocess again BEFORE you add the recursive call to itself. The benefit is that there's no theoretical limit to how many of these could happen. The problem is that there's no theoretical limit to how many of these could happen. Use recursion at your own risk. (For synchronous subprocesses, Appian thankfully adds a limit to how many process children can be in a process stack. For asynchronous, I haven't prototyped this myself. However, that may be a fun exercise for a docker instance.)
does deleting previous instances a good way?
Instead I'd suggest having a new parent process that calls the current process, and handles the looping and timer. Strip down the process you already have such that all it does is one instance of the work. In the parent, you could make a timer-based loop, and store a counter that counts up to a value less than 1000 (i'd suggest starting smaller than you need, like 10 or 100), and when the process reaches that counter, have it call a new copy of itself via the Start Process Smart Service, then exit. That way you'll never get close to the MNI / node execution limit of 1000 instances.
There's some good suggestions above on how to run this in a loop, but I'm also curious if this could be built in an entirely different way that would eliminate the need to create a loop at all. For example, is it possible to have whatever is updating the value in the database trigger an Appian process on any update? For instance, suppose that you have a web service that is inserting data into a database and that is what you are checking here. Instead, of running a loop, you could have the service also make a request to an Appian Web API to start the process at that point. Then it would be instantaneous and you wouldn't need to worry about creating a loop.
is there any function that i can use to count the active instance of the process model ?
Siddharth,
Actually the default node instance is 1000. If you want to run more than 1000 instance, select the option "Delete previously completed/cancelled instances." by doing this you can run n number of instances. But MNI is not a good practice.