Hi all,
I have a process that I want to run, for the amounts of data/records on an array, BUT, I want to control the number of instances it runs each time.
For example, I have 20k records but I want the process to execute 30 each time.
Discussion posts and replies are publicly visible
You could create a super-process that passes the next 30 from the list into a sub-process, which is synchronous and arranged in a loop. The sub-process it's called runs a subprocess MNI, once for each item in the 30. Configure it so that all instances start at the same time, but the node doesn't move on until all instances are complete. In the super-process, if there are more, collect the next 30, and pass as the parameter in the subprocess again. You need to nest the subprocesses 2 levels deep so you don't hit the 1000 limit (Instead you have a theoretical 1,000,000 limit).
That should get you continuously processing no more than 30 at a time until they're all done, (Though you may run into difficulty with the 1000 MNI limit should the numbers significantly change. You could do 50 in parallel per sub-loop and not find many problems 'til you hit 50,000)
I highly suggest some investigation to see if you can achieve the same result in an expression using a!forEach, no process modeler at all. If it is possible, it's also possibly several orders of magnitude faster than this approach.