a!Start process-more than 1000 instances

Hi aravinds0003, Appian doesn't allow more than 1000 nodes instances, even if we are using Start Process Smart Service with the MNI configurations. Please find the below screenshots as a PoC.

Parent process and configs:

Started process for debugging:

However, if you want to invoke a sub-process more than 1000 times, I would suggest creating another parent process and calling your original parent process with similar MNI configs.

Hope it answers your question.

Additionally, it is often that we can find a solution that does not involve starting thousands and thousands of processes.  Would be interested to hear the use case here and what type of processing needs to occur. 

As Chris said, the question you need to ask yourself is not "How can we make over 1000 processes?"  The question you need to ask yourself is rather, "How can I do what I want without creating over 1000 processes?"  Why?  Too many processes are bad.  That's why Appian limits to 1000.  That's also why, if I remember rightly, Appian won't allow you to run an a!startProcessLink with an a!forEach to let you spawn a million of them.

Using Start Process Smart Service lets the Appian decide to start the process on a different engine depending on the memory availability. Having multiple process instances is not bad, it completely depends on the use case. However, it is always advisable to optimize the solution.

P.S. a!startProcessLink creates a link to start a process, does not start it by itself.

Another solution which I can think of is, using a loop instead of MNI. You can create a loop and increment the count based on the number of instances you want to create. To avoid the 1000 node instances error, select the "Delete previously completed/cancelled instances" in the "Other" tab of the nodes (Script Task and Start Process Smart Service). Refer to the below screenshots.

Deleting the instances will free up the memory thus, won't affect even if you create 1000+ instances. However, I advise you to revisit your design and try to process in batches, if possible.

If you really insist on it, what you need to do is start a certain number of processes (20, 1000) and have each of those take a batch of your data, and each load their own 1000 processes with startProcess.  You can probably have all the spawners run asynchronously, and each of them will have 1000 subprocesses.  That can get you up to a million.  I've done that to try to quickly create 10 million folders on a Docker container before.  That said, I was doing it on a container where I wasn't worried about breaking anything.  If it got destroyed, I could just spin up a new one.  Let me tell you, it got destroyed.  The million processes didn't end before I ran out of system resources and my server died.  I was doing them sequentially; 1000 simultaneously making 1000 each one at a time.  Never more than 1000 running at a time, and my server still died.

Like others have said, we need to understand more about your root use case in order to be able to answer this question. FWIW, this is a great example of an XY problem - you're asking about how to solve problem Y (running more than 1000 instances), but actually this is only one possible solution for your base problem X. By understanding your base problem, we can provide alternate solutions that hopefully wouldn't run up against product limitations.

I strongly advise against this - it doesn't really alleviate the memory problems because there is still some memory usage by instances that are deleted. Plus, if you're deleting the instances it's impossible to troubleshoot what is going on. In general a loop in your process is worse than using MNI because you're executing triple the nodes (because now you've added a gateway and script task), you're updating a variable 1000 times (which also has an impact on memory), and of course you can still run into limits.

I agree with the other points but for the troubleshooting, using the Start Process Smart Service will still have the instances of the child process (accessible from Monitoring) even after "Delete previously completed/cancelled instances" is checked for the node. It just deletes the node instances. Am I missing something?

Hi All, 

Thanks for the quick reply on this case , 

Use case as you guys are interested, 

An external system will trigger the appian web-api, which inturn triggers the  process. The process will do some ETL operation and process need to update another external system with enriched data from process

When the process is triggered the request payload will have array of values , and each value should undergo an ETL operation, and the need to update the another system(B) (note: the integration from the process and accept one request per value, means :if the array 100, process need to make 100 integration call )