Connection Pool Issues

Question

Hi Everyone, 
 
 We have a requirement where Application is writing 500 rows parallely via startprocess smart service(Loop). 
 
 Now for every Write opertion sync will run that means sync will run 500 times in backend. It seems to be creating connection pool issue though connection pool is 200. 
 
 can some one suggest the best soultion in this case? 
 
 Thanks in Advance!!

Mike Schmitt · Answer

Yes, that's what you said originally. And I'm asking if there's any way to restructure this, so that all 500 of those writes could be done in a single instance somehow. Because it sounds as if you're referring to 500 instances that are launched and write 100% simultaneously, which leads me to conclude that it might be possible, instead of launching 500 instances, to launch 1 instance with 500 rows (or 5 instances with 100 rows each, etc...).

Mike Schmitt · Answer

Gopu Hema said: each row in an excel is prcessing separtely in each instance 
 Is there a particular reason why each row requires a separate instance to be launched for it?

Mike Schmitt · Answer

Gopu Hema said: Unfortunately,do not have any away to include those rows to be inserted at a time 
 Can you explain a bit more about the structure of this setup that requires each row to get its own... process instance, i assume? If that's correct, I don't understand why the process can be passed multiple rows (or even all rows), versus requiring each row to be processed in its own instance.

Mike Schmitt · Answer

I'd suggest you could still potentially batch the processing into larger batches than 1 row each - maybe 10 rows per process instance? This would reduce the number of sync calls considerably, and probably not increase your processing time very much overall. 
 An additional thing to consider would be to implement a timer at the end of each process that pauses it a few seconds. I've set up something before where each instance gets a random number of seconds between 1 and 30, and pauses that long, which gives the back-end engines enough time to process the different instances without accidentally trying to process every single one in one big lump, and thus getting hung up on itself.

Mike Schmitt · Answer

You add the delay timer prior to the Sync call (or if you're just talking about the syncing that happens automatically after a Write, then before the Write node) in each process instance. That means instead of 500 instances all hitting write and sync within a few milliseconds of each other, they'll execute in a staggered fashion across the entire set, allowing time for the connections to open and close between the different sync calls.