Connection Pool Issues

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 15:49:30 GMT

You add the delay timer prior to the Sync call (or if you're just talking about the syncing that happens automatically after a Write, then before the Write node) in each process instance. That means instead of 500 instances all hitting write and sync within a few milliseconds of each other, they'll execute in a staggered fashion across the entire set, allowing time for the connections to open and close between the different sync calls.

RE: Connection Pool Issues

piyusha6151 — Mon, 03 Jul 2023 15:39:38 GMT

But how timer will help if process are running parallely as sync will anyways run with each write operation.

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 15:21:21 GMT

I'd suggest you could still potentially batch the processing into larger batches than 1 row each - maybe 10 rows per process instance? This would reduce the number of sync calls considerably, and probably not increase your processing time very much overall.

An additional thing to consider would be to implement a timer at the end of each process that pauses it a few seconds. I've set up something before where each instance gets a random number of seconds between 1 and 30, and pauses that long, which gives the back-end engines enough time to process the different instances without accidentally trying to process every single one in one big lump, and thus getting hung up on itself.

RE: Connection Pool Issues

piyusha6151 — Mon, 03 Jul 2023 15:13:35 GMT

each row of excel needs to be parsed and based on value it has to update few other attributes. And we process it sequentially then it is taking huge time and by doing load balancing and parallel processing, we are able to reduce the end to end process time by 10 times.

But as data is being written for the Entity having sync record defined.

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 15:03:24 GMT

[quote userid="57059" url="~/discussions/f/general/29195/connection-pool-issues/115474#115474"]Unfortunately,do not have any away to include those rows to be inserted at a time[/quote]

Can you explain a bit more about the structure of this setup that requires each row to get its own... process instance, i assume? If that's correct, I don't understand why the process can be passed multiple rows (or even all rows), versus requiring each row to be processed in its own instance.

RE: Connection Pool Issues

Gopu Hema — Mon, 03 Jul 2023 15:01:20 GMT

It is an upload process where each row in an excel is getting processed. Unfortunately,do not have any away to include those rows to be inserted at a time because it is taking too much time to process all the rows at a time

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 15:00:11 GMT

[quote userid="57059" url="~/discussions/f/general/29195/connection-pool-issues/115472#115472"]each row in an excel is prcessing separtely in each instance[/quote]

Is there a particular reason why each row requires a separate instance to be launched for it?

RE: Connection Pool Issues

Gopu Hema — Mon, 03 Jul 2023 14:57:30 GMT

It is an upload process where each row in an excel is prcessing separtely in each instance. So when an instance is writing a row into db, sync happening

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 14:57:13 GMT

Yes, that's what you said originally. And I'm asking if there's any way to restructure this, so that all 500 of those writes could be done in a single instance somehow. Because it sounds as if you're referring to 500 instances that are launched and write 100% simultaneously, which leads me to conclude that it might be possible, instead of launching 500 instances, to launch 1 instance with 500 rows (or 5 instances with 100 rows each, etc...).

RE: Connection Pool Issues

Gopu Hema — Mon, 03 Jul 2023 14:54:37 GMT

500 instances are running parallely where each instnace is writing a row in db.

RE: Connection Pool Issues

Mathieu Drouin — Mon, 03 Jul 2023 14:51:11 GMT

Any reason why you need to do 500 writes vs just 1 write (and 1 sync with multiple ids)?

RE: Connection Pool Issues

Mike Schmitt — Mon, 03 Jul 2023 14:50:39 GMT

Could the 500 rows (or any subset of them) be written in one instance, versus looped?