Hi All,
I have performance issue with batch processing of data,
Required to do: Need to update around 3L of records in DB.
First Try: Implemented a process where 200 records will be fetched at once and passed to subprocess which is configured for MNI. But the issue here is when we configure MNI it has a limit of 1000 instances per node. So after that it moves to pause state.
Second Try: Updated the same process without configuring MNI, but passing 200 records to the subprocess and looping it internally till 200 records are done and then back to parent process will again fetch 200 records and will pass them to subprocess. This continues till all the records in DB completes. i.e 3L records.
So here finally in the Second Try i was able to loop it successfully with out any pauses or breaks or fails after 1000 records. But the performance is too low and taking 20 hrs for 35k records to update. In this case if we have to process 3L records it will take a week or more.
Can anyone suggest on this.
Thanks
Vasu J
Discussion posts and replies are publicly visible
Having data of 3L inside Appian or doing manipulation affects a huge on the Appian Server. I suggest going with the stored procedures and that to if it is nightly triggered from Appian or even it is better for job trigger from the Database itself.