I am running some processing in parallel that includes api calls.The result with then populate an interface attended by a user.
I have tried both a!forEach and MNI but for both the performance is not as expected.
By looking at the performance tab in the expression rule, it is clear that the items in the array are running sequentially and not in parallel,
as the total time for the expression rule is the sum of every iteration. As a test I tried to put only a GET api call in the foreach loop and the result is the same.
This is not what I was expecting, is there anything that can be done to enforce the threads to run in parallel to optimize the performance?
Discussion posts and replies are publicly visible
If foreach and MNI both are not helping, you can creating process model using parallel paths for each api call - basically call each api call independent of each other using AND node and a separate path per API. After the calls you can use complex node to ensure process returns to UI only after all the API executions are complete.
The problem is that I do not know how many paths I need, the content of the array depends of a previous api call.
Can you share the code of your expression?
Pretty sure with the a!forEach() function, it is an expectation that iterations run sequentially. For MNI, you would only expect parallel processing when specifying the correct node options, i.e. "launch all at once". If you're trying to chain across this, though, I would use caution as at best your users will experience pretty severe lag, and at worst, you'll hit the chaining limit due to internal node counts and the user will be dumped back to the site they were working on. I can only guess at your particular implemention aims, however.
If the aim is to create a seamless user experience across 2 tasks (one of which falls after the API calls / etc), as always my best suggestion is to design an intermediary task with a "refresh" button which checks for the value of an arbitrary PV you set to an arbitrary "complete" value after the separate thread is complete. This way you can stage the thread in your process in such a way that best allows for performance and processing, rather than trying to squeeze it into such a configuration that allows user chaining across it. I've done this here and there and honestly, the users don't mind waiting around for a few extra moments, especially when they're given a button they can spam-click with no negative consequences.
I somehow stumbled on a solution.
The original code was:
a!forEach( items: ri!ofscSlots, expression: rule!FF_processOFSCSlotsBridge( ofscSlot: fv!item, activityType: ri!activityType, postcode: ri!postcode, duration: ri!duration, products: ri!products ) )
Bizarrely the (ugly) code below trigger the parallel evaluation
a!localVariables( local!numberSlots: length( ri!ofscSlots), { if(1<=local!numberSlots, rule!FF_processOFSCSlotsBridge( ofscSlot: ri!ofscSlots[1], activityType: ri!activityType, postcode: ri!postcode, duration: ri!duration, products: ri!products ), {} ), if(2<=local!numberSlots, rule!FF_processOFSCSlotsBridge( ofscSlot: ri!ofscSlots[2], activityType: ri!activityType, postcode: ri!postcode, duration: ri!duration, products: ri!products ), {} ), ---- And so on ------
The expression rule has multiple api calls and some relatively simple data transformation.
Really strange
Thank you Mike, we have used such a solution in the past but here it would not have sufficed as the waiting time was too long at least in some scenarios. Looks like I stumbled on a solution though as per answer above
I hope your list does not have more than a billion items ;-)
Did you read this?
https://docs.appian.com/suite/help/25.2/expressions-parallel-evaluation.html
Yes, it appears to state that a!foreach should do the trick but I have evidence to refute it.
I can set an upper limit just below 100 items, while it looks ugly it works and whatever wizardry Appian is doing behind the scenes I will not be able to change it. There are probably nicer ways to "convince" the Appian engine but I cannot keep experimenting forever due to deadlines.
It looks like you mostly tried various approaches, but my recommendation is,Parallel gateways with subprocess nodes means using a process model where an AND gateway splits the process flow into multiple parallel paths, each containing a subprocess node that calls a separate process model to handle one API call. All subprocess nodes execute concurrently, then an AND gateway converges the paths back together when all subprocesses complete.
Thank you Shubham,
I have not tried that yet. The problems I have with that approach though is that:
1) I do not know beforehand exactly how many items are in the initial array (could be 20, could be 70). I think this would mean designing 70 parallel processes and have them test whether there is an occurrence in the array that matches that?
2) That would definitely break chaining, I would like to preserve it.