Hi everybody, We are looking for advices/best practices prior to deve

... solutions, we are really eagger to hear your insights on what would be the best design and what workarounds could be found on those limitations.

Thank you very much!

I suppose you could display the data via an excel file?

I think there is a component that does this via SQL data. At least this way you get a potentially large data set in one go and excel will allow users to filter etc.

Another way would be to display the data via a paging grid and query rule to support this. The downside is if it is a big data set then they could be paging for a while!

If I was looking at this I think I would be looking to keep the process as light as possible and perhaps take the approver to the data somehow and then a task to indicate whether it was approved or not.

When you read from the Excel file into PVs, can you use the filter() function to apply your custom expression to remove all but the matching items (forum.appian.com/.../Looping_Functions.

In order to get uniques, you can do union (pv!filteredList, pv!filteredList) which will remove any duplicates (forum.appian.com/.../Set_Functions.

By using using filter() instead of MNI, you won't run into the node limit. And using union() instead of a Query Database node, you won't run into the query limit.

One thing to keep in mind, however, is union will compare each CDT to each other. So if the CDTs themselves aren't the same but just have fields that are the same, you'll need to do union on the field, not the CDT, and then match the results back up with the individual CDTs.

Sorry for for the late reply and thank you for your answers.
@davidwi: To display the data via the excel file is not quite satisfactory for us as we'd like to provide a consitent user experience, eg to keep the user in Appian interface. I will make some tests with standard/paging grids.
@jed: in my use case, the matching must unfortunately be applied to all inputs. Each input has 2 kinks of properties – unique (ex: id,location) and common (ex: product, product type). We are performing the “matching” on common properties to “clean” the data. Sorry if it not quite clear but that is to say that we cannot use the filter function at first.
Our understanding right now is that we should avoid to manipulate big data with appian functions mainly because of performance issues and limitations. We are leaning towards custom smart services to run java programs that will provide outputs ready to be displayed into paging grids. Do you think that our assessment is correct?

Is there a reason why you are not using the database itself to do the number crunching. Stored procedures are definitely the way to go.

The performance would far exceed anything you could write in process or custom plugins.

Appian's strength lie in the presentation and sharing of data, not high load manipulation.

I totally agree with you, that s the way we are inclined to take now.