Hi everybody, We are looking for advices/best practices prior to deve

fabienb

A Score Level 1

over 12 years ago

Hi everybody,

We are looking for advices/best practices prior to develop an application around data manipulation. Among the requirements, we need to:

- create a table from an excel file
- improve data quality using matching alogrithm
- display the matching result and get approvals from users

The expected number of input rows is around 10k+ but the number of distinct "matching" is more about a couple of hundreds I suppose.

For now, we have been thinking at the following solutions :

1- To create the table : use the excel file plugin + create CDTs + use write to data store entity Smart Service
2- To perform the matching, we have a custom expression that works just fine
3- To display distinct matchings: use the Query Database Smart service (SELECT DISTINCT... ) + create CDTs + use of paging grids for display

We encounter some limitations for 1&3 (nodes limit (1000), number of max rows returned by a query (10), latency) and prior to move on custom...

Discussion posts and replies are publicly visible

Parents

0 Jed Fonner
Appian Employee
over 12 years ago

When you read from the Excel file into PVs, can you use the filter() function to apply your custom expression to remove all but the matching items (forum.appian.com/.../Looping_Functions.

In order to get uniques, you can do union (pv!filteredList, pv!filteredList) which will remove any duplicates (forum.appian.com/.../Set_Functions.

By using using filter() instead of MNI, you won't run into the node limit. And using union() instead of a Query Database node, you won't run into the query limit.

One thing to keep in mind, however, is union will compare each CDT to each other. So if the CDTs themselves aren't the same but just have fields that are the same, you'll need to do union on the field, not the CDT, and then match the results back up with the individual CDTs.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 Jed Fonner
Appian Employee
over 12 years ago

When you read from the Excel file into PVs, can you use the filter() function to apply your custom expression to remove all but the matching items (forum.appian.com/.../Looping_Functions.

In order to get uniques, you can do union (pv!filteredList, pv!filteredList) which will remove any duplicates (forum.appian.com/.../Set_Functions.

By using using filter() instead of MNI, you won't run into the node limit. And using union() instead of a Query Database node, you won't run into the query limit.

One thing to keep in mind, however, is union will compare each CDT to each other. So if the CDTs themselves aren't the same but just have fields that are the same, you'll need to do union on the field, not the CDT, and then match the results back up with the individual CDTs.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

No Data