Hi everybody, We are looking for advices/best practices prior to deve

fabienb

A Score Level 1

over 11 years ago

Hi everybody,

We are looking for advices/best practices prior to develop an application around data manipulation. Among the requirements, we need to:

- create a table from an excel file
- improve data quality using matching alogrithm
- display the matching result and get approvals from users

The expected number of input rows is around 10k+ but the number of distinct "matching" is more about a couple of hundreds I suppose.

For now, we have been thinking at the following solutions :

1- To create the table : use the excel file plugin + create CDTs + use write to data store entity Smart Service
2- To perform the matching, we have a custom expression that works just fine
3- To display distinct matchings: use the Query Database Smart service (SELECT DISTINCT... ) + create CDTs + use of paging grids for display

We encounter some limitations for 1&3 (nodes limit (1000), number of max rows returned by a query (10), latency) and prior to move on custom...

Discussion posts and replies are publicly visible

Parents

0 fabienb
A Score Level 1
over 11 years ago

Sorry for for the late reply and thank you for your answers.
@davidwi: To display the data via the excel file is not quite satisfactory for us as we'd like to provide a consitent user experience, eg to keep the user in Appian interface. I will make some tests with standard/paging grids.
@jed: in my use case, the matching must unfortunately be applied to all inputs. Each input has 2 kinks of properties – unique (ex: id,location) and common (ex: product, product type). We are performing the “matching” on common properties to “clean” the data. Sorry if it not quite clear but that is to say that we cannot use the filter function at first.
Our understanding right now is that we should avoid to manipulate big data with appian functions mainly because of performance issues and limitations. We are leaning towards custom smart services to run java programs that will provide outputs ready to be displayed into paging grids. Do you think that our assessment is correct?
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 fabienb
A Score Level 1
over 11 years ago

Sorry for for the late reply and thank you for your answers.
@davidwi: To display the data via the excel file is not quite satisfactory for us as we'd like to provide a consitent user experience, eg to keep the user in Appian interface. I will make some tests with standard/paging grids.
@jed: in my use case, the matching must unfortunately be applied to all inputs. Each input has 2 kinks of properties – unique (ex: id,location) and common (ex: product, product type). We are performing the “matching” on common properties to “clean” the data. Sorry if it not quite clear but that is to say that we cannot use the filter function at first.
Our understanding right now is that we should avoid to manipulate big data with appian functions mainly because of performance issues and limitations. We are leaning towards custom smart services to run java programs that will provide outputs ready to be displayed into paging grids. Do you think that our assessment is correct?
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

No Data