Hello Everyone, We have a scenario where an user will upload an excel

shailjas over 9 years ago

Hello Everyone,

We have a scenario where an user will upload an excel file containing data and system will parse the data and write the details to the database.
Each record in the excel corresponds to a record in the data base. We are using the Read Excel plugin to parse and Appian Write to data stores to write to the database.
There is no upper limit on the number of records in the excel. User can upload 40000 or even more records. If we try to write such large data sets into the database, it is causing a server outage.

Also, after the data is written to the database, an approval task is generated. In this task, an approver can upload a new version of the Excel File.
If a new version is available, then all the records written to the database earlier must be deleted and new records will be written for the case.

We would like to know the various design approaches we can consider to write and delete large number of records into the database.

Discussion posts and replies are publicly visible

Parents

0 shailjas over 9 years ago

@mohanp,

1) When Approver is uploading a new version of file, Is all the data is different than what previous data has been written? (i.e, all rows and column values are different or only few rows/column values are different) If all rows are not modified than it's better you indentify which rows are modified and only update modfied rows to DB.

When the user uploads a new file, there could be a possibility that few object numbers may be same.
2) Is there any specific need to write data into Database before Approval task? if not,I would suggest you to write data once when you have a final file from the Approval Task?
Yes. We also have a reporting functionality where users will search based on an excel data and the report will list out the cases that contain the data.
If we write the data after the approval, then the search operation between the first task and Approval Task will be broken.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 shailjas over 9 years ago

@mohanp,

1) When Approver is uploading a new version of file, Is all the data is different than what previous data has been written? (i.e, all rows and column values are different or only few rows/column values are different) If all rows are not modified than it's better you indentify which rows are modified and only update modfied rows to DB.

When the user uploads a new file, there could be a possibility that few object numbers may be same.
2) Is there any specific need to write data into Database before Approval task? if not,I would suggest you to write data once when you have a final file from the Approval Task?
Yes. We also have a reporting functionality where users will search based on an excel data and the report will list out the cases that contain the data.
If we write the data after the approval, then the search operation between the first task and Approval Task will be broken.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

No Data