Hello Everyone, We have a scenario where an user will upload an excel

shailjas over 8 years ago

Hello Everyone,

We have a scenario where an user will upload an excel file containing data and system will parse the data and write the details to the database.
Each record in the excel corresponds to a record in the data base. We are using the Read Excel plugin to parse and Appian Write to data stores to write to the database.
There is no upper limit on the number of records in the excel. User can upload 40000 or even more records. If we try to write such large data sets into the database, it is causing a server outage.

Also, after the data is written to the database, an approval task is generated. In this task, an approver can upload a new version of the Excel File.
If a new version is available, then all the records written to the database earlier must be deleted and new records will be written for the case.

We would like to know the various design approaches we can consider to write and delete large number of records into the database.

Discussion posts and replies are publicly visible

Parents

0 sikhivahans over 8 years ago

@mohamedb Hi, I haven't dealt with parsing a huge dataset so far, so I am not answering the first question.

2) After, you write your batches to DB - checking the data; have you ever had issues where not all of data was written to DB from PVs? or having empty rows/columns inserted which requires that you cross check the xls with DB manually and correct deviations?

To the best of my knowledge, there won't be any deviation and the operation completes perfectly as long as the data is properly parsed from source(excel, csv etc) and stored into PVs. I have written nearly hundreds of thousands of rows in a single shot into the datastore entity and I haven't seen any issue while writing such huge amounts of data.

If we keep the deviations aside, I don't think practitioners are writing such huge datasets to datastore entity in a single shot now a days since the introduction of Health Check. Health Check flags the operations that tries to interact with Datastore with huge datasets as High risk. But still, keeping the Health Check suggestions or its implementation guidelines aside, definitely there won't be any problem in making the updates as long as the PV holds a properly parsed data.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 sikhivahans over 8 years ago

@mohamedb Hi, I haven't dealt with parsing a huge dataset so far, so I am not answering the first question.

2) After, you write your batches to DB - checking the data; have you ever had issues where not all of data was written to DB from PVs? or having empty rows/columns inserted which requires that you cross check the xls with DB manually and correct deviations?

To the best of my knowledge, there won't be any deviation and the operation completes perfectly as long as the data is properly parsed from source(excel, csv etc) and stored into PVs. I have written nearly hundreds of thousands of rows in a single shot into the datastore entity and I haven't seen any issue while writing such huge amounts of data.

If we keep the deviations aside, I don't think practitioners are writing such huge datasets to datastore entity in a single shot now a days since the introduction of Health Check. Health Check flags the operations that tries to interact with Datastore with huge datasets as High risk. But still, keeping the Health Check suggestions or its implementation guidelines aside, definitely there won't be any problem in making the updates as long as the PV holds a properly parsed data.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

No Data