Best approach for migrating 18 million records into Appian Cloud database

Question

Hi,

We have a requirement to migrate approximately 18 million records from an external system into an Appian Cloud database.

We are currently considering the following approaches:

Export data from the source system and import it directly into the Appian Cloud database. However, we noticed there appears to be a file size limitation (around 2 MB per import file), which may make this approach challenging for such a large volume of data.
Use an Appian Process Model with the "Import CSV to Database" Smart Service to load the data in batches.

I would appreciate guidance from anyone who has handled a similar large-scale migration:

Are there any practical limitations or performance considerations when using the Import CSV to Database Smart Service for very large datasets?
Is there a recommended approach for handling data volumes of this size in Appian Cloud?
What is the best way to implement logging and audit tracking so that we can monitor successful and failed records during the migration process?

Any recommendations or lessons learned from similar implementations would be greatly appreciated.

Thank you!

mathieud0001 · Answer

I would recommend against using phpMyAdmin to do this kind of import. Like you said, it has the file size limitation but there are also other issues you could run into with this approach.You could try using theImport CSV to Database orLoad CSV to Database but there are file limitations there as well. If you do go with this approach, I would recommend loading the data in staging tables and then running a process to insert your records.My preferred approach would be to e​​​​​​xpose the table to Appian, you can then run a process that can access the external data source and do the import of the data in batches. Otherwise, the option above it probably best.

Sign up

Login with SSO

Login to the community

Login with SSO