Bulk Legacy Document Migration into Appian

Appian Max Team — Tue, 23 Apr 2024 13:04:11 GMT

Current Revision posted to Article by Appian Max Team on 4/23/2024 1:04:11 PM

Bulk Upload

When you need to migrate a large number of documents into Appian, first consider whether you need to maintain metadata or associations with other data from the old system. Appian supports bulk uploads via the Appian Designer interface. The out of the box approach will be a significantly faster solution than designing and testing a migration process. This is a valid approach if the following conditions are met:

The number of documents in the migration is small enough that the manual effort is much faster than an automated effort.
There is no metadata associated with the documents (e.g.: you don't need to associate documents with records in a database).

Automating Document Migration

If your migration can't be accomplished using bulk upload, use an Appian process (or processes) to run the migration and track progress using a database table.

Review the available approaches for moving documents into Appian and the general guidance for transferring/processing large data sets.
Use a temporary database table to hold document metadata and to track the status of each migrated document. Update the status of the documents in the temporary table as documents are migrated.
Implement a configurable delay between document transfers and a configurable limit on the number of documents migrated in each batch to throttle the migration volume.
Performance test your migration solution on a lower level environment whose hardware and data are equivalent to production to determine the correct throttling numbers.
Use the start process smart service to spread the load across process execution engines.
If necessary, update the security on the document folders as the documents are migrated.

Recommended Migration Guidelines

Practice	Reason
Archiving Policy	Delete process instances immediately after completion. Checkpoint and restart the engines after the migration to restore the engines to the proper size.
Reduce Manual Intervention	Incorporate sending status emails to provide updates on the migration.
Incorporate Validation Methods in Status Fields	Test sync failures as a part of your test plan to confirm that the document migration process is able to restart from where it left off in the event of a failure.
Process Optimization	Look at optimizing the process performing the migration. Optimize the database operations. Apply the best practices for creating memory efficient process models.

Tags: integrations, design patterns, Architecture

Bulk Legacy Document Migration into Appian

joel.larin — Thu, 02 Nov 2023 13:54:50 GMT

Revision 5 posted to Article by joel.larin on 11/2/2023 1:54:50 PM

Bulk Upload

The number of documents in the migration is small enough that the manual effort is much faster than an automated effort.
There is no metadata associated with the documents (e.g.: you don't need to associate documents with records in a database).

Automating Document Migration

If your migration can't be accomplished using bulk upload, use an Appian process (or processes) to run the migration and track progress using a database table.

Review the available approaches for moving documents into Appian and the general guidance for transferring/processing large data sets.
Use a temporary database table to hold document metadata and to track the status of each migrated document. Update the status of the documents in the temporary table as documents are migrated.
Implement a configurable delay between document transfers and a configurable limit on the number of documents migrated in each batch to throttle the migration volume.
Performance test your migration solution on a lower level environment whose hardware and data are equivalent to production to determine the correct throttling numbers.
Use the start process smart service to spread the load across process execution engines.
If necessary, update the security on the document folders as the documents are migrated.

Recommended Migration Guidelines

Practice	Reason
Archiving Policy	Delete process instances immediately after completion. Checkpoint and restart the engines after the migration to restore the engines to the proper size.
Reduce Manual Intervention	Incorporate sending status emails to provide updates on the migration.
Incorporate Validation Methods in Status Fields	Test sync failures as a part of your test plan to confirm that the document migration process is able to restart from where it left off in the event of a failure.
Process Optimization	Look at optimizing the process performing the migration. Optimize the database operations. Apply the best practices for creating memory efficient process models.

Tags: integrations, design patterns, Architecture

Bulk Legacy Document Migration into Appian

matt.cosenza — Wed, 01 Nov 2023 15:09:00 GMT

Revision 4 posted to Article by matt.cosenza on 11/1/2023 3:09:00 PM

Bulk Upload

The number of documents in the migration is small enough that the manual effort is much faster than an automated effort.
There is no metadata associated with the documents (e.g.: you don't need to associate documents with records in a database).

Automating Document Migration

If your migration can't be accomplished using bulk upload, use an Appian process (or processes) to run the migration and track progress using a database table.

Review the available approaches for moving documents into Appian and the general guidance for transferring/processing large data sets.
Use a temporary database table to hold document metadata and to track the status of each migrated document. Update the status of the documents in the temporary table as documents are migrated.
Implement a configurable delay between document transfers and a configurable limit on the number of documents migrated in each batch to throttle the migration volume.
Performance test your migration solution on a lower level environment whose hardware and data are equivalent to production to determine the correct throttling numbers.
Use the start process smart service to spread the load across process execution engines.
If necessary, update the security on the document folders as the documents are migrated.

Recommended Migration Guidelines

Practice	Reason
Archiving Policy	Delete process instances immediately after completion. Checkpoint and restart the engines after the migration to restore the engines to the proper size.
Reduce Manual Intervention	Incorporate sending status emails to provide updates on the migration.
Incorporate Validation Methods in Status Fields	Test sync failures as a part of your test plan to confirm that the document migration process is able to restart from where it left off in the event of a failure.
Process Optimization	Look at optimizing the process performing the migration. Optimize the database operations. Apply the best practices for creating memory efficient process models.

Tags: integrations, design patterns, Architecture

Bulk Legacy Document Migration into Appian

matt.cosenza — Wed, 01 Nov 2023 15:08:56 GMT

Revision 3 posted to Article by matt.cosenza on 11/1/2023 3:08:56 PM

Bulk Upload

The number of documents in the migration is small enough that the manual effort is much faster than an automated effort.
There is no metadata associated with the documents (e.g.: you don't need to associate documents with records in a database).

Automating Document Migration

If your migration can't be accomplished using bulk upload, use an Appian process (or processes) to run the migration and track progress using a database table.

Review the available approaches for moving documents into Appian and the general guidance for transferring/processing large data sets.
Use a temporary database table to hold document metadata and to track the status of each migrated document. Update the status of the documents in the temporary table as documents are migrated.
Implement a configurable delay between document transfers and a configurable limit on the number of documents migrated in each batch to throttle the migration volume.
Performance test your migration solution on a lower level environment whose hardware and data are equivalent to production to determine the correct throttling numbers.
Use the start process smart service to spread the load across process execution engines.
If necessary, update the security on the document folders as the documents are migrated.

Recommended Migration Guidelines

Practice	Reason
Archiving Policy	Delete process instances immediately after completion. Checkpoint and restart the engines after the migration to restore the engines to the proper size.
Reduce Manual Intervention	Incorporate sending status emails to provide updates on the migration.
Incorporate Validation Methods in Status Fields	Test sync failures as a part of your test plan to confirm that the document migration process is able to restart from where it left off in the event of a failure.
Process Optimization	Look at optimizing the process performing the migration. Optimize the database operations. Apply the best practices for creating memory efficient process models.

Bulk Legacy Document Migration into Appian

matt.cosenza — Wed, 18 Oct 2023 18:23:10 GMT

Revision 2 posted to Article by matt.cosenza on 10/18/2023 6:23:10 PM

Bulk Upload

The number of documents in the migration is small enough that the manual effort is much faster than an automated effort.
There is no metadata associated with the documents (e.g.: you don't need to associate documents with records in a database).

Automating Document Migration

If your migration can't be accomplished using bulk upload, use an Appian process (or processes) to run the migration and track progress using a database table.

Review the available approaches for moving documents into Appian and the general guidance for transferring/processing large data sets.
Use a temporary database table to hold document metadata and to track the status of each migrated document. Update the status of the documents in the temporary table as documents are migrated.
Implement a configurable delay between document transfers and a configurable limit on the number of documents migrated in each batch to throttle the migration volume.
Performance test your migration solution on a lower level environment whose hardware and data are equivalent to production to determine the correct throttling numbers.
Use the start process smart service to spread the load across process execution engines.
If necessary, update the security on the document folders as the documents are migrated.

Recommended Migration Guidelines

Practice	Reason
Archiving Policy	Delete process instances immediately after completion. Checkpoint and restart the engines after the migration to restore the engines to the proper size.
Reduce Manual Intervention	Incorporate sending status emails to provide updates on the migration.
Incorporate Validation Methods in Status Fields	Test sync failures as a part of your test plan to confirm that the document migration process is able to restart from where it left off in the event of a failure.
Process Optimization	Look at optimizing the process performing the migration. Optimize the database operations. Apply the best practices for creating memory efficient process models.

Bulk Legacy Document Migration into Appian

joel.larin — Wed, 06 Sep 2023 14:19:25 GMT

Revision 1 posted to Article by joel.larin on 9/6/2023 2:19:25 PM

fsdfs