Automation of Bulk Upload of data

Most databases have a way to bulk load/transform data as well as scheduling jobs. You could probably have your database run a job that checks a set directory for a certain file name and then parses the data, loading each row of data into the db in bulk using its native ETL/bulk loading processes. You would only use Appian to facilitate the user uploading the file and having it store a copy in a db accessible path on the server.

fyi, we have some apps that load CSV files nightly and it works ok for <100 rows of data, but one application loads hundreds of rows of data every 1-2 hours throughout the day. For each row we kick off a new process. Needless to say, starting a few hundred processes in quick succession with each process receiving a CDT of 20+ columns, running a few db queries, updating the CDT and then writing that to the datastore is NOT efficient. If we use subprocesses/MNI in sets of say 50 we have performance problems because we overload the engine (a parent process spawns subprocesses on the same engine). I am converting some of this to start processes via messages so that they run spread out on all of our execution engines/servers...hoping this helps stability & reduces issues and user interface problems for users on the same engine while loading data (code change goes live tonight).

Most databases have a way to bulk load/transform data as well as scheduling jobs. You could probably have your database run a job that checks a set directory for a certain file name and then parses the data, loading each row of data into the db in bulk using its native ETL/bulk loading processes. You would only use Appian to facilitate the user uploading the file and having it store a copy in a db accessible path on the server.

fyi, we have some apps that load CSV files nightly and it works ok for <100 rows of data, but one application loads hundreds of rows of data every 1-2 hours throughout the day. For each row we kick off a new process. Needless to say, starting a few hundred processes in quick succession with each process receiving a CDT of 20+ columns, running a few db queries, updating the CDT and then writing that to the datastore is NOT efficient. If we use subprocesses/MNI in sets of say 50 we have performance problems because we overload the engine (a parent process spawns subprocesses on the same engine). I am converting some of this to start processes via messages so that they run spread out on all of our execution engines/servers...hoping this helps stability & reduces issues and user interface problems for users on the same engine while loading data (code change goes live tonight).