Automation Bulk Upload

It's not entirely clear what problem you're trying to solve, but a common pattern for processing large volumes of data is to initially store the data in its raw, unprocessed format into a 'staging' table in a database. In this way you can provide an initial response to the to the sending system along the lines of "Your data has been successfully received". 

Now you can process the data in whatever way makes sense. Large volumes can be broken down into smaller batches (or even processed singly if that makes sense). The staging table can also be used to track the processing status of the individual 'transfer' (e.g. you can mark as 'Successfully processed' or 'Validation Error') and then you can subsequently respond to the sending system with "We received 10,000 transfer. 9,997 were successfully processed. 3 had validation errors and here are the details...". Like any large problem, break it down into a series of smaller ones and solve them one at a time.

It's not entirely clear what problem you're trying to solve, but a common pattern for processing large volumes of data is to initially store the data in its raw, unprocessed format into a 'staging' table in a database. In this way you can provide an initial response to the to the sending system along the lines of "Your data has been successfully received". 

Now you can process the data in whatever way makes sense. Large volumes can be broken down into smaller batches (or even processed singly if that makes sense). The staging table can also be used to track the processing status of the individual 'transfer' (e.g. you can mark as 'Successfully processed' or 'Validation Error') and then you can subsequently respond to the sending system with "We received 10,000 transfer. 9,997 were successfully processed. 3 had validation errors and here are the details...". Like any large problem, break it down into a series of smaller ones and solve them one at a time.