Best way to handle large dataset in cloud database

Hello, 

I am having trouble with the following use case : we receive 3 large Excel files daily (~60 000 rows and 40 columns), that we need to read and write into Appian Cloud Database. So far, we've been doing that this way : 

1. Use an ETL to read the file and batch the extracted JSON data

2. Send the data in batches to Appian with an API exposed by Appian

3. The API triggers a process that write the data to the table (Using the node Write to Data Store Entity)

However, this approach is starting to cause some memory issues, as there is more than 150 processes trigerred daily, and each of these processes contains large process variables (even with batching, we still have 150 processes dealing with 500 rows each).

I am wondering what would be the best way to deal with this issue according to best practices. I've thought about using a!writeToMultipleDataStoreEntities directly in our API instead of triggering a process, to avoid having too many process running, but I'm not sure if this would really be more efficient memory-wise ?

Thanks