Best way to handle large dataset in cloud databasehttps://community.appian.com/discussions/f/best-practices/38700/best-way-to-handle-large-dataset-in-cloud-databaseHello, I am having trouble with the following use case : we receive 3 large Excel files daily (~60 000 rows and 40 columns), that we need to read and write into Appian Cloud Database. So far, we&#39;ve been doing that this way : 1. Use an ETL to readen-USTelligent Community 12RE: Best way to handle large dataset in cloud databasehttps://community.appian.com/thread/146280?ContentTypeID=1Sun, 16 Mar 2025 17:42:17 GMTd3a83456-d57b-489c-a84c-4e8267bb592a:b9ce00ad-67ad-4192-ad03-b7b78fb1c0bbMathieu Drouin<p>&nbsp;<a href="/success/w/article/3048/how-to-create-memory-efficient-models">https://community.appian.com/success/w/article/3048/how-to-create-memory-efficient-models</a>&nbsp;</p> <ul> <li>Use activity class parameters (as opposed to process variables) when possible to limit the process history size.&nbsp;</li> <li>Release the system memory used by completed nodes by checking the&nbsp;<a href="https://docs.appian.com/suite/help/latest/Other_Tab.html#execution-options">Delete previously completed/cancelled instances</a>&nbsp;setting.</li> <li>If operating on large data sets frequently or for an extended period of time and through many steps, configure the process variable as&nbsp;<code>Hidden</code>. This way, frequent changes to these variables are not going to be reflected in process history, which will have a smaller impact on the process instance memory footprint.</li> </ul><div style="clear:both;"></div>RE: Best way to handle large dataset in cloud databasehttps://community.appian.com/thread/146264?ContentTypeID=1Fri, 14 Mar 2025 19:46:41 GMTd3a83456-d57b-489c-a84c-4e8267bb592a:c82dd129-14d1-493d-b117-4ff4d98458c9Stefan Helzle[quote userid="60156" url="~/discussions/f/best-practices/38700/best-way-to-handle-large-dataset-in-cloud-database"]I am wondering what would be the best way to deal with this issue according to best practices. I&#39;ve thought about using&nbsp;a!writeToMultipleDataStoreEntities directly in our API instead of triggering a process, to avoid having too many process running, but I&#39;m not sure if this would really be more efficient memory-wise ?[/quote] <p>Yes, do this. It will dramatically reduce the overhead.</p><div style="clear:both;"></div>