https://community.appian.com/discussions/f/best-practices/14922/processing-50000-entriesHi all, We have the requirement to read from different databases 50k + entries, apply some validation rules and for the ones that fail the validaiton generate an approval workflow. The recommendation that we have been given to the customer isen-USTelligent Community 12https://community.appian.com/thread/66788?ContentTypeID=1Wed, 22 May 2019 08:19:34 GMTd3a83456-d57b-489c-a84c-4e8267bb592a:009a7123-6384-48d0-99ec-72d31da5d432jesusa583<p>Hi Stewart. Option 1 is also our preferred way. We still Need to find out how complex is the logic to filter the data. If possible we will use a stored procedure. I dont know yet the size of the filtered data but even if it is big as it is already filtered we will only Need the list of Ids and Display them in some forms. Thanks for your answer.</p><div style="clear:both;"></div>https://community.appian.com/thread/66787?ContentTypeID=1Wed, 22 May 2019 08:06:18 GMTd3a83456-d57b-489c-a84c-4e8267bb592a:074ba98d-daab-495b-b6b7-da7ef2b960e7Stewart Burchell<ol> <li>How complex are the validation rules? Could these be effectively applied in the initial data-set selection? That is, the data is effectively pre-filtered before it ever gets to Appian?</li> <li>If you can do 1. (and you can&nbsp;conduct some quite sophisticated logic in the database layer e.g. using a Stored Procedure) then what&#39;s the likely size of the result-set? 1%? 10%? 50%? 95%? It&#39;s worthwhile profiling the data to understand the magnitude of the problem and then you can design accordingly.</li> </ol><div style="clear:both;"></div>https://community.appian.com/thread/66781?ContentTypeID=1Wed, 22 May 2019 01:31:26 GMTd3a83456-d57b-489c-a84c-4e8267bb592a:72bb3181-e5a2-49a4-9c7d-7e93f8a1f983TJ<p>Agreed. Appian advises complex data processes to be done outside the Appian platform.&nbsp;&nbsp;Please use a Java service or employ ETL and then trigger Appian workflow.</p><div style="clear:both;"></div>