How much max no of records appian gets from database

Question

Hi, 
 I am getting data from REST API, for 3 Lacs of only ID's getting from database I am getting below error. 
 Failed to connect to http://machine:5097/api/accountids Unable to process response body - The response body was greater than 5 MB and could not be processed 
 
 May i know why this is limitation. How to overcome it. 
 
 I need to get around 25 Lacs of ID's. The reason is in Appian Record it expects all Id's to be present in order to work record properly. (I have requirement like, open screen -> filter record -> fetch data using REST API & show on grid -> click on any record it should display summary of that record (To display summary it expects all records should present). 
 
 I am really surprised the way Appian works. Any help is highly appreciated.

Dave Lewis · Answer

I think there may be some misunderstanding. When you click on a record, you should query for that one record, using the filter provided, so you don't return all the records in the entire database. 
 
If you are looking at the list of records, it only ever shows 100, meaning there's no benefit to grabbing more than the first 100 records, and thereby not grabbing the entire database. If you want records that don't display in that first 100, you apply a filter at the top and requery the database. 
 
If you think you have to pull EVERY record and you have more than 100, let alone 2,500,000 records, I think you need to reconsider what you're doing to your database. You really want to pull the minimum amount of data possible at all times. 
 
The limitation is there to prevent you from writing queries that take over ten minutes for even a super fast database server to execute.