Hi,
What is the most efficient way to retrieve data from a large table (more than 4 million rows) in Appian?
When I use queryEntity to fetch data from this table, it takes a long time to execute, which negatively impacts Appian’s performance. How can I improve the data retrieval process?
queryEntity
My current approach to solve the issue:
I split the data retrieval into smaller batches. Specifically, I:
Created multiple records that point to the same source table.
Used the foreach function to query each record in batches of 5,000 rows.
foreach
Used the union function to combine the results from each batch.
union
Used the union function to combine the results from each query record.
Then, combined the results across all the records to get the complete dataset.
I’m concerned about future system complexity.Do I really need to create multiple records and change the system design just to retrieve all the data from a large table? Or is there a better, more efficient way to handle large data retrieval in Appian without adding complexity?
Thanks,
Discussion posts and replies are publicly visible
Mohammed Dyab said:What is the best way to get data from the database to Appian.
a!queryEntity or a!queryRecordType
And you are limited to 5k rows in data, and showing 5k elements on one screen will not be a great UX.
Thanks
Hi Mohammed,
Accidentally stumbled across this and wondered how did you go about this matter and maybe share my view, from a fellow banking sector employee, even if is a little late. :D
What I saw helped with our business colleagues when these similar cases appear is to get a clear objective as Stefan mentioned, after that you can add specific filter mandatory for the user to fill so that your fetched data could be already filtered so that it would be easier for the users to go through and then add pagination and sorting in case is needed. If you underline the benefits of shot sets of data over the drawbacks of "we need all the data" you have a chance to minimize your datasubset.
Regards!