We have an external event that can impact up to 1M+ Case records. For each affected Case, we need to:
We're evaluating the following approaches:
For workloads that can exceed 1 million records, which approach would you recommend and why?
While Option 1 benefits from automatic record synchronization, it may place significant load on Appian process execution and engines at this scale. Option 2 shifts the bulk processing to the database but requires additional ID tracking and explicit record synchronization.
Has anyone implemented a similar solution at this volume? Would you recommend Option 1A, Option 1B, Option 2, or another approach altogether for the best balance of performance, scalability, and Appian platform load?
Discussion posts and replies are publicly visible
Stored procedure (Option 2) is better because the database can do set-based bulk updates much faster and with less Appian engine load than millions of Write Records executions, especially at 1M+ scale.
thanks Manuel Otero & Shubham Aware
Is it fair to assume that using a stored procedure for the bulk update/insert, then querying the affected IDs (via CDTs or an unsynced Record Type) from staging tables and running Record Sync in batches of 1,000, would result in significantly lower Appian engine load than performing the updates through Write Records at 1M+ scale?
Given the volume, it may not be feasible to return and track all affected Case and Event History IDs as stored procedure output, so we'd likely need to persist them in staging tables and process them in batches for synchronization.
Also, would incremental syncs be a better option in this scenario? We'd need business validation, as they would introduce a synchronization delay and our current minimum acceptable latency is around 15 minutes.please share your thoughts experts...
Yes. Stored procedure + staging tables + 1,000-Id batch sync significantly cuts Appian engine load vs 1M+ Write Records. Incremental sync with a 5-15 min lag is a supported high-volume pattern.