hi, everyone!
I need to the automation process of bulk transfers. there are can be 1500-10000 transfers. We need to store, validate it with existing rules. What's the best way to do it? It will be ongoing process.
Discussion posts and replies are publicly visible
Can you add more information on what you mean by "transfers"? Are these documents? What's the source and destination? What type of validation are you looking for?
Just imagine it as an Object with attributes (CDT with some nested data types), which comes in JSON format ... we need to store it, do some transformation and check some fields using some rules. The point is large data for processing.
A sound approach to send data in JSON format to Appian would be across an Appian Web API.
Have at looks at the following documentation on Appian Web APIs:
https://docs.appian.com/suite/help/19.4/Web_APIs.html
It's not entirely clear what problem you're trying to solve, but a common pattern for processing large volumes of data is to initially store the data in its raw, unprocessed format into a 'staging' table in a database. In this way you can provide an initial response to the to the sending system along the lines of "Your data has been successfully received".
Now you can process the data in whatever way makes sense. Large volumes can be broken down into smaller batches (or even processed singly if that makes sense). The staging table can also be used to track the processing status of the individual 'transfer' (e.g. you can mark as 'Successfully processed' or 'Validation Error') and then you can subsequently respond to the sending system with "We received 10,000 transfer. 9,997 were successfully processed. 3 had validation errors and here are the details...". Like any large problem, break it down into a series of smaller ones and solve them one at a time.