I have handle two XML files daily, each with a size of 50MB or more. These files are transferred via SFTP or FTP. Our goal is to extract specific values from these files, check their existence in our database, and either update existing records or insert new ones as necessary.Requirements: File Retrieval: Automate the retrieval of XML files from an SFTP or FTP server. Store the files temporarily within Appian or an external file storage service.
Data Extraction: Parse the XML files to extract a list of values. Given the large file sizes, efficient parsing and processing strategies are needed.
Data Validation and Update: Compare extracted values against existing database records. Update existing records with the latest values or insert new records as required.
Performance Considerations: Handle large files efficiently, possibly by processing in chunks. Ensure database operations (updates and inserts) are optimized for performance.can any one suggest some best approach to do this.
Discussion posts and replies are publicly visible
I agree with Stefan, I would not build this in Appian if I can avoid it.
Normally, you would read the text file with Text File Reader plugin, then you would be able to extract the XML values and do the rest of the processing.
Problem with the Text File Reader plugin is that you are constrained by the amount of memory that you have available in your environment. I would not recommend reading large files with this plugin.