Hello Appian Community,
I've been given a somewhat complex JSON format to ingest into Appian and stage all the data into the DB. The intention of the data is for reference/comparison/reporting. I've tested the regular readtextfromfile/fromJson/jsonPath functions to grab each attribute but it feels very tedious/clunky to have an JSON path expression for ever attribute I need. Could I get any advice on a more efficient way to parse this data and get it in the DB?
Data Context:
One JSON will have thousands of "FullProduct" entities so I will be dealing with files with a size of at most 25MB. This structure is the final format so there is no altering the original file.
JSON Sample:
{ "FULLPRODUCTFILE": [ { "FullProduct": { "ID": "*dfg333", "Details": { "StatusType": "LIVE", "Status": "ACTIVE" }, "Classification": [ { "ID": "aaa111", "Classification": { "Class": { "ClassificationID": "ASDAFF", "ClassName": "Article 8", "ClassDescription": "Description", "ClassValue": "ARTICLE8" } }, "ClassificationPurpose": "Purpose2" }, { "ID": "aaa222", "Classification": { "Class": { "ClassificationID": "EETEWTT", "ClassName": "Article 2", "ClassDescription": "Description", "ClassValue": "ARTICLE2" } }, "ClassificationPurpose": "Purpose" } ], "Meetings": [ { "ID": "dfgd22f", "MeetingNumber": 2333, "MeetingDate": "2024-12-18T12:00:00.000", "MeetingDepartment": "DEPARTMENT 1", "MeetingComment": "test" } ], "Case": { "CaseDocument": "Document URL", "CaseDocumentDate": "2024-12-18T12:20:57.000" }, "AllMarks": [ { "ID": "hfhgsgfh", "MarksType": "Primary", "Marks": { "MarkDetails": { "MarkName": "TEST_NAME", "MarkType": "TYPE", "MarkDocument": { "MarkDocName": "TEST1" }, "MarkNotes": { "MarkComment": "Test COMMENT" } } } } ], "Rules": { "RuleYear": "2024", "RuleSet": 2 } } } ] }
Discussion posts and replies are publicly visible
What sort of structure are you looking to get in the DB? Is it multiple related tables or just one table? What data from the JSON sample are you wanting across?
Appian can do this of course but it would be remiss of me to not mention that other reporting tools would be better suited to what you are trying to do. There may be other reasons why to use Appian but needed to throw it out there.
According to my understanding, parsing data in Appian can be an efficient way to save information. While this approach may require some effort, it is worth pursuing. Constructing Common Data Types (CDTs) with the parsed data and saving it would be a good strategy. The below code gives you data in Appian
format. from there you can construct your CDT structures. and using wirteDatastorenetity and save it
a!localVariables( local!jsonData: readtextfromfile( txtFile: ri!doc, preserveLineBreaks:true()), local!parsedData: a!fromJson(local!jsonData), )
This would work, but not with 25MB of data. I think this needs a more scalable approach.
Do you need to do this frequently?
Where are these files coming from, and how do you plan to upload them into Appian?
When this data is coming from another system, why not directly connect to it?
Have you taken a look at this plugin? https://community.appian.com/b/appmarket/posts/load-json-file-to-database
Ideally, we want the least amount of tables but are open to multiple tables. We are looking to store all the date from the JSON to do a daily comparison/reporting of the incoming JSON file and any data that may have changed in Appian since the file's nightly load.
This file is being received via SFTP and being dropped into an Appian Folder to be processed on a daily basis. The external system where the JSON originates from does not have a usable API at the moment, so a file-based subscription is our only route.
The daily files we receive will be closer to 5MB-10MB about 90% of the time (Incremental Files). We do still need to be able to support large files to about 25MB (Full Load Files) for database refreshes.
Yes, I have installed this plug-in and tested it with our JSON file and it is useful. I just wanted to give a shout in the Community forums before coming to the conclusion that this is the "only" method of loading a really large and nested JSON somewhat effectively.