Skip to main content
S
stevenh6272
January 15, 2026
Question

How to handle Json data of unusual size

  • January 15, 2026
  • 6 replies
  • 0 views

Appian 24.1

My project receives a JSON document that is somewhere between 50 - 100 KB in size normally. We jamb it into a Record Type (as a json string), read it from a process model, convert this to a CDT (hierarchy), pass it to an interface (a hierarchy of them really). So far, all that we have built runs fine.

Now we had a case where the JSON document was over 1 MB. This threw an error when we tried to write it to a record type. Apparently, there is a record size limit of 1 MB within Appian for tables that sync to record types. So, now we are looking at redesigning the data storage.

Clearly, we need to break up/normalize the data into a set of relational tables (vs just 1 table as we have today). But from there I seem to have some options and I don't know what is best

Option 1 - Read the record types within the process model and rebuild the JSON object so it may be passed to the interfaces.
This moves the max size question from the table row to the process model variable. Can it hold 1-2 MB data? Should it? This also minimizes the changes to the current system as the interfaces need no alteration.

Option 2 - Have the interfaces read the record types dynamically when displaying the data.
Here the process model would pass PK information to the interface and the interface pulls the data when it renders. This changes the problem from a data size concern to a UX performance concern. I see lag when the process model reads/writes the record type of a few seconds. I would hate to see multiple, few second lags due to various interfaces loading data from different record types.

Which of these should I use? Or is there some other way to handle large JSON data?

Thanks.

6 replies

shubhama926776
shubhama926776
January 16, 2026

Option 2 with Document storage: Store raw JSON as Document (bypasses 1MB record limit), normalize to relational tables, and have process models pass only primary keys to interfaces which query records with pagination (avoids 5MB process variable limits and memory issues). 

S
stevenh6272Author
January 16, 2026

To save the JSON would I use Text Doc From Template smart service?  (I'm receiving the entire JSON document from an integration call).
EDIT:  I was able to create the json document using Text Doc From Template.  That leads to another question.... what limits apply to these documents?  Is it okay If I have, say, a quarter million json documents?

Would I need both to have the raw JSON as a document AND still need relational tables?  Currently, the JSON data is converted into a CDT structure which is passed to the interfaces.  If I read the JSON from a document then convert to CDT, couldn't I just pass the whole CDT (as I do today) to the interface?  

Thanks.

stefanhelzle0001
stefanhelzle0001
January 17, 2026

Working directly with external data structures is always a risk, as this data typically changes frequently.

See my podcast episode: https://appian.rocks/2025/11/06/episode-31-dealing-with-external-data-models/

Once you made that decision, I would probably transform it into a records structure after receiving.

The query time you observe in process is not the actual query time, but reflects the end-to-end node execution time which includes queuing etc. Appian does to take a second to fetch some fields from DB.

If you only need to display that data in an interface, pass only the PK, and fetch data as needed. If you need to modify the data, fetch the data in process, pass it to the interface, and write it back in a write records node.

harshas2775
harshas2775
January 20, 2026

Process models generally execute faster than expressions in UI. So I would suggest to go with Option 1. Do the major processing within process model then pass data into UI. As the process variable will have large amount of data, you can keep the variable hidden and also keep the data management of process minimal to save on process memory consumption. Consider deletion instead of archival if feasible. 

Can it hold 1-2 MB data

I am not sure size wise how much data it can hold, I assume it should work fine. But you can give it a try and let me know here if it works. 

S
stevenh6272Author
January 20, 2026

Based on the feedback I have received so far, I think I will go with my option 1 above.  Something I read somewhere said the limit for process variables is something like 5 MB which, if correct, is plenty of space for my need.

Thanks everyone for your input.

Terms & ConditionsAccessibility statement