Memory Circuit Breaker

Hi Shiva Prasad,

What is causing the memory circuit breaker error? Can you help with  the root cause, based on the we can think of solutions.

 Is it because of too many queries or number of lines of code, or execute SP's in interface? 

1. We had 2 local variables with datasubset, both the local variable contains same amount of data but the values are different, for the 1st grid we are using 1st local variable as data source and displaying the data in grid but here we are additional using tag field with white color for each value, in the 2nd grid we are using 2nd local variable as source and accordingly displaying the data using tag field, but for adding color to tag field we are comparing each cell of second grid with corresponding cell of 1st grid and using white color if the data is same else using red color.
2. We are highlighting the values that got changed. Here the data we are receiving is dynamic, so when the number of colomns are less it's working fine, if the colomns size increases we are getting Memory Circuit breaker. Here our requirement is like in such scenarios we should be able to capture the error and display a banner like we cannot compare. 

1. We had 2 local variables with datasubset, both the local variable contains same amount of data but the values are different, for the 1st grid we are using 1st local variable as data source and displaying the data in grid but here we are additional using tag field with white color for each value, in the 2nd grid we are using 2nd local variable as source and accordingly displaying the data using tag field, but for adding color to tag field we are comparing each cell of second grid with corresponding cell of 1st grid and using white color if the data is same else using red color.
2. We are highlighting the values that got changed. Here the data we are receiving is dynamic, so when the number of colomns are less it's working fine, if the colomns size increases we are getting Memory Circuit breaker. Here our requirement is like in such scenarios we should be able to capture the error and display a banner like we cannot compare. 

 Stefan Helzle karthikkanjerla 

To add details to above description, to that local variables we are gettiong data from below expression rule

To the above rule we are passing the documentID  from which we need to extract data, check the below sample document

In this document the 1st row always indicates column headers seperated with comma(A,B,C,D,E) and the rest of rows(rows starting from 2nd line) are actual rows need to be displayed in grid. In this file we have 5 columns with 7 rows but some times we might recieve huge files like 2000columns with 1000rows or more.

When the file has less data it is working fine but during huge data the expression rule is failing at line 8, where we are converting data to list of dictionary using foreach loop and resulting in memory circuit breaker. In such scenerios instead of giving memory circuit breaker error, we should show some user friendly error and allow the user to move forward in the interface.

 Stefan Helzle Cas we achieve this or as per your previous response we cant achieve this?

 Stefan Helzle karthikkanjerla and any one looking at the this discussion please add your comments

One issue with your approach is, that you store the whole data three times in different local variables. This will not help reducing memory consumption.

But, in general, you have to make sure that these files stay within certain limits. In case that's not an option, you will have to find a different way to do, what you need to do.

Final question Stefan Helzle 

If we do it in different way or if we change the logic and at the end, we get the error for some other huge file. Then we can't handle it other than restricting the file to certain limit right? is that the final conclusion

Appian is not made to handle "huge" files in memory. You either have to limit file size, or find a different way of implementation.

You can try the following approach:

• Instead of handling these large files in SAIL take help of database.
  • You can create a staging/temporary table in the databse to store data of these files.
  • Load data of files into these tables.
  • if data volume is large you can take help of batching.
• It will be easy to create/maintain grids from database.
• Instead of loading all the rows in the local variable, only load limited rows you are displaying in a page. You can achieve it keeping paginginfo same for local of datasubset and paginggrid.
• The logic for comparing rows from first file to another you can build on database side using view, materialized view or SP.