Huge data fetch - "Memory threshold" problem

Hi All,

I have a requirement in which there is a table of 30 columns and 1 million records.
Requirement is to fetch those records and write the data into an excel document.

1. I have created an xsd/cdt which got verfied and mapped with the table in DB
2. When i am trying to query the data with a query rule, the rule is fetching data only if the records number is <= 1400

If i try to fetch 1500 records, error occurs on the script node in which i am fetching and it sends me following error in the alert

There is a problem with task “Script Task” in the process “XXX Test”. ERROR:An error occurred while evaluating expression: CCC_CDTs:rule!YYY(pv!cid) (Expression evaluation error in rule 'YYY' at function 'queryAAA': Error evaluating function 'queryAAA' :

Memory threshold reached during output conversion

(rule: [YYY], type: [CCCDT28858], threshold: [1,048,576 bytes], objects successfully conver...

OriginalPostID-252548

  Discussion posts and replies are publicly visible

Parents
  • Also, this does not sound like a good idea. Why Excel? Is this to export data? Excel can't handle that many rows and the file will be extremely large, csv can be used with a plugin but even then, the file is unviewable. If you are exporting the data to another DB, a csv may be appropriate but you may want to explore other technologies (ETL tools, etc.)
Reply
  • Also, this does not sound like a good idea. Why Excel? Is this to export data? Excel can't handle that many rows and the file will be extremely large, csv can be used with a plugin but even then, the file is unviewable. If you are exporting the data to another DB, a csv may be appropriate but you may want to explore other technologies (ETL tools, etc.)
Children
No Data