Hi. I'm a novice so as much help as possible would be appreciated! I have managed to work out how to extract the data I want via an API, however, there's too much of it and I get a memory error when setting the batch size to -1. Smaller batches work fine, so:1. How do I loop the code below so that it returns smaller batches (1000)?2. How do I merge these batches back into one file as would happen in a single call to the endpoint?Thanks in advance,
Adrian
a!localVariables( local!entities: a!queryEntity( entity: cons!TDW_REVIEWERS_ANSWER_DS, query: a!query( selection: a!querySelection( columns: { a!queryColumn(field: "respondedDate"), a!queryColumn(field: "schemeId"), a!queryColumn(field: "answer"), a!queryColumn(field: "comment"), a!queryColumn(field: "reviewer") } ), logicalExpression: a!queryLogicalExpression( operator: "AND", filters: { a!queryFilter( field: "schemeId", operator: ">=", value: 70000 ), a!queryFilter(field: "comment", operator: ">", value: " "), a!queryFilter( field: "comment", operator: "<>", value: "Test" ) }, ignoreFiltersWithEmptyValues: true ), pagingInfo: a!pagingInfo(startIndex: 1, batchSize: - 1) ), fetchTotalCount: true ).data, a!httpResponse( headers: { a!httpHeader( name: "Content-Type", value: "application/json" ) }, body: a!toJson(value: local!entities) ) )
Discussion posts and replies are publicly visible
Hi, how many records you have in your table? i would have tried execute stored procedure function to get the data, but again it totally depends upon the use case and the amount of the data.
At the moment I think it's about 5000.I need to be able to get the data into PowerBI which I do for other tables using PowerQuery and API endpoints.Thanks.
for 5000 records , you can use queryRecordType but for more than 5000 records you can use function executestoredprocedure(). But you would need to write a procedure in your db.
Thanks but with my limited knowledge, they are not going to be possible.
You will need to implement paging in your API itself. The other system needs to send a startIndex and a batchSize to you so you can fetch the according batch of data and return it.