Skip to main content
adrians0003
March 29, 2022
Question

Looping API call

  • March 29, 2022
  • 5 replies
  • 0 views

Hi. I'm a novice so as much help as possible would be appreciated!

I have managed to work out how to extract the data I want via an API, however, there's too much of it and I get a memory error when setting the batch size to -1. Smaller batches work fine, so:

1. How do I loop the code below so that it returns smaller batches (1000)?
2. How do I merge these batches back into one file as would happen in a single call to the endpoint?

Thanks in advance,

Adrian

a!localVariables(
  local!entities: a!queryEntity(
    entity: cons!TDW_REVIEWERS_ANSWER_DS,
    query: a!query(
      selection: a!querySelection(
        columns: {
          a!queryColumn(field: "respondedDate"),
          a!queryColumn(field: "schemeId"),
          a!queryColumn(field: "answer"),
          a!queryColumn(field: "comment"),
          a!queryColumn(field: "reviewer")
        }
      ),
      logicalExpression: a!queryLogicalExpression(
        operator: "AND",
        filters: {
          a!queryFilter(
            field: "schemeId",
            operator: ">=",
            value: 70000
          ),
          a!queryFilter(field: "comment", operator: ">", value: " "),
          a!queryFilter(
            field: "comment",
            operator: "<>",
            value: "Test"
          )
        },
        ignoreFiltersWithEmptyValues: true
      ),
      pagingInfo: a!pagingInfo(startIndex: 1, batchSize: - 1)
    ),
    fetchTotalCount: true
  ).data,
  a!httpResponse(
    headers: {
      a!httpHeader(
        name: "Content-Type",
        value: "application/json"
      )
    },
    body: a!toJson(value: local!entities)
  )
)

5 replies

gopalk2865
Participating Frequently
March 29, 2022

Hi, how many records you have in your table? i would have tried execute stored procedure function to get the data, but again it totally depends upon the use case and the amount of the data.

adrians0003
March 29, 2022

At the moment I think it's about 5000.
I need to be able to get the data into PowerBI which I do for other tables using PowerQuery and API endpoints.

Thanks.

gopalk2865
Participating Frequently
March 29, 2022

for 5000 records , you can use queryRecordType but for more than 5000 records you can use function executestoredprocedure(). But you would need to write a procedure in your db.