how to fetch the data of query entity in batches?

RE: how to fetch the data of query entity in batches?

Stefan Helzle — Wed, 15 Sep 2021 07:04:33 GMT

Again, what is the purpose of fetching ALL data into memory?

RE: how to fetch the data of query entity in batches?

Klaus34 — Wed, 15 Sep 2021 06:21:03 GMT

i have provided the batch size 5405

RE: how to fetch the data of query entity in batches?

Klaus34 — Wed, 15 Sep 2021 06:20:20 GMT

Yea i increased it and it works but then it doesn't fetch the complete count of data

RE: how to fetch the data of query entity in batches?

Malathi — Wed, 15 Sep 2021 05:57:06 GMT

Seems you are fetching whole data by giving -1 batch size(though you have few columns). Can you try increasing the batch size gradually to find the threshold point?

RE: how to fetch the data of query entity in batches?

Klaus34 — Wed, 15 Sep 2021 05:41:56 GMT

a!queryEntity_18r3(
entity: cons!TEST_ENTITY_CONTENT_VW,
query: a!query(
selection:a!querySelection(
columns: {
a!queryColumn(
field: "contentId"
),
a!queryColumn(
field:"contentAreaId"
),
a!queryColumn(
field:"categoryId"
),
a!queryColumn(
field:"topicId"
),
a!queryColumn(
field:"UnitId"
),

a!queryColumn(
field:"Owner"
),
a!queryColumn(
field:"docName"
),

a!queryColumn(
field:"createdDt"
),
a!queryColumn(
field:"docDescription"
),
a!queryColumn(
field:"docLink"
)
}
)

,
logicalExpression: a!queryLogicalExpression(
operator: "AND",
filters: {
a!queryFilter(
field: "contentId",
operator: "not null"
),
if(
or(isnull(ri!contentAreaIds), rule!APN_isEmpty(ri!contentAreaIds)),
{},
a!queryFilter(
field: "contentAreaId",
operator: "in",
value: ri!contentAreaIds
)
),
if(
or(isnull(ri!categoryIds), rule!APN_isEmpty(ri!categoryIds)),
{},
a!queryFilter(
field: "categoryId",
operator: "in",
value: ri!categoryIds
)
),
if(
or(isnull(ri!topicIds), rule!APN_isEmpty(ri!topicIds)),
{},
a!queryFilter(
field: "topicId",
operator: "in",
value: ri!topicIds
)
),
if(
rule!APN_isBlank(
ri!buId
),
{},
a!queryFilter(
field: "UnitId",
operator: "=",
value: ri!buId
)
),
if(
rule!APN_isBlank(
ri!Owner
),
{},
a!queryFilter(
field: "Owner",
operator: "=",
value: ri!Owner
)
),
if(
rule!APN_isBlank(
ri!title
),
{},
a!queryFilter(
field: "docName",
operator: "includes",
value: ri!title
)
),
if(
or(rule!APN_isBlank(ri!startDate), rule!APN_isBlank(ri!endDate)),
{},
{
a!queryFilter(
field: "createdDt",
operator: ">=",
value: todatetime(ri!startDate)
),
a!queryFilter(
field: "createdDt",
operator: "<=",
value: todatetime(ri!endDate)
)
}
)
}
),
pagingInfo: a!pagingInfo(startIndex:1, batchSize:-1)
)
),

i have applied the selection also but still the issue persists.

RE: how to fetch the data of query entity in batches?

Klaus34 — Tue, 14 Sep 2021 12:57:40 GMT

can u please share the code for reference.

RE: how to fetch the data of query entity in batches?

Stefan Helzle — Tue, 14 Sep 2021 12:55:22 GMT

For what purpose?

RE: how to fetch the data of query entity in batches?

vimalkumars — Tue, 14 Sep 2021 12:50:36 GMT

It is not recommended to query more than 1MB of data to avoid memory issues. If you still need it, you can query them in batches. Like if there are 50k records, query them 5 times with a 10K batch size.

Also, if you can share your use-case, community members can share some best practices around it.

RE: how to fetch the data of query entity in batches?

Klaus34 — Tue, 14 Sep 2021 12:47:03 GMT

Actually i need all the data to be fetched.

RE: how to fetch the data of query entity in batches?

vimalkumars — Tue, 14 Sep 2021 12:42:33 GMT

Your query call has reached the memory threshold limit ~ 1MB.

docs.appian.com/.../Post-Install_Configurations.html

This usually happens when you try to call a query entity with batch size set as -1 and you are trying to pull more records from the database. Check if you can reduce your output size by applying filter or batch-size