Question

Record type data not syncing due to error in batches

Forum|Forum|4 years ago
October 25, 2021
9 replies
1 view

Hello!

I am working with an API response from BigQuery, I get the data to be on the record type from my input rule, however when I try to sync, it keeps getting an error as shown below

So, I'm not able to work with this record because of the error, I checked this article https://docs.appian.com/suite/help/21.3/configure-record-data-source.html and it seems like I have to modify the batch number rule but I'm not sure how to do it (like what I have to put in there) or if that is even the issue. My response gives a 632 record only so it is less than 1000, and I assume I shouldn't have to do sync in batches as is less than 1000, but still the error comes up, have you encountered this error or do you guys have any input on how to solve that?

stefanhelzle0001

Brainy

https://docs.appian.com/suite/help/21.3/configure-record-data-source.html#syncing-in-batches

I think this is a pretty good explanation. You need to make your expression work with the passed batch number.

Quote: "The expression will continue to execute until an empty set or null is returned."

This is what your expression needs to do.

peter.lewis

Employee

There's two options when performing a sync with a web service:

Sync all of the data in a single batch
Sync in batches up to 1000

You mentioned you have 632 records - do you expect that it will always be less than 1000? If so, you don't need to use batches at all. Just remove the rule input from your source expression and the record sync will assume all of your data is returned in a single batch.

If you expect that your data could grow to more than 1000 rows, then you will need to configure batches. The easiest way to do this like Stefan said is to return an empty set or null if there's no data.

So one common approach would be something like this:

a!localVariables(
  local!data: rule!MY_Rule(batchNumber: ri!batchNumber),
  if(
    or(
      isnull(local!data.result.body),
      length(local!data.result.body)=0
    ),
    {},
    local!data.result.body
  )
)

rodrigoe0001Author

Thank you both!

I do expect this to be bigger actually, I'm looking to get another connection with like 10k records, on that case, I will need to do the expression you mentioned

Now, my follow up question is, on the batchNumber: ri!batchNumber, that part ri!batchNumber I think it makes reference to the batchNumber rule input which (I think ) needs to be defined, isn't it?

Meaning, I can leave the rule input itself blank, or do I need to configure something there?

I currently have it like this:

On 'Expression' on rule input, should I do something there?

peter.lewis

Employee

I think the main thing is that if you use a batch number, you must make sure that you receive different results when different batches are provided. The way it will work is that Appian will iterate over the batch numbers indefinitely starting with 1. When batch 2 is provided, you need to make sure that it either returns the next 1000 results or returns nothing.

Usually this will require both the integration itself to accept a batch and return different results AND the logic I suggested above to return nothing if the batch doesn't have any results.

Without knowing your web service, what I would suggest is to try and test several batches 1, 2, 3. If you are receiving different results for each batch, you should be fine. If you're receiving the same results for each batch, you probably need to make an update to your integration to make a request for a different batch of data.

Sign up

Login with SSO

Login to the community

Login with SSO