Record type data not syncing due to error in batches

https://docs.appian.com/suite/help/21.3/configure-record-data-source.html#syncing-in-batches

I think this is a pretty good explanation. You need to make your expression work with the passed batch number.

Quote: "The expression will continue to execute until an empty set or null is returned."

This is what your expression needs to do.

There's two options when performing a sync with a web service:

• Sync all of the data in a single batch
• Sync in batches up to 1000

You mentioned you have 632 records - do you expect that it will always be less than 1000? If so, you don't need to use batches at all. Just remove the rule input from your source expression and the record sync will assume all of your data is returned in a single batch.

If you expect that your data could grow to more than 1000 rows, then you will need to configure batches. The easiest way to do this like Stefan said is to return an empty set or null if there's no data.

So one common approach would be something like this:

a!localVariables(
  local!data: rule!MY_Rule(batchNumber: ri!batchNumber),
  if(
    or(
      isnull(local!data.result.body),
      length(local!data.result.body)=0
    ),
    {},
    local!data.result.body
  )
)

Thank you both! 

I do expect this to be bigger actually, I'm looking to get another connection with like 10k records, on that case, I will need to do the expression you mentioned

Now, my follow up question is, on the batchNumber: ri!batchNumber, that part ri!batchNumber I think it makes reference to the batchNumber rule input which (I think ) needs to be defined, isn't it?

Meaning, I can leave the rule input itself blank, or do I need to configure something there?

I currently have it like this:

On 'Expression' on rule input, should I do something there? 

I think the main thing is that if you use a batch number, you must make sure that you receive different results when different batches are provided. The way it will work is that Appian will iterate over the batch numbers indefinitely starting with 1. When batch 2 is provided, you need to make sure that it either returns the next 1000 results or returns nothing.

Usually this will require both the integration itself to accept a batch and return different results AND the logic I suggested above to return nothing if the batch doesn't have any results.

Without knowing your web service, what I would suggest is to try and test several batches 1, 2, 3. If you are receiving different results for each batch, you should be fine. If you're receiving the same results for each batch, you probably need to make an update to your integration to make a request for a different batch of data.

Hey Peter, thanks for the response, I have been trying to get it to work, I applied the function you mentioned above and modified it to what I understand will work but now I get a different error

I did this on my input rule 

So I added thee function you mentioned and adapted it to my rule, then I used what you put as '{}' and replaced it with the for each loop to get the dictionary made. 

Now, with this, I understand I'm creatingcondition to return an empty value if the batch is empty as well as you mentioned, but I get this error now when trying to use this rule to populate data on my record type

For what the error says, I understand is because I may need to reduce the batch number but not sure how to do it, can you give me an insight on this please?

Ok cool, looks like you're headed in the right direction! Do you know if your source system has a method of sending the data in batches? A lot of integrations will accept a query parameter or something that allows you to determine how many items are returned at a time.

If your source system has this capability, then you'll just need to update your integration to support it by adding the necessary parameters.

If your source system doesn't support batching, then there are potentially ways you could batch the data in Appian after returning the response. For instance, you can use the todatasubset() function to apply a batch number and start index to your final list and you can make it return only 1000 at a time. However, I'd only advise this if it's still a relatively small amount of data (probably up to a few thousand rows). Your best bet is to work with your source system and look for a way to batch the results.

Thanks Peter, I've been checking and my source is BigQuery, but it seems like it is not possible to send information in batches, as you need to connect to it via API and the response ill come as s single response as far as I know. 

So this means I will need to use the todatasubset function, to apply a rule there or make it work on my own parameters correct? Can you provide an example of how that can be done? Also is the todatasubset function, based on a plugin? I'm asking this because I've encountered issues when using this type of functions as it will not let me use the sync option because of the function I'm using to map 

I had a quick glance at the BigQuery documentation. I saw that it supports pagination. At least when using SQL statements. I did not find out how to do that via the API, but I am pretty sure that it supports that as well.

Yeah I would highly recommend using the paging from the source system if it exists. If not, you might be able to use todatasubset like this to simulate it. However, keep in mind that simulating the paging like this will be very inefficient because you must return all the data every time.

a!localVariables(
  local!result: rule!YOUR_IntegrationHere(),
  todatasubset(
    local!result,
    a!pagingInfo(
      startIndex: 1 + (ri!batchNumber - 1) * 1000,
      batchSize: 1000
    )
  )
)

Basically the way this is working is that it returns 1000 items at a time given a batch number. As the batch number increases, it then gives you the next 1000 items.