Hi Community,
I need to create a record type based on an external Airtable table. AirTable provides a REST API for querying the data, so I decided to create a Web Service based record type.
I already created a connected system and confirm that is returning data.
The issue here is the paging (due to the expected number of records) - Airtable does not accept a page number as (query) parameter, only the page Size.
In each response, beside the record list, there is an attribute called "offset" that should be used in next request (send as a query parameter) to get the next page (and so on).
When we get the last page, the offset attribute will be empty. Additionally, there is no way to know (in advance) the number of records in the Airtable.
As the Appian paging is based on batch number, I do not have any idea how to proceed.
Hope that somebody knows a way how to handle this type of paging.
Thanks in advance
Discussion posts and replies are publicly visible
Do you make this a synced record? The mechanism to stop syncing is different. It relies on the last batch returning zero items. When synced, you have all the data available in a way more flexible way.
If that does not fit your use case, can you describe it in more detail?
My intention is to make the record type synced, but due the paging and the stopping mechanism I'm not able to do so.
My idea was:
1. Create connected system for AirTable cloud - successfully completed
2. Create integration object based on the connected system where I would
* pick the offset from the previous batch and send it as a query parameter to the web service (to get the next page of records) - this is the tricky part I'm not able to solve as apparently there is no "session" where the offset could be stored (during a record type full resync).
* check for existence of the offset attribute in Web service response - if not found (that means that we have reached the last page), I would return null; else the result from the WS is returned - that would solve the stopping mechanism
3. Create a record type based on the integration object (easy)
The more I look at this, I realize that it won't work as I have no mean to pass the offset from one iteration to the next one.
I'm trying now a another approach - creating a process model that is timer based and that will invoke the integration object, fetch the data and send it to a record type using the Write Records Smart Service (in a loop of course). That way, I should be able to store the offset and reuse it in the next iteration and check of exit conditions. I expect that the final outcome should be the same - a record type synced with the remote AirTable table.
Regards,
Please check the following assumption: "Airtable always returns as many rows a specified in the pageSize parameter, except for the last call".
If this is true, then you can just count the offset yourself like in the example in the documentation.
https://docs.appian.com/suite/help/24.3/configure-record-data-source.html#create-a-record-data-source
Dear Stefan,
I'm afraid I didn't explain it well enough - offset in AirTable REST API is not a numeric value but a string (it looks like an internal rowid, i.e. "itrh6WQn9QiECOUcx/rec2ll6BYnfG79HJk". It cannot be calculated - it is returned as part of the WebService response and has to be send in the next call as is (to obtain the next page). To detect the last page, we can check if the offset in response is null (as I already explained, stop condition is not an issue here).
The ultimate issue here is how to store and reuse the offset when record type sync data in batches.
Sure, we could probably have an auxiliary table where we could store the offset in one iteration and read it in the next one, but I hoped that there is a more elegant (=simple) solution.
Hm ... you could try to use the textcache plugin to "remember" that value across iterations. I never tried to use it for such a scenario.
Thx for prompt response. I was not aware of this plugin. Need to check.