Web service integration with specific paging

RE: Web service integration with specific paging

Habibija Safet — Tue, 03 Sep 2024 14:42:53 GMT

Thx for prompt response. I was not aware of this plugin. Need to check.

RE: Web service integration with specific paging

Stefan Helzle — Tue, 03 Sep 2024 14:24:00 GMT

Hm ... you could try to use the textcache plugin to "remember" that value across iterations. I never tried to use it for such a scenario.

RE: Web service integration with specific paging

Habibija Safet — Tue, 03 Sep 2024 14:05:45 GMT

Dear Stefan,

I'm afraid I didn't explain it well enough - offset in AirTable REST API is not a numeric value but a string (it looks like an internal rowid, i.e. "itrh6WQn9QiECOUcx/rec2ll6BYnfG79HJk". It cannot be calculated - it is returned as part of the WebService response and has to be send in the next call as is (to obtain the next page). To detect the last page, we can check if the offset in response is null (as I already explained, stop condition is not an issue here).

The ultimate issue here is how to store and reuse the offset when record type sync data in batches.

Sure, we could probably have an auxiliary table where we could store the offset in one iteration and read it in the next one, but I hoped that there is a more elegant (=simple) solution.

Regards,

RE: Web service integration with specific paging

Stefan Helzle — Mon, 02 Sep 2024 14:44:29 GMT

Please check the following assumption: "Airtable always returns as many rows a specified in the pageSize parameter, except for the last call".

If this is true, then you can just count the offset yourself like in the example in the documentation.

https://docs.appian.com/suite/help/24.3/configure-record-data-source.html#create-a-record-data-source

RE: Web service integration with specific paging

Habibija Safet — Mon, 02 Sep 2024 14:33:18 GMT

My intention is to make the record type synced, but due the paging and the stopping mechanism I'm not able to do so.

My idea was:

1. Create connected system for AirTable cloud - successfully completed

2. Create integration object based on the connected system where I would

* pick the offset from the previous batch and send it as a query parameter to the web service (to get the next page of records) - this is the tricky part I'm not able to solve as apparently there is no "session" where the offset could be stored (during a record type full resync).

* check for existence of the offset attribute in Web service response - if not found (that means that we have reached the last page), I would return null; else the result from the WS is returned - that would solve the stopping mechanism

3. Create a record type based on the integration object (easy)

The more I look at this, I realize that it won't work as I have no mean to pass the offset from one iteration to the next one.

I'm trying now a another approach - creating a process model that is timer based and that will invoke the integration object, fetch the data and send it to a record type using the Write Records Smart Service (in a loop of course). That way, I should be able to store the offset and reuse it in the next iteration and check of exit conditions. I expect that the final outcome should be the same - a record type synced with the remote AirTable table.

Regards,

RE: Web service integration with specific paging

Stefan Helzle — Mon, 02 Sep 2024 11:44:52 GMT

Do you make this a synced record? The mechanism to stop syncing is different. It relies on the last batch returning zero items. When synced, you have all the data available in a way more flexible way.

If that does not fit your use case, can you describe it in more detail?