Getting data from API - but API parameter is data from a recordType

Appian has to prevent you from using a query inside the source expression as this could lead to recursive calls and query escalations.

Maybe you go into more details so we have a chance to think about alternative solutions.

Here is the pictorial description of what I am trying to accomplish and the error I am getting.  Hope this provides additional details.

That's tricky.

When you create your source expression doing two calls does not work. But it should work, when you redesign the integration object to do this.

So that integration just takes the batchNumber, then queries the IDs for that batch and then does a second call to fetch the actual data. 

Where / How do I modify the integration?  Would you be able to provide resources where I  could learn / read-up on modifying the integration so that I can query the IDs for the batchNumber?

I tried to put an expression that would use the batchNumber but it gives me an error when I tried to include the batchNumber - tried various domain prefixes (ri!, pv!, etc.)

What error do you see? Please always include this when posting.

I suggest to do that in two steps. First, create an expression that returns that list of IDs. Take the snippet from here

Once that works, including batching etc, create the final integration.

Solving such a larger problem typically requires to split it into pieces, solve these, and then put the pieces together again.

I created a function to get a string of "IDs" (createIDString) that is the input to the synch expression (getFullProductDetails).  That works.  It returns a string of IDs.

So, I tried to use the createIDString as part of the ruleInput expression for fdcIdList.  If I don't provide a domain, it gives me the error below. I also tried 'pv!batchNumber'.

Not sure what other domain I should use, since my understanding is batchNumber is automatically incremented by Appian as part of the looping process?

That batchNumber should be a rule input in the integration object. This is explained here:

https://docs.appian.com/suite/help/23.1/configure-record-data-source.html#syncing-in-batches

Then you can reference it using "ri!batchNumber".

If I understand the documentation correctly, it is the Record Data Source that has the batchNumber as the rule-input.  However, in my case, the Record Data Source has the batchNumber as the rule-input but I have to use the rule-input to obtain the list of IDs to process.  The underlying API/integration doesn't have a 'startIndex' as the parameter.  The API/integration parameter is the list of IDs.

Try to make a step backwards. A service backed record is just using your expression to pass the batch number and get a datasubset returned. This is all your expression has to do. And it does not matter how it is doing it. You can do any number of web service calls to achieve this.

In the record source configuration, you just call your integration passing the batch number as a rule input. Inside that integration object, you call your existing expression to get the list of IDs by passing the batch number.

In the screenshot you write "Need a list of IDs for current batchNumber". What do you mean? Isn'1 that the job of this first expression? Can't you just pass that list of IDs stored in local!ListToProcess into "rule!FST_getFull...something..."?

Try to make a step backwards. A service backed record is just using your expression to pass the batch number and get a datasubset returned. This is all your expression has to do. And it does not matter how it is doing it. You can do any number of web service calls to achieve this.

In the record source configuration, you just call your integration passing the batch number as a rule input. Inside that integration object, you call your existing expression to get the list of IDs by passing the batch number.

In the screenshot you write "Need a list of IDs for current batchNumber". What do you mean? Isn'1 that the job of this first expression? Can't you just pass that list of IDs stored in local!ListToProcess into "rule!FST_getFull...something..."?

I think I have it working - thanks for your help! 

I was not putting the expression in the right place - had rule!input expression mixed up with query parameter expression.

Guess, I was wrong.  The RecordType synch failed :( .  Integration works.  They Record Source expression works - they both returns values as expected.  But configure data sources doesn't work. 

Any suggestions on how to load 10300 records?

Thanks.

This works - returns 25 records from the integration.

Syncing in batches is described in the documentation link I posted above.

Do you use any of the functions mentioned in the error message?

I use a!queryRecordType to retrieve the productIDs from the table with the productIDs.  The documentation link talks about how to use batchNumber to calculate 'pageSize' / 'startIndex' but this API parameter requires a list of productIDs.  I have over 10,000 productIDs in the application database and for each productID, I have to call the API to get current product details.

In the last attempt, I set one of the integration query parameters to retrieve the productIDs based on the batchNumber.  The integration worked fine and the RecordSource expression that calls the integration worked.  However, when I tried to synch the data, it gives the error about not using a!queryRecordType

Can you just try a list of 1000 product IDs (even if some of those don't have corresponding products)? Then you don't have to do any query to find the relevant product IDs from Appian. For instance, just make your batch 1 use values 1...1000, and then try 1001...2000, etc. Even if the first batch only returns a handful of products, it will still work.

We tried your suggestion but we were unable to retrieve all the information we needed.  The productIDs start at 321,358 and goes all the way to 2,512,370.  We were able to extract information up to 2,273,113 but the rest weren't extracted.  Since there is a rate limit on accessing API, we did adjust the rate using "Add rate limit".

In this case, the ProductIDs are numerical, so we could use this strategy, what approach should we use if the ProductIDs were Text?

Thanks.

Can you provide more information about what didn't work? I didn't quite follow which ones were missing.

You're correct that this method would only apply to numerical IDs. I think it would depend on what format you expect the text IDs to be whether you could do a similar strategy or not.

Here is my attempt at explaining the issue:

When you create your synced record and disable any automatic sync. Then you could run a process model that fetches IDs from any source and then calls the sync records smart service. Wouldn't that work in your scenario?

Peter Lewis, what do you think?

Yes I agree with Stefan's suggestion. The other option you could have is to simply create a scheduled process that could do your own method of syncing and store the data in a database table that could then be synced.