Web API Limitations

Certified Senior Developer
What's the best way to 'GET' a large amount of data via Web API? I initially created Web APIs that used the a!queryEntity function to pull incremental changes from several tables, but have been asked to revise them to pull all records from these tables. There are several tables that have ~ 1000 records, which is not a problem (yet), but others are quite large. In the testing I've done so far on one table, I am able to retrieve 15,000 records but any more returns an error. Is there a way to include a looping function in the API to retrieve multiple extracts if the number of records exceeds a certain count?

OriginalPostID-267239

  Discussion posts and replies are publicly visible

Parents
  • @judym598, I think I have a solution based on idea of @reginaldm377.. Have one webapi which will just start a process(using a!startProcess) - this process instance should get the data in batches. Lets call this webAPI1. webAPI1 should return the processId to the caller.

    The process started should execute and get the data in batches (by looping) using idea of @reginaldm377. And it should be stored in some CDT (list of CDT).

    Then, have another webAPI - lets call this webAPI2. This webAPI2 will take a processInstance as input - the caller will pass the process instance which it got from response of webAPI1.

    webAPI2 will just query that process instance (which will be passed as query param by the caller). It will check if the status of the process is complete, if it is complete it will get the data for the value of the list of CDT which it contains. If the status is complete, it means that the process has completed all the loops and extracted all the data in that multiple CDT.
    You can use the a!queryProcessAnalytics function to check the status of the process.
    To get the value of the multiple CDT PV, there is some plugin function which can allow you to get the value of PV when you pass the processInstanceId and the name of the PV.
    The webAPI2 can then return this data. webAPI2 should return the data only if the status is "Complete" for the passed process instance - otherwise it will return null.
    If the caller of webAPI2 gets a null, then its obvious that the looping has not yet completed, and the caller needs to call webAPI2 again after some time.

    This sounds little complicated, but it should solve the looping problem in webAPI
Reply
  • @judym598, I think I have a solution based on idea of @reginaldm377.. Have one webapi which will just start a process(using a!startProcess) - this process instance should get the data in batches. Lets call this webAPI1. webAPI1 should return the processId to the caller.

    The process started should execute and get the data in batches (by looping) using idea of @reginaldm377. And it should be stored in some CDT (list of CDT).

    Then, have another webAPI - lets call this webAPI2. This webAPI2 will take a processInstance as input - the caller will pass the process instance which it got from response of webAPI1.

    webAPI2 will just query that process instance (which will be passed as query param by the caller). It will check if the status of the process is complete, if it is complete it will get the data for the value of the list of CDT which it contains. If the status is complete, it means that the process has completed all the loops and extracted all the data in that multiple CDT.
    You can use the a!queryProcessAnalytics function to check the status of the process.
    To get the value of the multiple CDT PV, there is some plugin function which can allow you to get the value of PV when you pass the processInstanceId and the name of the PV.
    The webAPI2 can then return this data. webAPI2 should return the data only if the status is "Complete" for the passed process instance - otherwise it will return null.
    If the caller of webAPI2 gets a null, then its obvious that the looping has not yet completed, and the caller needs to call webAPI2 again after some time.

    This sounds little complicated, but it should solve the looping problem in webAPI
Children
No Data