Questions about Querying Data from an RDBMS

RE: Questions about Querying Data from an RDBMS

codys — Mon, 05 Mar 2018 17:40:44 GMT

This is consistent with my findings as well. a!queryEntity() appears to add a fixed overhead of about 10-20ms. I reported this to Support but they have not replied with any indication of fixing it. We still use a!queryEntity() everywhere as a best practice for:

(1) Consistency
(2) Traceable dependents
(3) More filter operators
(4) Expressionable filters
(5) Potential to improve performance and overcompensate for the 10-20ms fixed penalty

RE: Questions about Querying Data from an RDBMS

saicharanb — Tue, 03 Jan 2017 15:27:11 GMT

@philib : I am not using SAIL form. I am using Form Designer and PageGrid . Now my problem is to fetch all the records from DB in small batches(not at once), able to retrieve only first batch size. How to retrieve all records in small batches. The below query retrieves only 100 records. But records after 101 are not retrieved. I want to retrieve all records i.e., 1-100,101-200,201-300 so on .. So there will be multiple hits to DB in this case. How to implement this in Form Designer. Please help me on this.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 03 Jan 2017 14:35:21 GMT

Your grid saveinto needs to update the pagingInfo, which should have been initialised in a load() variable. This is clearly shown in the first example here: forum.appian.com/.../recipe_display_data_from_a_record_in_a_grid.html

RE: Questions about Querying Data from an RDBMS

saicharanb — Tue, 03 Jan 2017 12:53:47 GMT

@sparshs: I am using paging grid and Rows per page as 50. My query looks as below :
rule!WFT_tempoSearchQueryPagination(pv!process,pv!subprocess,pv!action,pv!requestorName,pv!company,pv!reference,pv!sapRefNo,pv!fromDate,pv!toDate,pv!monetaryValue,pv!priority,pv!wftRequestID,pv!salesOffice,pv!requestStatus,a!pagingInfo(startIndex: 1,batchSize: 100),loggedinuser())

Can you please help me how to implement a!queryEntity() for the above query rule ?

RE: Questions about Querying Data from an RDBMS

sparshs — Tue, 03 Jan 2017 11:22:51 GMT

No I think you are getting it wrong. Use a paging grid and first use query entity to fetch all the data you want to show on the grid. On the paging grid use the batchsize as 100 and it will work for you. No need to write different rules. you can also find a sail recipe for this. Let me know if you need anything else on this.

RE: Questions about Querying Data from an RDBMS

saicharanb — Tue, 03 Jan 2017 10:18:47 GMT

I have a query rule : =rule!Pagination(pv!wftRequestID,pv!salesOffice,pv!requestStatus,a!pagingInfo(startIndex: 1,batchSize: 100),loggedinuser()) which is returning me 100 records. Now after this, I need to retrieve records 201 to 300. Do I need to implement another rule with =rule!Pagination(pv!wftRequestID,pv!salesOffice,pv!requestStatus,a!pagingInfo(startIndex: 101,batchSize: 200),loggedinuser()) ? If so how do I integrate these two rules ?

RE: Questions about Querying Data from an RDBMS

saicharanb — Mon, 02 Jan 2017 17:20:15 GMT

@sparshs: thanks
So in order to resolve "Memory threshold error", decreasing the "RowsPerPage" in page grid is the only solution?

RE: Questions about Querying Data from an RDBMS

sparshs — Mon, 02 Jan 2017 17:11:47 GMT

saicharanb: I do not think you are doing right. BatchSize is the amount of data you want to retrieve from database , if batchsize is 50 and the total number of records are 100 in Database, it will only fetch first 50 records.

RE: Questions about Querying Data from an RDBMS

saicharanb — Mon, 02 Jan 2017 14:09:55 GMT

Hello, I have similar problem. We have a view which is and it is trying to fetch the data with pagination with 50 records per page. We are using Query rules and we are in version 7.4. But the problem is only first 50 records are displayed, but not all the records. As per my understanding if the total records in DB are 100, Batch Size is 50, then 2 hits are made to DB. Initial batch 50 anf next batch another 50. But in my case only initial 50 records are displayed. Can someone please explain how to retrieve data with batchSize?

RE: Questions about Querying Data from an RDBMS

Chris — Tue, 23 Feb 2016 18:50:27 GMT

To add briefly to the performance testing, I just ran a quick test from the Appian GUI on a basic queryEntity vs Query Rule, on 15 paged records over a DB containing 2.4 million records.

a!queryEntity() - 704 ms

720
762
665
599
774
694
716
725
758
625

Query Rule - 687 ms

838
672
697
720
619
561
790
637
672
660

Over 10 tests the QR wins out by 17 ms.

load(
local!pagingInfo: a!pagingInfo(1,15),
/* 704 ms avg
local!data: a!queryEntity(
entity: cons!ENTITY_TASK_HISTORY,
query: a!query(
pagingInfo: local!pagingInfo
)
),
*/
/* 687 avg */
local!data: rule!chrisTest_qrTaskHistory(local!pagingInfo),

a!textField(
readOnly: true,
value: count(local!data.data)
)
)

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 18:38:08 GMT

I would suggest you test this with a sample table; it's not too much effort to add a table with 100k+ entries and a data store entity - you could then run both query entity and query rules against the table and see how it performs. One thing I would say is if you're actually pulling in 100,000 records, then you're very likely to hit the 1Mb query limit regardless of method.

RE: Questions about Querying Data from an RDBMS

harshav — Tue, 23 Feb 2016 18:29:03 GMT

@philb,@tim,@narasimhaadityac,@sparshs Thank you so much for responding on this but still in the confusion. As suggested by philb in the example it says there is not much difference making with limited data but my report is fetching more than 100K of data so not sure how this would work at that time, also let me explain you how am I querying the data.
1.This is basically written for Report.
2.Rule contains a choose in load function ,based on some selection each rule is called in choose function lets say rule!xyz(parameters,paginginfor(1,25)). now when this is called my page is taking so much time to load. So in this case could I go for query entity or is there any way I can make my page load. Last but not the least, from the comments from philb, "No doubt Appian will be able to provide some more detailed analysis" could someone from Appian give us more detailed Analysis it would be great ..... Once again Thank you so much guys :) !!!

RE: Questions about Querying Data from an RDBMS

narasimhaadityac — Tue, 23 Feb 2016 17:56:03 GMT

@Tim, got it. I had a look at the documentation from and found the usage of the query rules with pagination. https://forum.appian.com/suite/help/7.8/Rules.html#Query_Rule_Paging_Parameter

unfortunately this link is not available in all the versions of the documentation.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 17:29:36 GMT

I should add - my tests were done on a 16.1 machine running MySQL 5.6.19, all running on Ubuntu server.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 17:26:51 GMT

I've done a bit of testing on this and the results are quite interesting.

Attached is a file containing the SQL generated for a query that gets all entries for a specific data entity, using paging info with a start index of 1, a batch size of 5 and a sort on ID ascending. The first set of SQL is that generated for a Query Rule, and the second is that generated for a Query Entity.

In this instance, the Query Rule method generates one extra SQL statement than the Query Entity - 3 statements vs 2. It seems that the Query Rule first obtains a count, then gets the primary keys that it's going to retrieve, then finally selects the data using those keys. The Query Entity does exactly the same query to retrieve the count, but then gets the data and applies the limit in a single statement.

There is, of course, arguably a performance benefit in not doing the second statement, but in this instance it would appear to be negligible; running the second statement directly in MySQL Workbench shows a query time of 0.000 seconds, so there's not even a millisecond of performance increase on the SQL side for this example.

In my opinion the performance benefits displayed don't justify the significantly longer development time required for a Query Entity method - but maybe there are limitations sitting above the MySQL level that cause further delays.

For me, the main observation from this testing is that the Query Rule applies the LIMIT 5; filter in (almost) exactly the same way as Query Entity, and so does not return all the data as suggested above.

Obviously my example is a simple one, but I think it evidences that the differences between the two methods aren't that great for this case. No doubt Appian will be able to provide some more detailed analysis; it would be good to know how complex a query needs to be occurring to realise the potential performance benefits.

QR_vs_QE.sql

RE: Questions about Querying Data from an RDBMS

Tim — Tue, 23 Feb 2016 16:58:15 GMT

@narasimhaadityac where is your understanding from? It's concerning that multiple users have now said that query rules do not obey the pagination parameter. If you provie a pagingInfo parameter the number of results will be limited exactyle like queryEntity()

RE: Questions about Querying Data from an RDBMS

narasimhaadityac — Tue, 23 Feb 2016 16:12:43 GMT

Hi Phil, as per my understanding query rule will try to get all rows and will not apply the pagination, query entity accepts the pagination as a parameter and will fetch only the required batch size from DB, it follows internally Hibernates to form the paginated query and use it to fetch only the required data subset.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 15:58:29 GMT

I think I will do some testing on this. I'm not sure I can accept that query entity performs faster than a query rule for the same number of columns and the same number of rows - especially given that (as I understand it) the access method (hibernate) is exactly the same.

RE: Questions about Querying Data from an RDBMS

sparshs — Tue, 23 Feb 2016 15:02:51 GMT

i agree with you on this but i my previous project we were facing issue like form taking long to load , so we replaced all query rules with query entity's and it kind of decreased overall laging time for a form to load and overall process execution time.Secondly if database table has columns around 10 or above then i think query entity will be more beneficial specially when we are fetching lot of records.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 14:33:27 GMT

@sparshs Have you tested that by turning on SQL trace and viewing the queries that are generated by hibernate? I don't see why it would work that way - fair enough if it does, but I'd be very surprised.

Agreed you can limit the selection in query entity but what are the actual performance benefits? 21ms vs 20ms "performs better" but is ultimately a very poor return on the inevitably increased development time. The "OR" option is, of course, a useful function but is not a performance increase.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 14:29:01 GMT

Just to add - some of this is documented here - forum.appian.com/.../Database_Performance_Best_Practices.html - but "[Use] a query rule against a data store entity when only a subset of the data is needed will still return all the data" is still far too broad; in most use cases this will remove a small percentage of the columns involved and produce a negligible performance increase for a higher development cost.

RE: Questions about Querying Data from an RDBMS

sparshs — Tue, 23 Feb 2016 14:27:53 GMT

Please correct me , if you think i am wrong here. Thanks in advance.

RE: Questions about Querying Data from an RDBMS

sparshs — Tue, 23 Feb 2016 14:26:46 GMT

philb: i am not implying that query rule ignores the paging given to it, but what i state here is that if we are fetching all records suppose 100 from our database and we want to show it as 10 per page , in this case query rule will fetch all 100 and but on our grid we are showing 10 per page. Secondly, when i say that performance wise query entity is better i say it because in query entity you can limit the selection and also you can have OR operator in query entity which we do not get there in query rule. I hope i am making sense here.

RE: Questions about Querying Data from an RDBMS

PhilB — Tue, 23 Feb 2016 14:18:40 GMT

@sharshs - A few comments:

re 1. Do you have metrics to back this up? What you're implying is that a query rule totally ignores the pagingInfo that you give it; I don't believe this is the case.
re 2. When you say "query entity has good performance than query rule" are you saying that query entity is ALWAYS faster? I don't believe this is correct. A comparison of a query rule returning all 100 fields of an entity and a query entity returning only 5 of those fields will, of course, show a performance increase, but I'd suggest that the difference is negligible for similar numbers of fields.
re 4. This simply isn't true; adding an input to a query rule and filtering by that input will limit the results. The performance benefit of query entity is shown when selecting subsets of columns for a given data entity.

I'd be interested in some input from Appian on this; the recommendation of "appian recommends use of query entity over query rule" seems far too broad and doesn't take into account the extra effort an maintenance of query entity over a query rule.

RE: Questions about Querying Data from an RDBMS

sparshs — Tue, 23 Feb 2016 13:59:47 GMT

hope this answers your concerns, if you want more detailed explanation , please let me know.