more than 5000 records

For what purpose?

I'm having a scenario to associate records for particular department, If I'm searching 5001 record It will not be displayed under the search or pagination itself

I still have a hard time understanding what you try to achieve.

My max number of rows in a grid is 20. Everything else is solved using paging, filtering and search.

Not clear. can you add some more details?

Sarathkumar R Stefan Helzle Yogi Patel Prasanta Paul 

Suppose This is custom filter.

Behind this logic I made a rule where I am getting 5000 record by a!queryrecordtype(),

If I want to search 5001th record, how can I ?

I do not think so that will be not a good user experience. If you are filtering with "A" there is so many countries, if you are searching with "AME" may be there has only few countries or America only. Query the data and configure the filter like that. And why do you need to show much more data in an interface? I think you should not. 

What is your rule looking like?

When implementing a custom picker, the trick is to feed the search string directly into the query, and only return the first few hits. In the Appian example, they define all data in local variables, which will not work in your case.

Yes, as suggested by  Stefan Helzle, you need to implement a backend search using a custom picker. Front-end search will not work. Stefan Helzle  Please create a blog for this if not available.

Why are you using a Custom Picker for your search if you're showing a paging grid with your record list?

Just use a text field for your filter.  When the user presses "search", apply the change to the existing query that's populating the paging grid - the results will be shown instantly.  This should work for even a very large number of records, probably hundreds of thousands at least before it gets unusably slow.  There's no reason whatsoever to be doing this using a Custom Picker.

Why are you using a Custom Picker for your search if you're showing a paging grid with your record list?

Just use a text field for your filter.  When the user presses "search", apply the change to the existing query that's populating the paging grid - the results will be shown instantly.  This should work for even a very large number of records, probably hundreds of thousands at least before it gets unusably slow.  There's no reason whatsoever to be doing this using a Custom Picker.