what are the things that we need to consider for better application performance
Discussion posts and replies are publicly visible
This is a very broad question. There are a lot of small things you can do in different parts of your application to improve performance such as how often you query data on an interface or how much looping you might do in a process. Do you have a more specific question? Appian already takes care of most performance concerns under the hood so that developers don't need to think about it.
Agreed with Danny and this is a broader subject. However, please refer to the following URL's for a better understanding of performance.
Interfaces: https://docs.appian.com/suite/help/21.1/SAIL_Performance.html
Database: https://community.appian.com/w/the-appian-playbook/197/database-performance-best-practices
Appian recommendations: https://docs.appian.com/suite/help/21.1/appian-recommendations.html
Performance monitoring: https://docs.appian.com/suite/help/21.1/Logging.html#monitoring-performance-and-usage
Which things are the slowest?
That's really the key. There's an old adage you'll find repeated over and over: "Premature optimization is the root of all evil."
If you have 2 nodes in the same PM that run for 10 seconds and 120 seconds, I don't care if you shave half the time of the 10 second one, not really. See if you can shave off even 10 % from the big one, and it will be better than making the small one completely free.
Determine how far upstream and how far downstream the problems are. If it involves a DB query, compare the time the DB portion takes to the time the Appian portion takes. If they're roughly the same, you've got a database problem. If the DB is fast in comparison, you've got an application problem. Always gather as little data as possible, filter as many rows as possible, return as few columns as possible. Learn explain plans.
Run your health check and within reason do what it says. Always remember to think about the reason why a best practice rather than blindly doing it. The first is wisdom; the second is cargo-cult.