What actually increases the performance of an Appian environment. Is it increasi

...deas to fix this issue are highly welcome

I'd recommend debugging as follows:
a) Reproduce on a non-production environment
b) Remove customizations and custom java code
c) Located offending custom code or plug-in through testing
If the above isn't followed, you will likely continue to have problems. Need to get to the source of the problem and missing controls that introduced the issue.

Follow the previous advice to get to a stable, testable state.

Then use Appian's performance logs to identify whether there's a bottleneck in your app server or your engines (or both): forum.appian.com/.../Logging.html

With that knowledge you can determine where you may want to scale your architecture: forum.appian.com/.../Scaling_Appian.html

If the platform itself is running slow, perhaps you could identify issues with the applications that are causing the slowdown. Things like Data Management of process models (Deleting after 1-3 days), thinning your process instances by making use of small, more modular process models with asynchronous sub-processes and termination end nodes, deleting any un-needed large documents in the knowledge centers, avoiding constant import/exports of large applications, etc might help the environment. Fixing the root cause is always better than putting on a Band-Aid if possible.

Unfortunately this application has large documents...some of them several MB stored inside within Appian. Process Models are huge with lot of nodes and most of the general common functionalities have been broken down to sub-processes which are also huge....They also use custom built-in plugins for integrating to external systems using web services.....this application actually talks with more than 10 external systems that have additional security layer built in and this application itself is on single-sign on as well.....but the good thing is that not that many people are using it at the same time(may be 200).....so I do not know where the bottleneck is that is causing the root cause of slowness as there are so many parameters........i don't think it is possible for us to create a non-production environment and simulate this on something like jmeter because of so many layers of authentication and bridges that has to be passed through.....but will it be helpful in any manner if we scale up the process-exec or some other engines to separate servers to increase the performance in any manner....if jboss has to be scaled i have to give solid evidence to my manager and director that we need to scale up more which is something that they don't want as they recently spend like 33k on red hat licenses....