I have an VM running Appian 7.2, on Windows Server 2008 R2, Windows SQL 2008 R2

What version of the application server are you running on?

We are using JBOSS 7.1.1 in Development Environment (Standard Appian 7.2 Installation). It will be JBOSS EAP 6.1.0 in Production.

One finding is the JAVA.exe grows from 1.5 GB (initialization after server startup) to around 2.5GB. Another finding is the performance drop while the free memory close to zero.

Will it be the issue of Cache Memory in Windows 2008 R2?

Note that we recommend using JBoss EAP 6.1 for Production. As noted in our documentation: forum.appian.com/.../System_Requirements Jboss AS 7.1.1 performed insufficiently for production-level usage and scaling.

I agree with Diana that you shouldn't be using JBOSS 7.1.1, notice how that version is not even supported anymore for Appian 7.3 for what Diana mentions in her post.

While you deploy a different application server and assuming a restart of the application server (i.e. JBoss) works around the issue for a few days you may want to review your standalone.bat and search for the word "TieredCompilation", if you find it then it means the environment doesn't have the hotfixes properly installed which is critical performance wise forum.appian.com/.../Hotfixes

Eduardo, Thanks for your input. I also guess this should be memory requirement issue at this moment. In my testing server, it is only 8.8GB but we have Appian Database Engine (around 3.5GB RAM), JBOSS Server is using 2.5GB, base component of the Windows 2008 R2 (1GB) so the total is 7GB. There is only 1.8 GB remaining and this may not be enough for Windows 2008 R2 running.

Diana,
Thanks for your input about the memory requirement and it triggers me to review the capacity of the server.

I applied the Hotfix B for version 7.2 today and it seems that the performance was improved a lot. I scheduled large process periodically to test the server performance.

By using same configuration with Hotfix B for version 7.2, I scheduled the process that caused performance issue every 15 minutes for 20 hours and the performance is still good. Without the Appian 7.2 Hotfix B, the system will be very slow after running the same processes around 10 times. I guess the Hotfix B fix the performance issue.