OS: Windows Server 2008 R2 Enterprise w/ SP1 System Type: 64 bit Insta

The Appian Configuration Manager (ACM) is an additional tool when setting up Appian. However, this still requires going over the production documentation.
The required configurations for custom.properties can be found here:
forum.appian.com/.../Post-Install_Configurations
You can always search the Documentation for information about any specific properties that are listed in custom.properties.

The ACM uses a *.properties file for configuring what custom.properties would be for each of your environments in an installation NOT using the ACM. Many of the properties are the same as custom.properties, and it also includes configurations for standalone.xml and appian-ds.xml.
standalone.xml per forum.appian.com/.../Configuring_JBoss
appian-ds.xml per forum.appian.com/.../Configuring_Relational_Databases

You'll need to configure JBoss if you are NOT using the ACM. Otherwise, you would define the location of JBoss (the exploded directory or the zip file) in *.properties at the appropriate value.

Have you defined an ANT_HOME system variable that points to the <APPIAN_HOME>\          hirdpartyapps\\apache-ant?

Thanks for the response, I've gone through the documentation for configuring JBoss and configuring relational databases. They weren't any help. Yes ANT_HOME system variable is set.

If you're doing a regular installation per the documentation, you won't need to use the ACM.

I thought the ACM did a regular install, but took care of configuring everything for you? What is ACM supposed to do?

I don't think the steps outlined are very well delineated for either method of installation. It shouldn't be this difficult to install software.

In both ways you need to configure some files, the ACM is a standardized deployment process for Appian. Since you are having problems using the ACM installation I recommend you to make a regular installation following step by step the following installation guide:
forum.appian.com/.../Windows_Installation_Guide_for_JBoss
When you get to the step of “Configuring the Application Server” in the guide, choose Configuring JBOSS. After configuring JBOSS you will need to configure the database, which is the next step of the installation guide.