Configure High Availability in Linux Env (1 Engine 2 Jboss)

Question

Hi everyone, 
 I am setting up with the 4 following servers to achieve High Availability (HA): 
 Host 1 : Application Server 1 with Web Server Host 2 : Application Server 2 with Web Server Host 3 : Search Server, Data Engines and Data Server Host 4: Database 
 
 Please let me know if my understanding is correct for my case: 
 1. I need to install Appian and Jboss for Host 1,2,3 ,but I only need k3 and k4 license for host 3. 
 2. No shared folders have to be built among host 1,2,3 
 3. Should the command script for host 3 with or without --cluster? (i.e. appian/services/bin/start.sh -p <Service Manager Password> -s all --cluster ) 
 4. Where should I go to Start JBoss? In host1 or 2 and 3? 
 5. while setting up the redirect in Apache Web Server in host 1,2 , what value should I use for the ServerName and Server Alias? (i.e. I should set host1 and host2 respectively(e.g. host1 & www. host1.com ; host2 & www.host2.com), or both set as the common domain name to be used in load balancer (e.g. www.testing.com)? 
 
 Any reply would be much appreciated. 
 Thanks in advance

josep · Accepted Answer

Hello Joanneh,

I assume you are installing the version 18.2? I still need to test this configuration but there are some things I know.

1) The license is needed by the hosts with the engines (which runs using Kdb)
A valid license (k4.lic) is required to run the data server

2) A shared Folder should be there just be careful with the latency(This is something crucial when you distribute the internal data docs.appian.com/.../Internal_Data.html),
to answer specifically your question the share folder is needed. But you need it just for the Host 1 and Host 2 (which share the "App server") for example the plugins or documents (knowledge center).

keep in mind this naming convention
APPIAN_HOME/shared-logs/<machine X>

Take a look at this so you can review which folders to be shared
docs.appian.com/.../High_Availability_and_Distributed_Installations.html

3) without (that is just needed when clustering the engines)

4) on host 1 and 2 but I want to highlight this from the documentation
"The first time you start Appian (either after a fresh installation or update), start only one server and wait for it to complete startup before starting the rest. The remaining application servers may be started simultaneously. For subsequent restarts of Appian on the same version you may start all application servers at the same time."

5. I don't remember this exactly but what I do remember is that when you have multiple configurations on the httpd.conf you need to add the host to the "VirtualHost ip:443>"
normally appian that apache is dedicated

-----
<VirtualHost *:443>
ServerName yourhostname.yourdomain.com
-----
Since you have a load balancer before your Apache both will dispatch the same, so use the same
At this point I recomend the Sticky sessions
docs.appian.com/.../Configuring_Apache_Web_Server_with_JBoss.html

Note: I strongly recommend this link if you haven't read them yet.

community.appian.com/.../high-availability-and-disaster-recovery-configurations
docs.appian.com/.../High_Availability_and_Distributed_Installations.html
docs.appian.com/.../Initial_Setup_for_Advanced_Multi-Server_Configuration.html
docs.appian.com/.../Appian_Engine_Connection_Restrictions.html

I hope this helps
Jose

Mike Cichy · Answer

joanneh To add to the excellent points that josep wrote, a few more questions/comments:

Do you need HA? HA setups are at least an order of magnitude more complicated to stand up, and more importantly, to maintain. Installing HFs, upgrades, OS patches, etc. is going to take longer with more possibility for a mistake. Any issues are going to be much harder to troubleshoot since the requests are distributed, you will have to check all hosts to track down potential issues. If you are having problems maintaining your Appian instance now, HA is going to be 10x harder. The decision to go HA should not be taken lightly.
For ease of installation and maintenance, I would recommend a consolidated setup to start with that is diagrammed here: https://community.appian.com/w/the-appian-playbook/110/configuring-an-environment-for-high-availability#overall-recommendation. You end up with 3 identical hosts - I highly recommend this setup.
Shared-logs setup: as noted, latency can be an issue. I highly recommend cross-mounting the logs between the hosts rather than using central storage where all hosts write logs. I have seen situations where NAS is overwhelmed with the amount of logging going on.
Share files setup: for app server file sharing and for kdb sharing, something like NAS can be used, since the files are usually not read from frequently.

Regarding your follow up questions:

Shared-logs: yes, you have to create a new folder on each host, create a symlink from local /logs to /shared-logs/<this-host> and then create symlinks to /shared-logs/<this other hosts logs>

Simultaneous start-up is not necessary but if you want to, you can create a script to do so.

Each web server needs to point to the same domain name, i.e myhost.atmycompany.com. You configure workers to point to a specific host. docs.appian.com/.../Configuring_Apache_Web_Server_with_JBoss.html

joanneh · Answer

Dear Mike Cichy (mike.cichy), 
 
 I have found the solution! 
 The problem lies in the ERROR: 
 Caused by: java.io.FileNotFoundException: /home/user01/appian/_admin/shared/appian.keystore.lock (Permission denied) 
 indicating that there should be "read and write"permission to the mounted files ( such permission should be set on the TO-mount server, coz it is unable to change the permission of files on the TO-BE-mounted server) 
 
 ref: 
 KB-1103 "Unable to build keystore lock file" error thrown during JBoss startup 
 community.appian.com/.../kb-1103-unable-to-build-keystore-lock-file-error-thrown-during-jboss-startup 
 
 Now my second Jboss is able to start. Thanks for your help!!!