Configure High Availability in Linux Env (1 Engine 2 Jboss)

Hi everyone,

I am setting up with the 4 following servers to achieve High Availability (HA):

Host 1 :  Application Server 1 with Web Server
Host 2 :  Application Server 2 with Web Server
Host 3 : Search Server, Data Engines and Data Server
Host 4:  Database

Please let me know if my understanding is correct for my case:

1. I need to install Appian and Jboss for Host 1,2,3 ,but I only need k3 and k4 license for host 3.

2. No shared folders have to be built among host 1,2,3

3. Should the command script for host 3 with or without --cluster? (i.e. appian/services/bin/start.sh -p <Service Manager Password> -s all --cluster )

4. Where should I go to Start JBoss? In host1 or 2 and 3?

5. while setting up the redirect in Apache Web Server in host 1,2 , what value should I use for the ServerName and Server Alias? (i.e. I should set host1 and host2 respectively(e.g. host1 & www. host1.com ; host2 & www.host2.com), or both set as the common domain name to be used in load balancer (e.g. www.testing.com)?

 

Any reply would be much appreciated.

Thanks in advance

  Discussion posts and replies are publicly visible

Parents
  • Hello Joanneh,

    I assume you are installing the version 18.2? I still need to test this configuration but there are some things I know.

    1) The license is needed by the hosts with the engines (which runs using Kdb)
    A valid license (k4.lic) is required to run the data server

    2) A shared Folder should be there just be careful with the latency(This is something crucial when you distribute the internal data docs.appian.com/.../Internal_Data.html),
    to answer specifically your question the share folder is needed. But you need it just for the Host 1 and Host 2 (which share the "App server") for example the plugins or documents (knowledge center).

    keep in mind this naming convention
    APPIAN_HOME/shared-logs/<machine X>

    Take a look at this so you can review which folders to be shared
    docs.appian.com/.../High_Availability_and_Distributed_Installations.html

    3) without (that is just needed when clustering the engines)

    4) on host 1 and 2 but I want to highlight this from the documentation
    "The first time you start Appian (either after a fresh installation or update), start only one server and wait for it to complete startup before starting the rest. The remaining application servers may be started simultaneously. For subsequent restarts of Appian on the same version you may start all application servers at the same time."

    5. I don't remember this exactly but what I do remember is that when you have multiple configurations on the httpd.conf you need to add the host to the "VirtualHost ip:443>"
    normally appian that apache is dedicated

    -----
    <VirtualHost *:443>
    ServerName yourhostname.yourdomain.com
    -----
    Since you have a load balancer before your Apache both will dispatch the same, so use the same
    At this point I recomend the Sticky sessions
    docs.appian.com/.../Configuring_Apache_Web_Server_with_JBoss.html


    Note: I strongly recommend this link if you haven't read them yet.

    community.appian.com/.../high-availability-and-disaster-recovery-configurations
    docs.appian.com/.../High_Availability_and_Distributed_Installations.html
    docs.appian.com/.../Initial_Setup_for_Advanced_Multi-Server_Configuration.html
    docs.appian.com/.../Appian_Engine_Connection_Restrictions.html


    I hope this helps
    Jose
  • Dear

    A million thanks!!

    Please allow me to ask a few more follow up questions and clarify if my understanding is correct:
    -- point (2)
    a. Shared Logs
    Do you mean all I need to do is to open up the new directory in host 1&2 respectively:
    APPIAN_HOME/shared-logs/host1
    APPIAN_HOME/shared-logs/host2

    then add the link between them? Do you have any idea how to link them in Linux?

    -- point (4)
    Do you know how to start them "simultaneously"? Do I have to config a special script?

    -- point (5)
    Do you mean I should set the 443 in host1 and host2 respectively(e.g host1 & www. host1.com ; host2 & www.host2.com)?


    Thanks again for your kindly help :)
  • +1
    Appian Employee
    in reply to joanneh

    joanneh To add to the excellent points that josep wrote, a few more questions/comments:

    1. Do you need HA? HA setups are at least an order of magnitude more complicated to stand up, and more importantly, to maintain. Installing HFs, upgrades, OS patches, etc. is going to take longer with more possibility for a mistake. Any issues are going to be much harder to troubleshoot since the requests are distributed, you will have to check all hosts to track down potential issues. If you are having problems maintaining your Appian instance now, HA is going to be 10x harder. The decision to go HA should not be taken lightly.
    2. For ease of installation and maintenance, I would recommend a consolidated setup to start with that is diagrammed here: https://community.appian.com/w/the-appian-playbook/110/configuring-an-environment-for-high-availability#overall-recommendation. You end up with 3 identical hosts - I highly recommend this setup.
    3. Shared-logs setup: as  noted, latency can be an issue. I highly recommend cross-mounting the logs between the hosts rather than using central storage where all hosts write logs. I have seen situations where NAS is overwhelmed with the amount of logging going on.
    4. Share files setup: for app server file sharing and for kdb sharing, something like NAS can be used, since the files are usually not read from frequently.

    Regarding your follow up questions:

    Shared-logs: yes, you have to create a new folder on each host, create a symlink from local /logs to /shared-logs/<this-host> and then create symlinks to /shared-logs/<this other hosts logs>

    Simultaneous start-up is not necessary but if you want to, you can create a script to do so.

    Each web server needs to point to the same domain name, i.e myhost.atmycompany.com. You configure workers to point to a specific host. docs.appian.com/.../Configuring_Apache_Web_Server_with_JBoss.html

  • Dear  and 

    Thanks a lot! My concepts are clearer and just a small futher question

    Regarding to the share-logs part, please let me know which of the following approch do you actually means:

    [A]
    e.g. currently running in host1
    1. I create the all 3 directories in all 3 hosts
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    2. I create the symlink for host1's /<APPIAN_HOME>/logs in host1's /<APPIAN_HOME>/shared-logs/host1
    3. I mount host1's /<APPIAN_HOME>/logs to other hosts' /<APPIAN_HOME>/shared-logs/host1

    4.repeat step 1-3 for other hosts

    ==> so that at the end, the 3 directories in all 3 hosts
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    are actually pointing only to the different host's <APPIAN_HOME>/logs folder, either through the symlink to the local host's <APPIAN_HOME>/logs, or through the mount folders from other host.

    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    [B]
    e.g. currently running in host1
    1. I create the /<APPIAN_HOME>/shared-logs/host1 directory
    2. I create the symlink for /<APPIAN_HOME>/logs
    --> thus the structure now is a /<APPIAN_HOME>/shared-logs/host1 directory containing a link to the local machine's (i.e. "host1" here but would be other value for other host) /<APPIAN_HOME>/logs.
    3. I mount host1's /<APPIAN_HOME>/shared-logs/host1 directory to other hosts' <APPIAN_HOME>/shared-logs directory.
    4. repeat step 1-3 for other hosts

    ==> So at the end, for each host's <APPIAN_HOME>/shared-logs directory, they are containing 3 directories,
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    however, all the folders are actually pointing only to the localhost's <APPIAN_HOME>/logs folder at that moment (e.g in the host2 env, when I open up the folder link in /<APPIAN_HOME>/shared-logs/host1, the folder I end up seeing would be host2's /<APPIAN_HOME>/logs. This is because the Properties of the <APPIAN_HOME>/logs folders are saying that "Link target: /<APPIAN_HOME>/logs")

Reply
  • Dear  and 

    Thanks a lot! My concepts are clearer and just a small futher question

    Regarding to the share-logs part, please let me know which of the following approch do you actually means:

    [A]
    e.g. currently running in host1
    1. I create the all 3 directories in all 3 hosts
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    2. I create the symlink for host1's /<APPIAN_HOME>/logs in host1's /<APPIAN_HOME>/shared-logs/host1
    3. I mount host1's /<APPIAN_HOME>/logs to other hosts' /<APPIAN_HOME>/shared-logs/host1

    4.repeat step 1-3 for other hosts

    ==> so that at the end, the 3 directories in all 3 hosts
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    are actually pointing only to the different host's <APPIAN_HOME>/logs folder, either through the symlink to the local host's <APPIAN_HOME>/logs, or through the mount folders from other host.

    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    [B]
    e.g. currently running in host1
    1. I create the /<APPIAN_HOME>/shared-logs/host1 directory
    2. I create the symlink for /<APPIAN_HOME>/logs
    --> thus the structure now is a /<APPIAN_HOME>/shared-logs/host1 directory containing a link to the local machine's (i.e. "host1" here but would be other value for other host) /<APPIAN_HOME>/logs.
    3. I mount host1's /<APPIAN_HOME>/shared-logs/host1 directory to other hosts' <APPIAN_HOME>/shared-logs directory.
    4. repeat step 1-3 for other hosts

    ==> So at the end, for each host's <APPIAN_HOME>/shared-logs directory, they are containing 3 directories,
    /<APPIAN_HOME>/shared-logs/host1
    /<APPIAN_HOME>/shared-logs/host2
    /<APPIAN_HOME>/shared-logs/host3

    however, all the folders are actually pointing only to the localhost's <APPIAN_HOME>/logs folder at that moment (e.g in the host2 env, when I open up the folder link in /<APPIAN_HOME>/shared-logs/host1, the folder I end up seeing would be host2's /<APPIAN_HOME>/logs. This is because the Properties of the <APPIAN_HOME>/logs folders are saying that "Link target: /<APPIAN_HOME>/logs")

Children
  • Dear Mike Cichy (mike.cichy)  and josep ,

    Besides that, according to your comment:

    -- Take a look at this so you can review which folders to be shared

    What I understand from the line “The following directories must be shared across all servers that run that component.“ is that, for example, in my case, if there are two Application servers, the files that need to be shared just between host1 and host2 are:

    Component Name Folder Name

    Application Server APPIAN_HOME/_admin/accdocs1/

    Application Server APPIAN_HOME/_admin/accdocs2/

    Application Server APPIAN_HOME/_admin/accdocs3/

    Application Server APPIAN_HOME/server/archived-process/

    Application Server APPIAN_HOME/_admin/search/

    Application Server APPIAN_HOME/server/msg/

    Application Server APPIAN_HOME/_admin/mini/

    Application Server APPIAN_HOME/_admin/models/

    Application Server APPIAN_HOME/_admin/process_notes/

    Application Server APPIAN_HOME/_admin/shared/

     

    Is my understanding correct, or there are indeed have to be shared among all servers? 

     

    Moreover, some directories (such as archived-process,search,msg)  might be absent. Should I create one by my own, or should I skip it directly during mounting?

    (e.g. as you may see below, there would be even less folders for an app server which did not run before. If the file sharing step is to be completed before any app server runs, there would be more folder missing from the list above and could not be mount)

    host1: App server that already run to set up

     

    host2: new App server to be added

  • 0
    Appian Employee
    in reply to joanneh

    At this point, I highly recommend reaching out to Product Support with your questions.
    Regarding mounting, you want to use option A. host1/appian_home/shared-logs/host2 should be pointing to host2/appian_home/logs

    Regarding the files shared b/w the app servers, since they need to be truly *shared* they need to point to the same location. You can either designate one of the servers as storage and then point the other server's folders to it, or use NAS to create a common location.

    Can you share your planned architecture diagram with us? Like I said before, I highly recommend 3 identical hosts running the same setup with a NAS to share files and cross-linked log folders.
  • Dear 

    Thank you so much for your help :D

    The diagram in the question box is indeed my situation. The mounting method I used for the shared files would result in a cross reference between the hosts (i.e. any update of one host would reflect in the mounted directories of others), so does that mean the "real share"?

    However, I ran into the problem that the second jboss (host2) fails to run (the first jboss host1 started ok),  although the mappings and schema are validated successfully.

     

    Do you have any clues from the server log? 

     

    Btw, it looks quite similar to this case you have solved before:

    community.appian.com/.../i-m-installing-an-environment-with-2-jboss-2-different-machines-pointing-t

    734176.server.log

  • Dear Mike Cichy (mike.cichy),

     

    I have found the solution!

    The problem lies in the ERROR:

    Caused by: java.io.FileNotFoundException: /home/user01/appian/_admin/shared/appian.keystore.lock (Permission denied)

    indicating that there should be "read and write"permission to the mounted files ( such permission should be set on the TO-mount server, coz it is unable to change the permission of files on the TO-BE-mounted server)

     

    ref:

    KB-1103 "Unable to build keystore lock file" error thrown during JBoss startup

    community.appian.com/.../kb-1103-unable-to-build-keystore-lock-file-error-thrown-during-jboss-startup

     

    Now my second Jboss is able to start. Thanks for your help!!!