KB-2231 Unable to shut down Appian Java components with OOTB stop script

Symptoms

1. Zookeeper, Kafka, and/or Tomcat intermittently are not able to be stopped by Appian's out-of-the-box (OOTB) stop.sh script in <APPIAN_HOME>/services/bin/ or stop-appserver.sh in <APPIAN_HOME>/tomcat/apache-tomcat/bin/.
2. The process IDs (PIDs) of Zookeeper, Kafka, and/or Tomcat are not shown in the output of jps command, but able to be identified in the output of ps -ef | grep java command.
3. /tmp/hsperfdata_<current_user> does not contain the PIDs of Zookeeper, Kafka, and/or Tomcat.

Cause

The stop.sh and stop-appserver.sh script searches for Zookeeper, Kafka, and Tomcat in Java processes, which are referred to in the jps command. jps searches /tmp/hsperfdata_<current_user>, where it expects Java processes to have created a directory with metadata.

When the /tmp/hsperfdata_<current_user> has a file for each of the Appian Java process' PID, the corresponding process will show up in the jps command output and the OOTB stop scripts should work.

When the /tmp/hsperfdata_<current_user> does not contain a file for each of the Appian Java process' PID, the corresponding process will not show up in the output of jps command, resulting in the failure of the  OOTB stop scripts to stop Appian Java components. 

Action

1. Confirm if the stop script is running as a different user than the Appian Java process, as the directory that the jps command searches in is dependent on the current user. If the user is different, make sure the stop script is ran by the same user as the Appian Java process in question.
2. If the stop script is ran by the exact same user as the running Java processes and the issue only happens after Appian runs for more than a certain period of time, confirm with your server team to see if there is any cronjob/mechanism that is cleaning up the /tmp/hsperfdata_<current_user> folder or the entire /tmp directory.

Workaround

Bring down Tomcat

If Tomcat cannot be stopped with the OOTB stop-appserver script, follow the below steps to bring down tomcat:

1. Run ps -ef | grep tomcat to identify the PID for Tomcat.
2. Manually terminate Tomcat by using kill -9 <Tomcat_PID> 

Bring down Kafka and Zookeeper

If Kafka and Zookeeper cannot be stopped with the OOTB stop script, follow the below steps to bring down Kafka and Zookeeper:

1. Make sure app server(s), search server(s), data server(s), and engines have all been stopped.
2. Run ps -ef | grep kafka and ps -ef | grep zookeeper on all nodes hosting Kafka and Zookeeper to identify the PIDs respectively. 
3. Manually terminate Kafka on all nodes hosting Kafka using the following command: kill -9 <Kafka_PID>
4. Next, manually terminate Zookeeper on all nodes hosting Zookeeper using the following command: kill -9 <Zookeeper_PID>
5. Confirm all Appian components are down on all nodes.
6. Start up Internal Messaging Services using <APPIAN_HOME>/services/bin/start.sh -p <password> -s manager on all nodes hosting Kafka and Zookeeper. This should start up Service Manager, Kafka, and Zookeeper. This should also generate the expected data in /tmp/hsperfdata_<current_user> and as a result the jps command should return PIDs for Service Manager, Kafka and Zookeeper. 
7. Confirm that Zookeeper cluster and Kafka cluster are up and running using status script.
8. Stop all the components gracefully depending on the installation type listed below
   1. Standalone: run <APPIAN_HOME>/services/bin/stop.sh -p <password> -s all on the node hosting Zookeeper and Kafka.
   2. HA: run <APPIAN_HOME>/services/bin/stop.sh -p <password> on all nodes hosting Zookeeper and Kafka.

Affected Versions

This article applies to all self-managed versions of Appian on Linux.

Last Reviewed: January 2023