KB-2314 Kafka/Zookeeper/Tomcat related data ownership being non-Appian user account prevents services from starting up

Symptoms

Users may experience one or more of the following symptoms when starting services.

Zookeeper:

Upon starting services, the console output shows the following:

Starting QuorumPeerMain...
process QuorumPeerMain running
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available

Inside the zookeeper.log, found in <APPIAN_HOME>/logs/service-manager/zookeeper, one or more of following error(s) are shown:

YYYY-MM-DD HH:MM:SS,mmm [myid:] - ERROR [PurgeTask:o.a.z.s.DatadirCleanupManager$PurgeTask@143] - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory <APPIAN_HOME>/services/data/zookeeper/version-2

YYYY-MM-DD HH:MM:SS,mmm [myid:0] - ERROR [main:o.a.z.s.ZooKeeperServerMain@81] - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory <APPIAN_HOME>/services/data/zookeeper/version-2

YYYY-MM-DD HH:MM:SS,mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 3

Kafka:

Upon starting services, the console output shows the following:

Starting Kafka...
waiting for process Kafka
process Kafka running
local Kafka broker is not registered
<<< Kafka broker initialization failed for localhost:9092. Check the broker logs for more information >>>
Deleting Kafka index files in case initialization failed due to a corrupt index file.
Unable to start service Kafka
com.appian.komodo.translog.KafkaNotRunningException: Kafka process died

Inside the server.log, found inside <APPIAN_HOME>/logs/service-manager/kafka, the one or more of following error(s) are shown:

[YYYY-MM-DD HH:MM:SS,mmm] ERROR Error while loading log dir /usr/local/appian/appian224/appian/services/data/kafka-logs (kafka.server.LogDirFailureChannel) java.nio.file.AccessDeniedException: /usr/local/appian/appian224/appian/services/data/kafka-logs/serviceManager.transaction.<engine>-#/####################.log

[YYYY-MM-DD HH:MM:SS,mmm] ERROR Error while deleting the clean shutdown file in dir <APPIAN_HOME>/services/data/kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: <APPIAN_HOME>/services/data/kafka-logs/serviceManager.transaction.<engine>-#/####################.log

Tomcat:

Inside the tomcat-stdout.log, found inside <APPIAN_HOME>/logs/, the one or more of following error(s) are shown:

YYYY-MM-DD HH:MM:SS,mmm SEVERE [main] org.apache.catalina.startup.HostConfig.deployDescriptor Error deploying deployment descript
or [<APPIAN_HOME>/tomcat/apache-tomcat/conf/Catalina/localhost/suite.xml]
 java.io.FileNotFoundException: ../../../conf/tomcatResources.xml (Permission denied)

YYYY-MM-DD HH:MM:SS,mmm [localhost-startStop-1] ERROR com.appiancorp.common.initialize.MigrateNavigationConfig - Could not merge navigation-config stored in content with the one provided by default. java.io.FileNotFoundException: <APPIAN_HOME>/_admin/accdocs*/2/*.xml (Permission denied)

Cause

Necessary component files are owned by non-Appian user accounts. Appian should have its own user and root must not be used as it may lead to unexpected behavior. For more information, please see the following documentation.

Action

To check the file permissions inside the environment, run the following:

1. Check to see if files are owned by different users inside <APPIAN_HOME> . 
   1. sudo find <APPIAN_HOME> -printf '%u:%g\n' | sort -t: -u
   2. For High Availability (HA) Sites: sudo find -L <APPIAN_HOME> -printf '%u:%g\n' | sort -t: -u
2. Change ownership of the affected files to be under the Appian user.
3. Attempt startup.
4. If the issue persists, please open a support ticket for further investigation.

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: July 2024