[DRAFT SP-9623] Kafka/Zookeeper/Tomcat related data ownership being non-Appian user account prevents services from starting up

Symptoms

Users may experience one or more of the following symptoms when starting services.

Zookeeper:

Upon starting services, the console output shows the following:

Starting QuorumPeerMain...
process QuorumPeerMain running
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available
waiting for Zookeeper to be available

Inside the zookeeper.log, found in <APPIAN_HOME>/logs/service-manager/zookeeper, one or more of following error(s) are shown:

YYYY-MM-DD HH:MM:SS,mmm [myid:] - ERROR [PurgeTask:o.a.z.s.DatadirCleanupManager$PurgeTask@143] - Error occurred while purging.
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory <APPIAN_HOME>/services/data/zookeeper/version-2

YYYY-MM-DD HH:MM:SS,mmm [myid:0] - ERROR [main:o.a.z.s.ZooKeeperServerMain@81] - Unable to access datadir, exiting abnormally
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory <APPIAN_HOME>/services/data/zookeeper/version-2

YYYY-MM-DD HH:MM:SS,mmm [myid:0] - ERROR [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 3

Kafka:

Upon starting services, the console output shows the following:

Starting Kafka...
waiting for process Kafka
process Kafka running
local Kafka broker is not registered
<<< Kafka broker initialization failed for localhost:9092. Check the broker logs for more information >>>
Deleting Kafka index files in case initialization failed due to a corrupt index file.
Unable to start service Kafka
com.appian.komodo.translog.KafkaNotRunningException: Kafka process died

Inside the server.log, found inside <APPIAN_HOME>/logs/service-manager/kafka, the one or more of following error(s) are shown:

[YYYY-MM-DD HH:MM:SS,mmm] ERROR Error while loading log dir /usr/local/appian/appian224/appian/services/data/kafka-logs (kafka.server.LogDirFailureChannel) java.nio.file.AccessDeniedException: /usr/local/appian/appian224/appian/services/data/kafka-logs/serviceManager.transaction.<engine>-#/####################.log

[YYYY-MM-DD HH:MM:SS,mmm] ERROR Error while deleting the clean shutdown file in dir <APPIAN_HOME>/services/data/kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: <APPIAN_HOME>/services/data/kafka-logs/serviceManager.transaction.<engine>-#/####################.log

Tomcat:

Upon starting the app server, one or more of the following errors are displayed:
YYYY-MM-DD HH:MM:SS SEVERE [main] org.apache.catalina.startup.HostConfig.deployDescriptor Error deploying deployment descript
or [<APPIAN_HOME>/tomcat/apache-tomcat/conf/Catalina/localhost/suite.xml]
java.io.FileNotFoundException: ../../../conf/tomcatResources.xml (Permission denied)

2024-04-05 14:20:35 INFO [main] org.apache.coyote.AbstractProtocol.start Startingq ProtocolHandler ["http-nio-8080"]
2024-04-05 14:20:35 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to start component [Connector["http-nio-8080"]]
org.apache.catalina.LifecycleException: Protocol handler start failed

Cause

Necessary component files are owned by non-Appian user accounts.

Action

To check the file permissions inside the environment, run the following:

1. Check to see if files are owned by different users inside <APPIAN_HOME> . 
   1. sudo find <APPIAN_HOME> -printf '%u:%g\n' | sort -t: -u
   2. For High Availability (HA) Sites: sudo find -L <APPIAN_HOME> -printf '%u:%g\n' | sort -t: -u
2. Change ownership of the affected files to be under the Appian user.
3. Attempt startup.
4. If the issue persists, please open a support ticket for further investigation.

Affected Versions

This article applies to all self-managed versions of Appian.

Last Reviewed: April 2024