KB-XXXX High Availability Service Manager cluster refuses to shutdown

Symptoms

While attempting to shutdown the engines in an HA cluster, the service manager process refuses to carry out the shutdown and ignores the stop script.

Users may see the following message in the logs:

INFO  com.appian.komodo.admin.route.ManagerActionsRoute - Manager action 'shutdown-cluster' invoked

However the shutdown does not proceed as expected.

Alternatively, if there was an issue with a graceful shutdown, the cluster-stop state is most likely the root cause if every "'shutdown-cluster' invoked" message is immediately followed by a "Cluster shutdown already in progress!" message in the SM log. For example:

INFO  com.appian.komodo.admin.route.ManagerActionsRoute - Manager action 'shutdown-cluster' invoked
INFO  com.appian.komodo.admin.route.ManagerActionsRoute - Cluster shutdown already in progress!
INFO  com.appian.komodo.admin.route.ManagerActionsRoute - Manager action 'shutdown-cluster' invoked
INFO  com.appian.komodo.admin.route.ManagerActionsRoute - Cluster shutdown already in progress!

Summary of symptoms:

Failed graceful shutdown for HA sites
Service Manager is unresponsive to the stop script, even with the -c option.
Engines fail to promote to PRIMARY on a particular node (even if there are no other primaries and the engine is a RUNNING REPLICA)

Cause

Typically, a user is trying to shutdown service manager on just one node in an HA configuration, but accidentally does a cluster stop which affects all other service managers in a cluster. SM could also get into this state if there is an issue with an intentional cluster shutdown, but only the engines are restarted and not service manager.

Action

This issue can be remediated by restarting service manager.

Affected Versions

This article applies to all versions of Appian running with a High Availability Topology

Last Reviewed: May 2020