KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

There are three possible causes to this issue:

• Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
• The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
• The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.

If the conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined or is using the wrong property name, then Appian will use <BASE_PATH>/search-local/index-primaryas the value of that property. For the purposes of this KB article, <BASE_PATH> denotes the directory defined in the property conf.suite.BASE_PATH in custom.properties.

If no value is set for conf.suite.BASE_PATH and conf.data.APPIAN_DATA_SOURCE.search.index(conf.data.primary.datasource.search.index) is not defined correctly (or defined at all), Appian will use <APPIAN_HOME>/_admin/search-local/index-primary on that particular machine as the value of conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index).

Action

1. Ensure that the correct property name is being used:
   • Appian 7.11 and earlier: conf.data.primary.datasource.search.index
   • Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
2. Ensure that the property itself is only referenced once in each custom.properties file.
3. Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive.

An example of how each custom.properties file should be defined:

• custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
• custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
• Similar notation for any additional application server.

Affected Versions

This article applies to all versions of Appian.