KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

The following are possible causes to this issue:

• Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
• The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
• The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.
• The application server is improperly shut down.
• In a distributed environment, the network drive where <APPIAN_HOME>/_admin resides becomes unavailable while the application server is starting up or shutting down. The network drive becoming unavailable would prevent the deletion of the write.lock file.

If the conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined or is using the wrong property name, then Appian will use <BASE_PATH>/search-local/index-primaryas the value of that property. For the purposes of this KB article, <BASE_PATH> denotes the directory defined in the property conf.suite.BASE_PATH in custom.properties.

If no value is set for conf.suite.BASE_PATH and conf.data.APPIAN_DATA_SOURCE.search.index(conf.data.primary.datasource.search.index) is not defined correctly (or defined at all), Appian will use <APPIAN_HOME>/_admin/search-local/index-primary on that particular machine as the value of conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index).

Action

1. Ensure that the correct property name is being used:
   • Appian 7.11 and earlier: conf.data.primary.datasource.search.index
   • Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
2. Ensure that the property itself is only referenced once in each custom.properties file.
3. Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive. Note: You can create two separate folders within the same network drive and have each application server point to the different folders, as this is a valid configuration.

An example of how each custom.properties file should be defined:

• custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
• custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
• Similar notation for any additional application server.

If the above steps don't work, perform the following:

1. Stop the application server.
2. Delete the write.lock file in <APPIAN_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry.
3. Restart the application server.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017