KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Devon Lee — Thu, 18 Jan 2024 03:22:38 GMT

Current Revision posted to Appian Knowledge Base by Devon Lee on 1/18/2024 3:22:38 AM

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

The following are possible causes to this issue:

Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.
The application server is improperly shut down.
In a distributed environment, the network drive where <APPIAN_HOME>/_admin resides becomes unavailable while the application server is starting up or shutting down. The network drive becoming unavailable would prevent the deletion of the write.lock file.

If the conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined or is using the wrong property name, then Appian will use <BASE_PATH>/search-local/index-primaryas the value of that property. For the purposes of this KB article, <BASE_PATH> denotes the directory defined in the property conf.suite.BASE_PATH in custom.properties.

If no value is set for conf.suite.BASE_PATH and conf.data.APPIAN_DATA_SOURCE.search.index(conf.data.primary.datasource.search.index) is not defined correctly (or defined at all), Appian will use <APPIAN_HOME>/_admin/search-local/index-primary on that particular machine as the value of conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index).

Action

Ensure that the correct property name is being used:
- Appian 7.11 and earlier: conf.data.primary.datasource.search.index
- Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
Ensure that the property itself is only referenced once in each custom.properties file.
Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive. Note: You can create two separate folders within the same network drive and have each application server point to the different folders, as this is a valid configuration.

An example of how each custom.properties file should be defined:

custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
Similar notation for any additional application server.

If the above steps don't work, perform the following:

Stop the application server.
Delete the write.lock file in <APPIAN_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry.
Restart the application server.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: database, installation, application server, windows, infrastructure, jboss

KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Jordan Horwat — Mon, 23 Jul 2018 17:33:36 GMT

Revision 3 posted to Appian Knowledge Base by Jordan Horwat on 7/23/2018 5:33:36 PM

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

The following are possible causes to this issue:

Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.
The application server is improperly shut down.
In a distributed environment, the network drive where <APPIAN_HOME>/_admin resides becomes unavailable while the application server is starting up or shutting down. The network drive becoming unavailable would prevent the deletion of the write.lock file.

Action

Ensure that the correct property name is being used:
- Appian 7.11 and earlier: conf.data.primary.datasource.search.index
- Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
Ensure that the property itself is only referenced once in each custom.properties file.
Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive. Note: You can create two separate folders within the same network drive and have each application server point to the different folders, as this is a valid configuration.

An example of how each custom.properties file should be defined:

custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
Similar notation for any additional application server.

If the above steps don't work, perform the following:

Stop the application server.
Delete the write.lock file in <APPIAN_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry.
Restart the application server.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: database, installation, application server, windows, infrastructure, jboss

KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Nick Vigilante — Tue, 07 Mar 2017 10:07:47 GMT

Revision 2 posted to Appian Knowledge Base by Nick Vigilante on 3/7/2017 10:07:47 AM

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

The following are possible causes to this issue:

Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.
The application server is improperly shut down.
In a distributed environment, the network drive where <APPIAN_HOME>/_admin resides becomes unavailable while the application server is starting up or shutting down. The network drive becoming unavailable would prevent the deletion of the write.lock file.

Action

Ensure that the correct property name is being used:
- Appian 7.11 and earlier: conf.data.primary.datasource.search.index
- Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
Ensure that the property itself is only referenced once in each custom.properties file.
Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive. Note: You can create two separate folders within the same network drive and have each application server point to the different folders, as this is a valid configuration.

An example of how each custom.properties file should be defined:

custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
Similar notation for any additional application server.

If the above steps don't work, perform the following:

Stop the application server.
Delete the write.lock file in <APPIAN_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry.
Restart the application server.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: database, application server, windows, jboss

KB-1223 "Could not initialize search engine... Lock obtain timed out" error thrown after starting second application server in an environment with multiple application servers

Nick Vigilante — Mon, 06 Mar 2017 21:17:55 GMT

Revision 1 posted to Appian Knowledge Base by Nick Vigilante on 3/6/2017 9:17:55 PM

Symptoms

In a distributed environment containing two JBoss application servers, users will notice the following error upon starting the second application server:

12:38:22,974 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 89) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.config.ConfigurationLoader: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianPrimaryDS] Could not initialize search engine.
...
Caused by: java.lang.IllegalStateException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@<BASEPATH_HOME>/_admin/search-local/index-primary/com.appiancorp.tempo.rdbms.EventFeedEntry/write.lock
...

The second application server started will always fail to deploy.

Cause

There are three possible causes to this issue:

Starting with Appian 16.1, the name of the conf.data.primary.datasource.search.index property was updated to conf.data.APPIAN_DATA_SOURCE.search.index. However, custom.properties was not updated to reflect this on both application servers.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property points to the same shared network drive in both application servers' custom.properties file.
The conf.data.APPIAN_DATA_SOURCE.search.index (conf.data.primary.datasource.search.index) property is not defined.

Action

Ensure that the correct property name is being used:
- Appian 7.11 and earlier: conf.data.primary.datasource.search.index
- Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index
Ensure that the property itself is only referenced once in each custom.properties file.
Ensure that there is a value set for the appropriate property and that the two application servers do not point to the same shared network drive.

An example of how each custom.properties file should be defined:

custom.properties on application server 1: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-1/index-primary
custom.properties on application server 2: conf.data.APPIAN_DATA_SOURCE.search.index=/path/to/search-local-2/index-primary
Similar notation for any additional application server.

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2017

Tags: database, application server, windows, jboss