KB-1082 Issues related to search server indices

Symptoms

Users may experience one of the following symptoms related to search server indices when using any search server-related functionality.

Symptom 1

For Appian 7.10 and later, when running Rule Performance in the Admin Console, the report is not displayed and a Waiting indicator is shown. After waiting for a few minutes, the indicator disappears, and the following error is displayed in the application server log:

ERROR com.appiancorp.common.logging.GWTRemoteLoggingService - Swallowed an error with no error code. ResponseClass: class com.appiancorp.gwt.tempo.client.designer.EvaluateUiResponse

Note: The above error message is very generic and there are multiple possibilities for a root cause. In this case, the rule performance timing out is the root cause.

Symptom 2

When running an impact analysis, users will see the following error modal in their browser:

A server error was encountered while processing your request. Please try again.

After seeing this error, the following error will be printed in the application server log:

[[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR com.appiancorp.gwt.ia.server.GetImpactAnalysisImpl - Error running impact analysis using search server. targetObjects=[TypedValue[it=80,v=10568]]
java.lang.IllegalStateException: Data in the index is over 10 minutes behind the system of record. upToDateAsOfBySource: {k-content=Optional.of(2015-12-28 03:01:23.49), k-process-design=Optional.of(2015-12-28 03:01:33.17), k-personalization=Optional.of(2015-12-28 03:01:34.2), rdbms-primary=Optional.of(2015-12-28 03:01:34.21)}

Symptom 3

The following error will be repeated in the application server log:

[elasticsearch[Client 2636E3BE][transport_client_worker][T#5]{New I/O worker #5}] ERROR com.appian.dl.repo.es.LoggingBulkResponseActionListener - Bulk request failed item: [opType=index, index=xray-rule-execution, type=05cce14550a54e01b503027da635996c2, id=DT-1, status=SERVICE_UNAVAILABLE, message=UnavailableShardsException[[xray-rule-execution][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@3ec3bf65]]

Symptom 4

During an attempt to upload a file, object, or application into Appian, the progress bar stalls at 100% and does not allow import or upload.

Symptom 5

File uploads take an inordinate amount of time, sometimes longer than 15 minutes.

The following is printed in the application server log:

INFO  [stdout] (elasticsearch[Client 7FE43B33][transport_client_worker][T#4]{New I/O worker #4})  [elasticsearch[Client 7FE43B33][transport_client_worker][T#4]{New I/O worker #4}] 
ERROR com.appian.dl.repo.es.LoggingBulkResponseActionListener - Bulk request failures occurred. Summary: [opType=index, index=designer-objects-ia, type=DT-10, status=SERVICE_UNAVAILABLE, count=1, firstMessage=UnavailableShardsException[[designer-objects-ia][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@515eabc9]].

The following is printed in the search-server.log:

[WARN ][org.elasticsearch.indices.cluster] [Node localhost:9300] [designer-objects-ia][0] sending failed shard after recovery failure
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [designer-objects-ia][0] failed to recover shard
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:290)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
    at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:72)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:260)
    ... 4 more
Caused by: org.elasticsearch.ElasticsearchException: failed to read [DT-10][_a-0000dd1b-8f2d-8000-0315-010000010000_18993]
    at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:522)
    at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68)
    ... 5 more
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [48]
    at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
    at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:519)
    ... 6 more

Symptom 6

The following error is printed in the application server log:

[ServerService Thread Pool -- 65] ERROR com.appiancorp.ix.analysis.LoadingPagedIterator - Error getting content item [identifier=441937]
com.appian.dl.repo.QueryException: Query failed [request=QueryRequest{from=Type -10 (id=-10), timeZone=null, query=Query[Selection[relationships.uuid (show)], criteria[((type = TypedValue[it=3,v=Application]) AND (relationships.uuid in TypedValue[it=103,v={SYSTEM_CONTENT_ICON_NEWS_EVENT_SHOPPING_CART_RED}]))], PagingInfo[startIndex=0, batchSize=-1, sort=[]], options=QueryOptions{dataLimitInBytes=100000000, cardinalityPrecisionThreshold=0, timeoutMs=-1}]}, ES search request={
...
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed

Symptom 7

The following error is printed in the application server log:

[ServerService Thread Pool -- 63] FATAL com.appiancorp.common.web.StartupHaltingServletContextListener - Halting JVM startup: An unexpected error occurred while trying to initialize and validate the Appian data source.
com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianDS] Could not synchronize the search index with the database data.
...
Caused by: java.lang.IllegalStateException: unexpected docvalues type NONE for field 'id' (expected=SORTED). Use UninvertingReader or index with docvalues.

Symptom 8

Users will notice the Appian Designer freeze when creating new objects or performing basic operations to existing objects.

Symptom 9

The following error is printed in the application server log:

ERROR com.appiancorp.tempo.abdera.TempoEntryAdapter - Error retrieving entries.
java.lang.IllegalArgumentException: No enum constant com.appian.dl.repo.PersistenceMetadataImpl.Field.textField 

Symptom 10

After upgrading Appian, when navigating to Tempo (or any of its tabs), the page fails to load certain user interface components. The components that do not load include the navigation tabs, user avatar, left-side search bars and pickers, and all News posts. The following image is an example of what is seen in this situation:

Symptom 11

The following warn trace is seen in the application server log:

WARN com.appiancorp.security.auth.activity.UserActivityFilter - Could not record user activity: secCtx=XXXXXXX, authDetails=AuthenticationDetails [Details] -- MasterNotDiscoveredException[null]
 

The MasterNotDiscoveredException points to a no known master node error in the search-server.log:

[DEBUG][org.elasticsearch.action.admin.cluster.health] [Node SITE_FQDN:PORT#] no known master node, scheduling a retry
[DEBUG][org.elasticsearch.action.admin.cluster.health] [Node SITE_FQDN:PORT#] timed out while retrying [cluster:monitor/health] after failure (timeout [Xs])

The search_server_cluster.csv log file located in <APPIAN_HOME>/logs/data-metrics/ will show that the search server was/is in a down state (RED):

DAY/MONTH/YEAR TIME GMT,appian-search-cluster,RED,false,X,X,X,X,X,X,X

Cause

The search indices were corrupted due to one of the following reasons:

  1. The machine the search server is hosted on ran out of disk space.
  2. The search server did not shut down properly.
  3. The search server indices experienced an unknown corruption.

Action

Recreate the search server indices using the following steps.

Note: Please refer to the Starting and Stopping Appian documentation when performing these steps:

  1. Stop the application server.
  2. Stop the search server.
  3. Recreate the search indices using the steps here. Note: If you are not using the latest version of Appian, be sure to replace the version number in the URL with the relevant Appian version.
  4. Start the search server. Note: In order for the search server indices to recreate, you must wait for the search server to start completely before starting the application server.
  5. Start the application server.

In some instances, users may choose to use a different directory for the search indices. To confirm the location of the search indices, perform the following:

  1. Open custom.properties.
  2. Search for one of the following properties, depending on your version of Appian:
    • Appian 7.11 and earlier: conf.data.primary.datasource.search.index
    • Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: February 2019

Related
Recommended