KB-1082 Issues related to search server indices

Symptoms

Users may experience one of the following symptoms related to search server indices when using any search server-related functionality.

Symptom 1

For Appian 7.10 and later, when running Rule Performance in the Admin Console, the report is not displayed and a Waiting indicator is shown. After waiting for a few minutes, the indicator disappears, and the following error is displayed in the application server log:

ERROR com.appiancorp.common.logging.GWTRemoteLoggingService - Swallowed an error with no error code. ResponseClass: class com.appiancorp.gwt.tempo.client.designer.EvaluateUiResponse

Note: The above error message is very generic and there are multiple possibilities for a root cause. In this case, the rule performance timing out is the root cause.

Symptom 2

When running an impact analysis, users will see the following error modal in their browser:

A server error was encountered while processing your request. Please try again.

After seeing this error, the following error will be printed in the application server log:

[[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR com.appiancorp.gwt.ia.server.GetImpactAnalysisImpl - Error running impact analysis using search server. targetObjects=[TypedValue[it=80,v=10568]]
java.lang.IllegalStateException: Data in the index is over 10 minutes behind the system of record. upToDateAsOfBySource: {k-content=Optional.of(2015-12-28 03:01:23.49), k-process-design=Optional.of(2015-12-28 03:01:33.17), k-personalization=Optional.of(2015-12-28 03:01:34.2), rdbms-primary=Optional.of(2015-12-28 03:01:34.21)}

Symptom 3

The following error will be repeated in the application server log:

[elasticsearch[Client 2636E3BE][transport_client_worker][T#5]{New I/O worker #5}] ERROR com.appian.dl.repo.es.LoggingBulkResponseActionListener - Bulk request failed item: [opType=index, index=xray-rule-execution, type=05cce14550a54e01b503027da635996c2, id=DT-1, status=SERVICE_UNAVAILABLE, message=UnavailableShardsException[[xray-rule-execution][0] Primary shard is not active or isn't assigned is a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@3ec3bf65]]

Symptom 4

During an attempt to upload a file, object, or application into Appian, the progress bar stalls at 100% and does not allow import or upload.

Symptom 5

File uploads take an inordinate amount of time, sometimes longer than 15 minutes.

The following is printed in the application server log:

INFO  [stdout] (elasticsearch[Client 7FE43B33][transport_client_worker][T#4]{New I/O worker #4})  [elasticsearch[Client 7FE43B33][transport_client_worker][T#4]{New I/O worker #4}] 
ERROR com.appian.dl.repo.es.LoggingBulkResponseActionListener - Bulk request failures occurred. Summary: [opType=index, index=designer-objects-ia, type=DT-10, status=SERVICE_UNAVAILABLE, count=1, firstMessage=UnavailableShardsException[[designer-objects-ia][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@515eabc9]].

The following is printed in the search-server.log:

[WARN ][org.elasticsearch.indices.cluster] [Node localhost:9300] [designer-objects-ia][0] sending failed shard after recovery failure
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [designer-objects-ia][0] failed to recover shard
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:290)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
    at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:72)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:260)
    ... 4 more
Caused by: org.elasticsearch.ElasticsearchException: failed to read [DT-10][_a-0000dd1b-8f2d-8000-0315-010000010000_18993]
    at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:522)
    at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68)
    ... 5 more
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [48]
    at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
    at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:519)
    ... 6 more

Symptom 6

The following error is printed in the application server log:

[ServerService Thread Pool -- 65] ERROR com.appiancorp.ix.analysis.LoadingPagedIterator - Error getting content item [identifier=441937]
com.appian.dl.repo.QueryException: Query failed [request=QueryRequest{from=Type -10 (id=-10), timeZone=null, query=Query[Selection[relationships.uuid (show)], criteria[((type = TypedValue[it=3,v=Application]) AND (relationships.uuid in TypedValue[it=103,v={SYSTEM_CONTENT_ICON_NEWS_EVENT_SHOPPING_CART_RED}]))], PagingInfo[startIndex=0, batchSize=-1, sort=[]], options=QueryOptions{dataLimitInBytes=100000000, cardinalityPrecisionThreshold=0, timeoutMs=-1}]}, ES search request={
...
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed

Symptom 7

The following error is printed in the application server log:

[ServerService Thread Pool -- 63] FATAL com.appiancorp.common.web.StartupHaltingServletContextListener - Halting JVM startup: An unexpected error occurred while trying to initialize and validate the Appian data source.
com.appiancorp.suiteapi.common.exceptions.AppianException: An unexpected error occurred while trying to initialize and validate the Appian data source. (APNX-1-4179-004)
...
Caused by: java.lang.IllegalStateException: [jdbc/AppianDS] Could not synchronize the search index with the database data.
...
Caused by: java.lang.IllegalStateException: unexpected docvalues type NONE for field 'id' (expected=SORTED). Use UninvertingReader or index with docvalues.

Symptom 8

Users will notice the Appian Designer freeze when creating new objects or performing basic operations to existing objects.

Symptom 9

The following error is printed in the application server log:

ERROR com.appiancorp.tempo.abdera.TempoEntryAdapter - Error retrieving entries.
java.lang.IllegalArgumentException: No enum constant com.appian.dl.repo.PersistenceMetadataImpl.Field.textField 

Symptom 10

After upgrading Appian, when navigating to Tempo (or any of its tabs), the page fails to load certain user interface components. The components that do not load include the navigation tabs, user avatar, left-side search bars and pickers, and all News posts. The following image is an example of what is seen in this situation:

Symptom 11

The following warn trace is seen in the application server log:

WARN com.appiancorp.security.auth.activity.UserActivityFilter - Could not record user activity: secCtx=XXXXXXX, authDetails=AuthenticationDetails [Details] -- MasterNotDiscoveredException[null]
 

The MasterNotDiscoveredException points to the following error in the search-server.log:

[DEBUG][org.elasticsearch.action.admin.cluster.health] [Node SITE_FQDN:PORT#] no known master node, scheduling a retry
[DEBUG][org.elasticsearch.action.admin.cluster.health] [Node SITE_FQDN:PORT#] timed out while retrying [cluster:monitor/health] after failure (timeout [Xs])

The search_server_cluster.csv log file located in <APPIAN_HOME>/logs/data-metrics/ will show that the search server was/is in a down state (RED):

DAY/MONTH/YEAR TIME GMT,appian-search-cluster,RED,false,X,X,X,X,X,X,X

Cause

The search indices were corrupted due to one of the following reasons:

1. The machine the search server is hosted on ran out of disk space.
2. The search server did not shut down properly.
3. The search server indices experienced an unknown corruption or are in unhealthy state.

Action

Unblock the search indices

Follow the steps in KB-1763 to first attempt unblocking the search indices. If this does not resolve the issue, continue with the below steps. 

Use the delete API to rebuild individual search server indices

Starting from Appian 20.3, refreshing the indices by deleting search server data directory should be treated as a last resort due to the introduction of document extraction. If the customer is only reporting issues in certain functions, attempt to rebuild the individual index with the delete API. If the environment is highly available (HA), the deletion only needs to happen on one of the nodes hosting search servers.

1. Stop all application servers if not yet stopped.
2. Run the following command to list the indices and find the name of the index in question
   1. Appian 24.3 and later: curl -s localhost:9200/_cat/indices?v
   2. Appian 20.4 to 24.2: curl localhost:9200/_cat/indices?v
3. Authenticate with the Search Server
   1. Appian 24.3 and later: AUTHHEADER="Authorization: Basic $(awk '/^conf.search-server.user.appian.password=/ { match($0, /conf.search-server.user.appian.password=(.*)/, arr); print arr[1] }' /usr/local/appian/ae/search-server/conf/custom.properties | awk '{print "appian:"$1}' | xargs echo -n | base64 -w0)"
alias curl='curl --header "$AUTHHEADER"'
   2. Appian 20.4 to 24.2: APIKEY=$(awk '/^conf.data.search-server.restclient.apiKey=/ { match($0, /conf.data.search-server.restclient.apiKey=(.*)/, arr); print arr[1] }' <APPIAN_HOME>/conf/custom.properties); AUTHHEADER="Authorization: ApiKey $(echo -n $APIKEY | base64 -w0)"; alias curl='curl --header "$AUTHHEADER"'
4. Issue with importing/exporting application: curl -XDELETE localhost:9200/ix-activity-<version>
5. Issue with reviewing dependencies of objects: curl -XDELETE localhost:9200/designer-objects-ia-<version>
6. Issue with viewing news (tempo posts): curl -XDELETE localhost:9200/news-<version>
7. After the command finishes, the following message will be outputted: {"acknowledged":true}
8. Verify that the index in question is not longer displayed in the output of the following command:
   1. Appian 24.3 and later: curl -s localhost:9200/_cat/indices?v
   2. Appian 20.4 to 24.2: curl localhost:9200/_cat/indices?v
9. Start the application server(s).

If the issue persists after running the delete API to rebuild search server indices, proceed with removing the search server data.  

Recreate the search indices

Note: Keep in mind that running the below steps has the caveat of losing prediction data, if any, for documentation extraction.

1. Stop all app servers and search servers according to the documentation.
2. Recreate the search indices using the steps below. All search indices, except ones related to document extraction can be rebuilt upon application server restart:
   1. (Appian 18.3 and later) Delete the contents of the <APPIAN_HOME>/search-server/data/ directory from every server. Do not delete the directory itself.
   2. (Appian 18.2 and earlier) Remove the <APPIAN_HOME>/_admin/search-local/ directory. Repeat this for every node hosting application server.
3. Start the search server according to the documentation.

In some instances, users may choose to use a different directory for the search indices. To confirm the location of the search indices, perform the following:

1. Open custom.properties.
2. Search for one of the following properties, depending on your version of Appian:
   • Appian 7.11 and earlier: conf.data.primary.datasource.search.index
   • Appian 16.1 and later: conf.data.APPIAN_DATA_SOURCE.search.index

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: September 2024