KB-1763 "HTTP Code: 500" error observed when performing several actions in the environment

Symptoms

When performing several actions on the environment (for example, importing or exporting an application), the following error is observed:

The following error is seen in the application server log:

blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; (APNX-1-4198-000)

One or both of the following errors are also seen in the search server log (located at <APPIAN_HOME>/logs/search-server):

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]
[WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] [Node <HOSTNAME>:<SEARCH_SERVER_PORT>] flood stage disk watermark [95%] exceeded on [<HOSTNAME>][Node <HOSTNAME>:<SEARCH_SERVER_PORT>][<APPIAN_HOME>/search-server/data/nodes/0] free: X.Xgb[X.X%], all indices on this node will be marked read-only

Cause

These above exceptions are related to a property in Elasticsearch: cluster.routing.allocation.disk.watermark.flood_stage. When Elasticsearch detects that the disk usage is more than 95%, it changes the indices to read-only, which makes several actions unavailable on the environment. More information about this property can be found in this documentation.

Action

To solve this issue, first reduce the disk usage or add more disk space to the server. Then, execute the following command to clear the indices:

Linux

curl -XPUT --header "Authorization: Basic YWRtaW46YQ==" "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '

Windows

Note: Run the command in PowerShell or Command Prompt (cmd.exe), if it is configured to run as PowerShell. 

Invoke-RestMethod -Method Put -Uri "http://localhost:9200/_all/_settings" -Header @{"Authorization" = "Basic YWRtaW46YQ=="} -Body '{"index.blocks.read_only_allow_delete": null}' -ContentType "application/json"

Verify that the following message appears after running the above command:

{"acknowledged":true}

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: November 2019

Related
Recommended