KB-1763 "HTTP Code: 500" error observed with ClusterBlockException or DiskThresholdMonitor error in the search server log


When performing several actions on the environment (for example, importing or exporting an application), the following error is observed:

The following error is seen in the application server log:

blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; (APNX-1-4198-000)

One or both of the following errors are also seen in the search server log (located at <APPIAN_HOME>/logs/search-server):

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]
[WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] [Node <HOSTNAME>:<SEARCH_SERVER_PORT>] flood stage disk watermark [95%] exceeded on [<HOSTNAME>][Node <HOSTNAME>:<SEARCH_SERVER_PORT>][<APPIAN_HOME>/search-server/data/nodes/0] free: X.Xgb[X.X%], all indices on this node will be marked read-only


These above exceptions are related to a property in Elasticsearch: cluster.routing.allocation.disk.watermark.flood_stage. When Elasticsearch detects that the disk usage is more than 95%, it changes the indices to read-only, which makes several actions unavailable on the environment. More information about this property can be found in this documentation.


To solve this issue, first reduce the disk usage or add more disk space to the server. Then, execute the following command to unblock the indices:


curl -XPUT --header "Authorization: Basic YWRtaW46YQ==" "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '


Note: Run the command in PowerShell or Command Prompt (cmd.exe), if it is configured to run as PowerShell. 

Invoke-RestMethod -Method Put -Uri "http://localhost:9200/_all/_settings" -Header @{"Authorization" = "Basic YWRtaW46YQ=="} -Body '{"index.blocks.read_only_allow_delete": null}' -ContentType "application/json"

Verify that the following message appears after running the above command:


Affected Versions

This article applies to all versions of Appian.

Last Reviewed: January 2020