KB-1763 "HTTP Code: 500" error observed with ClusterBlockException or DiskThresholdMonitor error in the search server log

Symptoms

When performing several actions on the environment (for example, importing or exporting an application), the following error is observed:

The following error is seen in the application server log:

blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; (APNX-1-4198-000)

One or both of the following errors are also seen in the search server log (located at <APPIAN_HOME>/logs/search-server):

org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

[WARN ][org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor] [Node <HOSTNAME>:<SEARCH_SERVER_PORT>] flood stage disk watermark [95%] exceeded on [<HOSTNAME>][Node <HOSTNAME>:<SEARCH_SERVER_PORT>][<APPIAN_HOME>/search-server/data/nodes/0] free: X.Xgb[X.X%], all indices on this node will be marked read-only

Cause

These above exceptions are related to a property in Elasticsearch: cluster.routing.allocation.disk.watermark.flood_stage. When Elasticsearch detects that the disk usage is more than 95%, it changes the indices to read-only, which makes several actions unavailable on the environment. More information about this property can be found in this documentation.

Action

To solve this issue, first reduce the disk usage or add more disk space to the server. Then, execute the following commands to unblock the indices:

Linux

1. Authenticate with the search server

   APIKEY=$(awk '/^conf.data.search-server.restclient.apiKey=/ { match($0, /conf.data.search-server.restclient.apiKey=(.*)/, arr); print arr[1] }' <APPIAN_HOME>/conf/custom.properties);
   AUTHHEADER="Authorization: ApiKey $(echo -n $APIKEY | base64 -w0)";
   alias curl='curl --header "$AUTHHEADER"'
   

2. Unblock the indices that are in read-only state

   curl -XPUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '

3. After the command finishes, the following message will be outputted: {"acknowledged":true}

Windows

1. Note: Run the command in PowerShell or Command Prompt (cmd.exe), if it is configured to run as PowerShell. 

   Invoke-RestMethod -Method Put -Uri "http://localhost:9200/_all/_settings" -Header @{"Authorization" = "Basic YWRtaW46YQ=="} -Body '{"index.blocks.read_only_allow_delete": null}' -ContentType "application/json"

2. Verify that the following message appears after running the above command: {"acknowledged":true}

Affected Versions

This article applies to all versions of Appian.

Last Reviewed: March 2024