Unable to acquire a Read connection. Safe to retry.

Hi all, I am getting this error after starting the gateway. This is after engines and search started up (in that order) and seem to be running correctly.

05:47:43,127 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/suite]] (ServerService Thread Pool -- 77) JBWEB000287: Exception sending context initialized event to listener instance of class com.appiancorp.common.Bootst
rapSpringContextLoaderListener: com.appiancorp.common.config.FatalConfigurationException: com.appiancorp.suiteapi.common.exceptions.AppianException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localeCo
nfig' defined in com.appiancorp.common.spring.BootstrapSpringConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.appiancorp.ap2.LocaleConfig]:
 Factory method 'localeConfig' threw exception; nested exception is com.appiancorp.kougar.driver.exceptions.SafeRetryException: Unable to acquire a Read connection. Safe to retry.                                                                  
        at com.appiancorp.common.BootstrapSpringContextLoaderListener.createBootstrapContext(BootstrapSpringContextLoaderListener.java:84) [appian-ae-17.4.29.0.jar:]                                                                                
        at com.appiancorp.common.BootstrapSpringContextLoaderListener.lambda$doContextInitialized$1(BootstrapSpringContextLoaderListener.java:71) [appian-ae-17.4.29.0.jar:]                                                                         
        at com.appiancorp.security.changelog.AuditLogLocationService.callWithLocation(AuditLogLocationService.java:20) [appian-ae-17.4.29.0.jar:]                                                                                                    
        at com.appiancorp.common.BootstrapSpringContextLoaderListener.doContextInitialized(BootstrapSpringContextLoaderListener.java:67) [appian-ae-17.4.29.0.jar:]                                                                                  
        at com.appiancorp.common.web.StartupHaltingServletContextListener$1.run(StartupHaltingServletContextListener.java:74) [appian-ae-17.4.29.0.jar:]                                                                                             
        at com.appiancorp.security.changelog.AuditLogLocationService.runWithLocation(AuditLogLocationService.java:38) [appian-ae-17.4.29.0.jar:]                                                                                                     
        at com.appiancorp.common.web.StartupHaltingServletContextListener.contextInitialized(StartupHaltingServletContextListener.java:71) [appian-ae-17.4.29.0.jar:]                                                                                
        at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:3339) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]                                                                                       
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:3780) [jbossweb-7.5.24.Final-redhat-1.jar:7.5.24.Final-redhat-1]                                                                                                      
        at org.jboss.as.web.deployment.WebDeploymentService.doStart(WebDeploymentService.java:163) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]                                                                                    
        at org.jboss.as.web.deployment.WebDeploymentService.access$000(WebDeploymentService.java:61) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]                                                                                  
        at org.jboss.as.web.deployment.WebDeploymentService$1.run(WebDeploymentService.java:96) [jboss-as-web-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]                                                                                       
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_141]                                                                                                                                                
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_141]                                                                                                                                                               
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]                                                                                                                                        
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]                                                                                                                                        
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]                                                                                                                                                                                  
        at org.jboss.threads.JBossThread.run(JBossThread.java:122)  

The gateway, search and engines are on three separate machines. The appian-topology is configured accordingly. I wasn't sure where to put Kafka/ZooKeeper so I put them as the same machine as the engines.

Someone told me I need to run "checkengine.sh" but I don't seem to have this file in my Appian installation. I am running Appian 17.4. Also, I don't know if it should have been run on the gateway machine or engine machine or both? When engines start up, it reports they started successfully.

When I go to /suite/designer in a web browser, I get an error message "JBWEB000065: HTTP Status 404 - /suite/designer". This may be due to the earlier errors.

Your help would be much appreciated!

  Discussion posts and replies are publicly visible

Parents
  • Hi Adam,

    In regards to "checkengine.sh" this is a script used to check the status of your engines in versions of Appian 17.2 and prior. Since Appian 17.3 there have been some infrastructure changes and the checkengine script have been replaced by the status.sh script. Please refer to the following documentation on how to run the status script:

    docs.appian.com/.../Status_Script.html

    Furthermore, the issue you are facing, "Unable to acquire a Read connection. Safe to retry." could mean a couple of things.
    - There may be an issue with your engines and some/all of them may be down. Check this using the status script mentioned above.
    - Since you are running a distributed installation, there may be a network restriction which does not allow connection to the engines from the application server. You will need to ensure that all servers are accessible to each other and all engine ports are open and available. You can try and telnet from one server to another and vice-versa to confirm the connections.
    - Less likely in your situation however a possibility, the engines may be held busy with a very resource intensive operation and cannot respond to new requests. If you are still seeing this error and cannot access the environment this is most likely not the issue at hand.

    My recommendation would be to start with the status script to ensure that all of your engines are successfully running and then check connections between servers.
Reply
  • Hi Adam,

    In regards to "checkengine.sh" this is a script used to check the status of your engines in versions of Appian 17.2 and prior. Since Appian 17.3 there have been some infrastructure changes and the checkengine script have been replaced by the status.sh script. Please refer to the following documentation on how to run the status script:

    docs.appian.com/.../Status_Script.html

    Furthermore, the issue you are facing, "Unable to acquire a Read connection. Safe to retry." could mean a couple of things.
    - There may be an issue with your engines and some/all of them may be down. Check this using the status script mentioned above.
    - Since you are running a distributed installation, there may be a network restriction which does not allow connection to the engines from the application server. You will need to ensure that all servers are accessible to each other and all engine ports are open and available. You can try and telnet from one server to another and vice-versa to confirm the connections.
    - Less likely in your situation however a possibility, the engines may be held busy with a very resource intensive operation and cannot respond to new requests. If you are still seeing this error and cannot access the environment this is most likely not the issue at hand.

    My recommendation would be to start with the status script to ensure that all of your engines are successfully running and then check connections between servers.
Children
  • Hi, thanks and sorry for the late reply as I was away on vacation. First of all, as per my original post, I'm not sure if I should be running this status script on the engine machine or gateway machine or both?

    Secondly, my servers don't have telnet access enabled. Is there some kind of HTTP request I can make to the Appian engine server port using curl or wget, to test the engine can be reached?

    Thirdly, this is the first time I am starting Appian on this infrastructure, so I think it's impossible the engines are busy with a resource-intensive operation.