Hi , We setup a distributed system on our app servers. Please find th

21:27:10,124 INFO [stdout] (Appian PoolManager.CheckInvalidConnections - 1164) 2016-03-01 21:27:10,124 [Appian PoolManager.CheckInvalidConnections - 1164] ERROR com.appiancorp.kougar.driver.ipc.IpcConnection - Could not initialize IpcConnection (forums (macpro.cms.gov:5000)), closing immediately.
21:27:10,216 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /10.243.4.245:59829. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
21:27:10,216 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 3a661cb8-dfea-11e5-a7f9-493e4dbc1ba8
21:27:10,217 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 3a661cb8-dfea-11e5-a7f9-493e4dbc1ba8
21:27:17,066 INFO [stdout] (Appian PoolManager.CheckInvalidConnections - 1165) 2016-03-01 21:27:17,066 [Appian PoolManager.CheckInvalidConnections - 1165] ERROR com.appiancorp.kougar.driver.ipc.IpcConnection - Could not initialize IpcConnection (collaboration-statistics (macpro.cms.gov:5010)), closing immediately.

org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponentDescription$5$1.processInvocation(MessageDrivenComponentDescription.java:203)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:185)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
20:49:18,773 INFO [stdout] (default-threads - 5)           at com.appiancorp.ra.workpoller.UnattendedRequestListener$$$view2.onMessage(Unknown Source)
20:49:18,773 INFO [stdout] (default-threads - 5)           at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source)
20:49:18,773 INFO [stdout] (default-threads - 5)           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
20:49:18,773 INFO [stdout] (default-threads - 5)           at java.lang.reflect.Method.invoke(Method.java:497)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:140)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)
20:49:18,773 INFO [stdout] (default-threads - 5)           at com.sun.proxy.$Proxy495.onMessage(Unknown Source)
20:49:18,773 INFO [stdout] (default-threads - 5)           at com.appiancorp.ra.workpoller.WorkItem.run(WorkItem.java:54)
20:49:18,773 INFO [stdout] (default-threads - 5)           at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:218)
20:49:18,773 INFO [stdout] (default-threads - 5)           ... 6 more
20:49:18,773 INFO [stdout] (default-threads - 5) Caused by: java.lang.RuntimeException: Could not prepare email for: CompatibleEmailRecipients{locale=en_US, timeZone=GMT, calendarId=gregorian, emailAddresses=[ssolige@qssinc.com,MACPRO_Operations_Team@qssinc.com]}

Above errors are from other 2 servers

The attached server.log does not show anything regarding the errors you mention in your subsequent posts. Can you provide the logs from the machines which contain the messages in the 2 subsequent posts?

The first subsequent post shows something relating to the engines, but I'd need to see that log in order to diagnose further.

With regards to distributed setups and copy data, you need to copy the engine data to each engine machine so that the engines are in sync. The engine data is contained within the .kdb files.

previous log is after fresh restart of Jboss, Please find all server logs from all 3 servers before restarting. Thank you

serverapp1.log

serverapp2.log

serverapp3.log

serverapp1 seems to start up just fine, and nothing seems to have issues there.

serverapp2 appears to have multiple shutdowns and restarts logged. I see that there appears to be a connection issue between serverapp2 and wherever it is configured to connect with regards to the engines. I see these messages regarding the engines:

Caused by: java.net.ConnectException: Connection refused

java.net.SocketException: Connection reset

All of your machines in your distributed setup need to be able to connect and communicate with one another. Are all 3 machines added to each machine's hosts file?

serverapp3 has similar errors as serverapp2 with regards to connections. Your machines do not appear to be able to connect with one another. The errors in the logs tell you where it is trying to connect to with failure:

ERROR com.appiancorp.kougar.driver.ipc.IpcConnection - Could not initialize IpcConnection (content (yourHOST:5008)), closing immediately.

I would recommend testing telnet commands to these host names and ports to verify if you can connect. The ports may be closed or occupied, or you may not even be able to communicate with the host until you update your hosts files.

Thank you Sean, Yes connection is refused. Can i know where all i need to do changes and what to do in order to connect them. Thank you

You can add the IP address, FQDN, and short hand name for your machines in the hosts file. This is a plain text file that is used by the operating system to map IP addresses to host names.

Windows default location: C:\\Windows\\System32\\drivers\\etc

Unix default location: /etc/

To modify it, read the instructions within the hosts file itself. It provides you with what syntax you require, and includes an example.

You can also refer to our documentation below in the second section called Network Configuration:

forum.appian.com/.../Initial_Setup_for_Advanced_Multi-Server_Configuration.html

Thank you , let me try.

Sean, now i don't see any errors but i can see blank screen after logging in. i restarted Apache in webservers too. No problem with them