How to prevent Ignite from “dying” if first attempt to join the cluster failed?
up vote
1
down vote
favorite
I'm using docker-compose to start several containers with java web application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will join cluster no problem.
But if I start them simultaneously (that's how it works by default with docker-compose) they both will fail with exception and will never retry or get back.
class org.apache.ignite.IgniteException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501, errMsg="Remote host closed connection during handshake"]
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66)
at org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 41 more
2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol successfully stopped: TCP binary
2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No verification for local node leave has been received from coordinator (will stop node anyway).
I've implemented custom IpFinder so it will NOT return IPs of other nodes until Ignite is fully online. It helped to get rid of this exception. And once they both fully initialized they're trying to talk to each other, but nothing happens -- they wont join the same cluster anyway.
2018-11-22 14:45:25.974 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery accepted incoming connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.983 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery spawning a new thread for connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.985 INFO tcp-disco-sock-reader-#12 Log4JLogger Started serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Received ping request from the remote node [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished writing ping response [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.994 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392
UPD: Here is a simple project to help reproduce the issue
java spring docker-compose ignite
asked Nov 22 at 15:06
Sergey Karpushin
4341722
add a comment |
up vote
1
down vote
favorite
I'm using docker-compose to start several containers with java web application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will join cluster no problem.
But if I start them simultaneously (that's how it works by default with docker-compose) they both will fail with exception and will never retry or get back.
class org.apache.ignite.IgniteException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501, errMsg="Remote host closed connection during handshake"]
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66)
at org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 41 more
2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol successfully stopped: TCP binary
2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No verification for local node leave has been received from coordinator (will stop node anyway).
I've implemented custom IpFinder so it will NOT return IPs of other nodes until Ignite is fully online. It helped to get rid of this exception. And once they both fully initialized they're trying to talk to each other, but nothing happens -- they wont join the same cluster anyway.
2018-11-22 14:45:25.974 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery accepted incoming connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.983 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery spawning a new thread for connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.985 INFO tcp-disco-sock-reader-#12 Log4JLogger Started serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Received ping request from the remote node [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished writing ping response [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.994 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392
UPD: Here is a simple project to help reproduce the issue
java spring docker-compose ignite
asked Nov 22 at 15:06
Sergey Karpushin
4341722
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
1
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I'm using docker-compose to start several containers with java web application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will join cluster no problem.
But if I start them simultaneously (that's how it works by default with docker-compose) they both will fail with exception and will never retry or get back.
class org.apache.ignite.IgniteException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501, errMsg="Remote host closed connection during handshake"]
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66)
at org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 41 more
2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol successfully stopped: TCP binary
2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No verification for local node leave has been received from coordinator (will stop node anyway).
I've implemented custom IpFinder so it will NOT return IPs of other nodes until Ignite is fully online. It helped to get rid of this exception. And once they both fully initialized they're trying to talk to each other, but nothing happens -- they wont join the same cluster anyway.
2018-11-22 14:45:25.974 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery accepted incoming connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.983 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery spawning a new thread for connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.985 INFO tcp-disco-sock-reader-#12 Log4JLogger Started serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Received ping request from the remote node [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished writing ping response [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.994 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392
UPD: Here is a simple project to help reproduce the issue
java spring docker-compose ignite
asked Nov 22 at 15:06
Sergey Karpushin
4341722
I'm using docker-compose to start several containers with java web application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will join cluster no problem.
But if I start them simultaneously (that's how it works by default with docker-compose) they both will fail with exception and will never retry or get back.
class org.apache.ignite.IgniteException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501, errMsg="Remote host closed connection during handshake"]
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649)
at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66)
at org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.OutputStream.write(OutputStream.java:75)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 41 more
2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol successfully stopped: TCP binary
2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No verification for local node leave has been received from coordinator (will stop node anyway).
I've implemented custom IpFinder so it will NOT return IPs of other nodes until Ignite is fully online. It helped to get rid of this exception. And once they both fully initialized they're trying to talk to each other, but nothing happens -- they wont join the same cluster anyway.
2018-11-22 14:45:25.974 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery accepted incoming connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.983 INFO tcp-disco-srvr-#3 Log4JLogger TCP discovery spawning a new thread for connection [rmtAddr=/192.168.160.1, rmtPort=43392]
2018-11-22 14:45:25.985 INFO tcp-disco-sock-reader-#12 Log4JLogger Started serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Received ping request from the remote node [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.993 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished writing ping response [rmtNodeId=b940dd90-0b94-42ba-b09a-5e0c7c8599e4, rmtAddr=/192.168.160.1:43392, rmtPort=43392]
2018-11-22 14:45:25.994 INFO tcp-disco-sock-reader-#12 Log4JLogger Finished serving remote node connection [rmtAddr=/192.168.160.1:43392, rmtPort=43392
UPD: Here is a simple project to help reproduce the issue
java spring docker-compose ignite
java spring docker-compose ignite
asked Nov 22 at 15:06
Sergey Karpushin
4341722
asked Nov 22 at 15:06
Sergey Karpushin
4341722
edited Nov 23 at 0:47
asked Nov 22 at 15:06
Sergey Karpushin
4341722
asked Nov 22 at 15:06
Sergey Karpushin
4341722
asked Nov 22 at 15:06
Sergey Karpushin
4341722
4341722
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
1
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33
add a comment |
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
1
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
1
1
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433749%2fhow-to-prevent-ignite-from-dying-if-first-attempt-to-join-the-cluster-failed%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you file a ticket against Apache Ignite JIRA? This is not normal behavior.
– alamar
Nov 22 at 15:24
1
@alamar, FYI: JIRA task created issues.apache.org/jira/browse/IGNITE-10388 and also I've created small project to help reproduce the issue: github.com/skarpushin/issue20181122
– Sergey Karpushin
Nov 23 at 0:46
There is a VERY good chance that the cause of this issue is the same as discussed here: apache-ignite-users.70518.x6.nabble.com/…
– Sergey Karpushin
Nov 29 at 6:33