General Questions

Expand all | Collapse all

Thick client and Blocked system-critical thread has been detected.

  • 1.  Thick client and Blocked system-critical thread has been detected.

     
    Posted 11-20-2019 04:50 AM
    Edited by Dmitriy Shubin 11-20-2019 05:14 AM
    Good day!

    I have a test cluster of two nodes. There is a distributed cache between these nodes. Persistence is included.
    I want to write a thick client in Java (get data direct from node).
    I wrote this code:

    package com.bercut.thick;

    import org.apache.ignite.Ignite;
    import org.apache.ignite.IgniteCache;
    import org.apache.ignite.Ignition;
    import org.apache.ignite.configuration.IgniteConfiguration;
    import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
    import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
    import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;

    import java.util.Arrays;

    public class ThickClientTest {

    public static void main (String[] args) {
    TcpDiscoverySpi spi = new TcpDiscoverySpi();
    TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
    ipFinder.setAddresses(Arrays.asList("192.168.16.31", "192.168.16.32"));
    spi.setIpFinder(ipFinder);

    IgniteConfiguration cfg = new IgniteConfiguration();

    cfg.setClientMode(true);
    cfg.setDiscoverySpi(spi);

    Ignite ignite = Ignition.start(cfg);

    IgniteCache<String, String> cache = ignite.getOrCreateCache("bercut");

    for (int i = 1; i <= 100; i++) {
    cache.put(Integer.toString(i), "1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890");
    }

    ignite.close();
    }

    }

    But at startup, the following error occurs:

    [15:29:55] __________ ________________
    [15:29:55] / _/ ___/ |/ / _/_ __/ __/
    [15:29:55] _/ // (7 7 // / / / / _/
    [15:29:55] /___/\___/_/|_/___/ /_/ /___/
    [15:29:55]
    [15:29:55] ver. 8.7.6#20190704-sha1:6449a674
    [15:29:55] 2019 Copyright(C) Apache Software Foundation
    [15:29:55]
    [15:29:55] Ignite documentation: http://gridgain.com
    [15:29:55]
    [15:29:55] Quiet mode.
    [15:29:55] ^-- Logging by 'JavaLogger [quiet=true, config=null]'
    [15:29:55] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
    [15:29:55]
    [15:29:55] OS: Windows 10 10.0 amd64
    [15:29:55] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13
    [15:29:55] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
    [15:29:55] Initial heap size is 256MB (should be no less than 512MB, use -Xms512m -Xmx512m).
    [15:29:55] Configured plugins:
    [15:29:55] ^-- None
    [15:29:55]
    [15:29:55] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
    [15:30:02] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
    [15:30:02] Security status [authentication=off, tls/ssl=off]
    [15:30:03] REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property.
    ноя 20, 2019 3:26:06 PM org.apache.ignite.logger.java.JavaLogger errorноя 20, 2019 3:26:06 PM org.apache.ignite.logger.java.JavaLogger errorSEVERE: Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=52s]ноя 20, 2019 3:26:06 PM java.util.logging.LogManager$RootLogger logSEVERE: Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1574252714044]]]class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1574252714044] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1832) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1827) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:232) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:296) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:220) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) at java.lang.Thread.run(Thread.java:748)

    Tell me what is wrong with the client or the settings (client/server)?

    Log from Server:
    [13:07:15,939][INFO][grid-nio-worker-tcp-comm-2-#26%L2Cache%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/192.168.16.31:37982, rmtAddr=shubin--note.office.bercut.ru/192.168.7.137:47100]
    [13:07:15,940][INFO][rest-#50%L2Cache%][TcpCommunicationSpi] TCP client created [client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2, bytesRcvd=78173, bytesSent=6623, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2, igniteInstanceName=L2Cache, finished=false, heartbeatTs=1574255235935, hashCode=793398, interrupted=false, runner=grid-nio-worker-tcp-comm-2-#26%L2Cache%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=0, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=2e17f9cf-4332-43f9-920c-47918e838b8f, consistentId=2e17f9cf-4332-43f9-920c-47918e838b8f, addrs=[0:0:0:0:0:0:0:1, 10.0.75.1, 127.0.0.1, 172.25.195.129, 192.168.7.137], sockAddrs=[/172.25.195.129:0, shubin--note.office.bercut.ru/192.168.7.137:0, /0:0:0:0:0:0:0:1:0, /10.0.75.1:0, /127.0.0.1:0], discPort=0, order=25, intOrder=14, lastExchangeTime=1574255199069, loc=false, ver=8.7.6#20190704-sha1:6449a674, isClient=true], connected=false, connectCnt=2, queueLimit=4096, reserveCnt=8, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=0, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=2e17f9cf-4332-43f9-920c-47918e838b8f, consistentId=2e17f9cf-4332-43f9-920c-47918e838b8f, addrs=[0:0:0:0:0:0:0:1, 10.0.75.1, 127.0.0.1, 172.25.195.129, 192.168.7.137], sockAddrs=[/172.25.195.129:0, shubin--note.office.bercut.ru/192.168.7.137:0, /0:0:0:0:0:0:0:1:0, /10.0.75.1:0, /127.0.0.1:0], discPort=0, order=25, intOrder=14, lastExchangeTime=1574255199069, loc=false, ver=8.7.6#20190704-sha1:6449a674, isClient=true], connected=false, connectCnt=2, queueLimit=4096, reserveCnt=8, pairedConnections=false], super=GridNioSessionImpl [locAddr=/192.168.16.31:37982, rmtAddr=shubin--note.office.bercut.ru/192.168.7.137:47100, createTime=1574255235935, closeTime=0, bytesSent=0, bytesRcvd=0, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1574255235935, lastSndTime=1574255235935, lastRcvTime=1574255235935, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@6ce1c90b, directMode=true], GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]], super=GridAbstractCommunicationClient [lastUsed=1574255235935, closed=false, connIdx=0]], duration=30902ms]
    [13:07:26,668][INFO][grid-nio-worker-tcp-comm-3-#27%L2Cache%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/192.168.16.31:47100, rmtAddr=/192.168.7.137:54577]
    [13:07:26,669][INFO][grid-nio-worker-tcp-comm-3-#27%L2Cache%][TcpCommunicationSpi] Received incoming connection from remote node while connecting to this node, rejecting [locNode=86060ccb-bc15-4f16-be8d-8a65036491a0, locNodeOrder=1, rmtNode=2e17f9cf-4332-43f9-920c-47918e838b8f, rmtNodeOrder=25]
    [13:07:27,511][INFO][tcp-disco-sock-reader-#20%L2Cache%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/192.168.7.137:54546, rmtPort=54546
    [13:07:33,035][INFO][db-checkpoint-thread-#77%L2Cache%][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modified) [checkpointBeforeLockTime=1ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=1ms, checkpointLockHoldTime=2ms, reason='timeout']
    [13:07:35,195][SEVERE][tcp-disco-msg-worker-#2%L2Cache%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=18s]
    [13:07:35,196][WARNING][tcp-disco-msg-worker-#2%L2Cache%][G] Thread [name="tcp-comm-worker-#1%L2Cache%", id=52, state=RUNNABLE, blockCnt=0, waitCnt=2093]

    [13:07:35,196][SEVERE][tcp-disco-msg-worker-#2%L2Cache%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=L2Cache, finished=false, heartbeatTs=1574255236207]]]
    class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=L2Cache, finished=false, heartbeatTs=1574255236207]
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1832)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1827)
    at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:232)
    at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:296)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2792)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7519)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2854)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7457)
    at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)
    [13:07:35,197][WARNING][tcp-disco-msg-worker-#2%L2Cache%][FailureProcessor] No deadlocked threads detected.
    [13:07:35,331][WARNING][tcp-disco-msg-worker-#2%L2Cache%][FailureProcessor] Thread dump at 2019/11/20 13:07:35 GMT
    Thread [name="sys-#8483%L2Cache%", id=8557, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@434d908c, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="sys-#8482%L2Cache%", id=8556, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@434d908c, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    ...

    Thread [name="srvc-deploy-#82%L2Cache%", id=142, state=WAITING, blockCnt=0, waitCnt=23]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@63d03a91, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    ...

    Thread [name="db-checkpoint-thread-#77%L2Cache%", id=133, state=TIMED_WAITING, blockCnt=0, waitCnt=2809]
    Lock [object=o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer@1579520c, ownerName=null, ownerId=-1]
    at java.lang.Object.wait(Native Method)
    at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.waitCheckpointEvent(GridCacheDatabaseSharedManager.java:3632)
    at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3179)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="wal-segment-syncer-#74%L2Cache%", id=130, state=TIMED_WAITING, blockCnt=0, waitCnt=167730]
    at java.lang.Thread.sleep(Native Method)
    at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7656)
    at o.a.i.i.processors.cache.persistence.wal.filehandle.FileHandleManagerImpl$WalSegmentSyncer.body(FileHandleManagerImpl.java:595)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="wal-file-archiver%L2Cache-#73%L2Cache%", id=129, state=WAITING, blockCnt=0, waitCnt=1]
    Lock [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage@3a9f69df, ownerName=null, ownerId=-1]
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.awaitSegment(SegmentCurrentStateStorage.java:72)
    at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.waitNextSegmentForArchivation(SegmentCurrentStateStorage.java:89)
    at o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.waitNextSegmentForArchivation(SegmentAware.java:78)
    at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1749)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="tcp-disco-sock-reader-#7%L2Cache%", id=118, state=RUNNABLE, blockCnt=1, waitCnt=1]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    - locked java.io.BufferedInputStream@5468f8f2
    at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:52)
    at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
    at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:42)
    at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:136)
    at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93)
    at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:9967)
    at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6535)
    at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)

    Thread [name="rest-#54%L2Cache%", id=106, state=TIMED_WAITING, blockCnt=10, waitCnt=27040]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b5e3941, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    Thread [name="rest-#53%L2Cache%", id=105, state=TIMED_WAITING, blockCnt=9, waitCnt=27045]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b5e3941, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="rest-#52%L2Cache%", id=104, state=TIMED_WAITING, blockCnt=38, waitCnt=27035]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b5e3941, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Thread [name="rest-#51%L2Cache%", id=103, state=TIMED_WAITING, blockCnt=49, waitCnt=27040]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b5e3941, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Thread [name="rest-#50%L2Cache%", id=102, state=WAITING, blockCnt=33, waitCnt=27038]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
    at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
    at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
    at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2956)
    at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2758)
    at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2717)
    at o.a.i.i.managers.communication.GridIoManager.send(GridIoManager.java:1655)
    at o.a.i.i.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730)
    at o.a.i.i.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1435)
    at o.a.i.i.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:665)
    at o.a.i.i.processors.task.GridTaskWorker.body(GridTaskWorker.java:537)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at o.a.i.i.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:809)
    at o.a.i.i.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:533)
    at o.a.i.i.IgniteComputeImpl.executeAsync0(IgniteComputeImpl.java:487)
    at o.a.i.i.IgniteComputeImpl.executeAsync(IgniteComputeImpl.java:467)
    at o.a.i.i.v.compute.VisorGatewayTask$VisorGatewayJob.execute(VisorGatewayTask.java:461)
    at o.a.i.i.processors.job.GridJobWorker$2.call(GridJobWorker.java:567)
    at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6843)
    at o.a.i.i.processors.job.GridJobWorker.execute0(GridJobWorker.java:561)
    at o.a.i.i.processors.job.GridJobWorker.body(GridJobWorker.java:490)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at o.a.i.i.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1124)
    at o.a.i.i.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1419)
    at o.a.i.i.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:665)
    at o.a.i.i.processors.task.GridTaskWorker.body(GridTaskWorker.java:537)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at o.a.i.i.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:809)
    at o.a.i.i.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:533)
    at o.a.i.i.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:513)
    at o.a.i.i.processors.rest.handlers.task.GridTaskCommandHandler.handleAsyncUnsafe(GridTaskCommandHandler.java:226)
    at o.a.i.i.processors.rest.handlers.task.GridTaskCommandHandler.handleAsync(GridTaskCommandHandler.java:162)
    at o.a.i.i.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:318)
    at o.a.i.i.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:99)
    at o.a.i.i.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:174)
    at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
    java.util.concurrent.ThreadPoolExecutor$Worker@6364cf3c
    Thread [name="rest-#49%L2Cache%", id=101, state=TIMED_WAITING, blockCnt=10, waitCnt=27033]
    Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@b5e3941, ownerName=null, ownerId=-1]
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    ...



    ------------------------------
    Dmitriy Shubin
    DB Developer
    Bercut
    ------------------------------


  • 2.  RE: Thick client and Blocked system-critical thread has been detected.

     
    Posted 11-20-2019 06:10 AM
    Try generate cluster config from WEB-console and then in ClientNodeCodeStartup.java -> main write this:

    public static void main(String[] args) throws Exception {
    Ignite ignite = Ignition.start(ClientConfigurationFactory.createConfiguration());

    IgniteCache<String, String> cache = ignite.getOrCreateCache("bercut");

    int rec_count = 100000;

    for (int i = 0; i < rec_count; ++i) {
    cache.put(String.valueOf(i), "1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890");
    }
    }

    Same problem :(

    ноя 20, 2019 5:06:57 PM org.apache.ignite.logger.java.JavaLogger errorноя 20, 2019 5:06:57 PM org.apache.ignite.logger.java.JavaLogger errorSEVERE: Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-comm-worker, blockedFor=11s]ноя 20, 2019 5:06:57 PM java.util.logging.LogManager$RootLogger logSEVERE: Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=L2Cache, finished=false, heartbeatTs=1574258805441]]]class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=L2Cache, finished=false, heartbeatTs=1574258805441] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1832) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1827) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:232) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:296) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2960) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2899) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) at java.lang.Thread.run(Thread.java:748)

    ------------------------------
    Dmitriy Shubin
    DB Developer
    Bercut
    ------------------------------



  • 3.  RE: Thick client and Blocked system-critical thread has been detected.

    Posted 11-20-2019 06:13 AM
    Hi Dmitriy,

    According to the logs I see that server node (192.168.16.31) rejected connection from the client node (192.168.7.137) since server node already in progress of establishing conection to the client node and seems like it's unable to do that. Could you please check your network configuration and ensure that server node has access to the client node and able to establish connection?

    ------------------------------
    Igor Belyakov
    Software Engineer
    GridGain
    ------------------------------



  • 4.  RE: Thick client and Blocked system-critical thread has been detected.

     
    Posted 11-20-2019 08:46 AM
    I see client node (192.168.7.137) in WEB-console:


    and 1-2 row sometimes add


    but client node (192.168.7.137) not in baseline. Maybe I need to add it? But how? Consistent ID changes on client start :(


    And when use this code all work well (thin client)
    :
    public static void main(String[] args) {
    ClientConfiguration cfg = new ClientConfiguration().setAddresses("192.168.16.31:10800", "192.168.16.32:10800");
    try (IgniteClient client = Ignition.startClient(cfg)) {

    ClientCache<String, String> cache = client.getOrCreateCache("bercut");

    for (int i = 0; i < 10000; ++i) {
    cache.put(String.valueOf(i), "123456789012345678901234567890123456789012345678901234567890");
    }
    }
    catch (ClientException e) {
    System.err.println(e.getMessage());
    }
    catch (Exception e) {
    System.err.format("Unexpected failure: %s\n", e);
    }
    }




    ------------------------------
    Dmitriy Shubin
    DB Developer
    Bercut
    ------------------------------



  • 5.  RE: Thick client and Blocked system-critical thread has been detected.

     
    Posted 11-21-2019 03:05 AM
    When I try run client on 192.168.16.31 (node 1) - all OK​
    When I try run client on 192.168.16.32 (node 2) - connection Rejected

    ------------------------------
    Dmitriy Shubin
    DB Developer
    Bercut
    ------------------------------