Skip to main content

clouddriver-caching errors | ClusteredAgentScheduler - Unable to run agents

Issue

When attempting to run agents, the following errors can be found in the clouddriver-caching logs:

Exception: ERROR 1 --- [ClusteredAgentScheduler-0] c.n.s.c.r.c.ClusteredAgentScheduler : Unable to run agents redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool redis.clients.jedis.util.Pool.getResource(Pool.java:59) ~[jedis-3.1.0.jar:na] redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[jedis-3.1.0.jar:na] com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:60) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:26) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.kork.jedis.JedisClientDelegate.withCommandsClient(JedisClientDelegate.java:54) ~[kork-jedis-7.41.0.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquireRunKey(ClusteredAgentScheduler.java:183) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquire(ClusteredAgentScheduler.java:136) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.runAgents(ClusteredAgentScheduler.java:163) ~[cats-redis-GCSFIX.jar:na] com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.run(ClusteredAgentScheduler.java:156) ~[cats-redis-GCSFIX.jar:na] java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na] java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na] java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na] java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na] java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na] java.base/java.lang.Thread.run(Thread.java:834) ~[na:na] Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host redis-clouddriver-master:6379 redis.clients.jedis.Connection.connect(Connection.java:204) ~[jedis-3.1.0.jar:na] redis.clients.jedis.BinaryClient.connect(BinaryClient.java:100) ~[jedis-3.1.0.jar:na] redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1866) ~[jedis-3.1.0.jar:na] redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:117) ~[jedis-3.1.0.jar:na] org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889) ~[commons-pool2-2.7.0.jar:2.7.0] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424) ~[commons-pool2-2.7.0.jar:2.7.0] org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[commons-pool2-2.7.0.jar:2.7.0] redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[jedis-3.1.0.jar:na] \t... 14 common frames omitted Caused by: java.net.SocketTimeoutException: connect timed out java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na] java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na] java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:na] java.base/java.net.Socket.connect(Socket.java:609) ~[na:na] redis.clients.jedis.Connection.connect(Connection.java:181) ~[jedis-3.1.0.jar:na] \t... 21 common frames omitted

Cause

Rare Redis issue where when two pods try to start at the same time.  Because they start at the same time, they end up competing for shared resources, and only one pod ends up starting.

AIDA logo
AIDA logo

Harness AIDA Chatbot

AI Development Assistant


Today, March 16, 9:52am

AIDA logo

Accelerate your software delivery with the powerful capabilities of Harness’s Platform.

AIDA logo

How can I help?

Log into your Harness Account to access AIDA