Externalize KVM HA heartbeat frequency #6892

stephankruggg · 2022-11-09T19:46:34Z

Description

Currently, the heartbeat frequency of the KVM HA is fixed at 60,000 milliseconds. This property was externalized as kvm.heartbeat.update.frequency in agent.properties.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

How Has This Been Tested?

I applied the changes in a local lab, changed the frequency, and checked if heartbeats were occurring at the frequency I specified.

sonarqubecloud · 2022-11-09T20:53:36Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

50.0% Coverage
0.0% Duplication

DaanHoogland

code LGTM

blueorangutan · 2022-12-01T16:25:37Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4727

DaanHoogland · 2022-12-01T16:56:38Z

@blueorangutan test

blueorangutan · 2022-12-01T16:57:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-12-01T17:34:59Z

Trillian Build Failed (tid-5332)

blueorangutan · 2022-12-02T09:02:45Z

Trillian Build Failed (tid-5342)

blueorangutan · 2022-12-02T15:00:51Z

Trillian Build Failed (tid-5375)

DaanHoogland · 2022-12-02T15:07:43Z

@stephankruggg in my test env I keep seeing this:

"2022-12-02 14:36:53 Adding KVM host pr6892-t5375-kvm-centos7-kvm1.", "🙈 Error: (HTTP 530, error code 9999) Could not add host at [http://10.0.33.209/] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!]

can you explain that?

stephankruggg · 2022-12-02T18:11:00Z

@stephankruggg in my test env I keep seeing this:

"2022-12-02 14:36:53 Adding KVM host pr6892-t5375-kvm-centos7-kvm1.", "🙈 Error: (HTTP 530, error code 9999) Could not add host at [http://10.0.33.209/] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!]

can you explain that?

Hi, @DaanHoogland, I have not observed this error during my tests. The changes do not seem to affect certificates in KVM, thus, could you send some of the logs?

DaanHoogland · 2022-12-05T10:15:57Z

management:

022-12-02 14:36:53,653 DEBUG [c.c.a.ApiServlet] (qtp262366552-16:ctx-738fd6f7) (logid:944d439c) ===START===  10.0.35.111 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&clusterid=a94
89c75-dbcc-4a51-8081-a76fdff1345b&command=addHost&hypervisor=KVM&podid=1ca6f4d3-fe8b-4eaf-be94-915fcd313a08&response=json&url=http%3A%2F%2F10.0.33.209&username=root&zoneid=ddb1c828-a5eb-4010-83c6-cd9a681a9972&signature=VVRH4DxK1Kow9bs%2B
jnuzGlWRam0%3D
2022-12-02 14:36:53,657 DEBUG [c.c.a.ApiServer] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"e1e1ae59-724d-11ed-9f35-1e002d000402"}]' is allowed to 
perform API calls: 0.0.0.0/0,::/0
2022-12-02 14:36:53,659 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) Account [Account [{"accountName":"admin","id":2,"uuid":"e1e1ae59-724d-11ed-9f35-1e002d000402"}]] is Roo
t Admin or Domain Admin, all APIs are allowed.
2022-12-02 14:36:53,660 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [addHost] for
 user [User {"username":"admin","uuid":"6ca2d90b-98d3-4e6a-ba3b-328507d9044c"}.] as allowed.
2022-12-02 14:36:53,660 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2022-12-02 14:36:53,661 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) API rate limiting is disabled. We will not use ApiRateLimitService.
2022-12-02 14:36:53,667 INFO  [c.c.r.ResourceManagerImpl] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Trying to add a new host at http://10.0.33.209 in data center 1
2022-12-02 14:36:53,901 WARN  [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Failed to authenticate with ssh key
2022-12-02 14:36:53,902 ERROR [c.c.h.k.d.LibvirtServerDiscoverer] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Failed to authenticate with ssh key
2022-12-02 14:36:53,927 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Executing cmd: ls /dev/kvm
2022-12-02 14:36:55,159 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH command: ls /dev/kvm
SSH command output:/dev/kvm


2022-12-02 14:36:55,171 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-12-02 14:36:56,054 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-cb611b95) (logid:78fc051a) HA health check task is running...
2022-12-02 14:36:57,537 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:

2022-12-02 14:36:57,597 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:36:58,864 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-ca157098) (logid:c345ffc8) Skip capacity scan as there is no Primary Storage in 'Up' state
2022-12-02 14:36:59,921 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory

2022-12-02 14:36:59,921 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:37:00,055 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-b9a7c1a3) (logid:b4f987fc) HA health check task is running...
2022-12-02 14:37:02,202 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory

2022-12-02 14:37:02,202 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:37:04,058 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-4b9064e6) (logid:88af62e1) HA health check task is running...
2022-12-02 14:37:04,441 DEBUG [c.c.u.s.SSHCmdHelper] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory

2022-12-02 14:37:04,443 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
2022-12-02 14:37:04,443 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:147)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:144)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:98)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-12-02 14:37:04,444 DEBUG [c.c.r.ResourceManagerImpl] (qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Could not add host at [http://10.0.33.209] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:147)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:144)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:98)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        ... 60 more
etc ...

I must admit that I do not directly relate this error to your code either, but other PRs are passing so I can not attribute it to the environment either.

the kvm actually fails earlier than with the agent:

Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05 10:08:48.807+0000: 19923: info : libvirt version: 4.5.0, package: 36.el7_9.5 (CentOS BuildSystem <http://bugs.centos.org>, 2021-04-28-13:32:22, x86-01.bsys.centos.
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05 10:08:48.807+0000: 19923: info : hostname: pr6892-t5375-kvm-centos7-kvm1
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05 10:08:48.807+0000: 19923: error : virNetTLSContextCheckCertFile:112 : Cannot read private key '/etc/pki/libvirt/private/serverkey.pem': No such file or directory
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 systemd[1]: libvirtd.service: main process exited, code=exited, status=6/NOTCONFIGURED
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 systemd[1]: Failed to start Virtualization daemon.
-- Subject: Unit libvirtd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit libvirtd.service has failed.
-- 
-- The result is failed.

blueorangutan · 2022-12-14T09:39:29Z

Packaging result: ✖️ el7 ✔️ el8 ✖️ debian ✔️ suse15. SL-JID 4920

blueorangutan · 2022-12-16T13:23:05Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4995

DaanHoogland · 2022-12-19T09:12:32Z

@blueorangutan test

blueorangutan · 2022-12-19T09:13:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-12-19T09:52:25Z

Trillian Build Failed (tid-5553)

blueorangutan · 2022-12-19T11:47:49Z

Trillian Build Failed (tid-5564)

blueorangutan · 2023-10-31T14:08:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-10-31T15:07:06Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7580

DaanHoogland · 2023-11-01T09:12:30Z

@blueorangutan test alma9 kvm-alma9

blueorangutan · 2023-11-01T09:14:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

blueorangutan · 2023-11-01T22:02:10Z

[SF] Trillian test result (tid-8190)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 44469 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6892-t8190-kvm-alma9.zip
Smoke tests completed. 115 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-11-02T08:26:13Z

@GaOrtiga @stephankruggg , is this ready for merge?

GaOrtiga · 2023-11-03T16:22:42Z

@GaOrtiga @stephankruggg , is this ready for merge?

@DaanHoogland, Yes.

github-actions · 2023-11-06T07:56:46Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2023-11-06T08:22:08Z

ai, conflicts, Can you have a look @GaOrtiga (cc @stephankruggg )

Co-authored-by: dahn <daan.hoogland@gmail.com>

….java Co-authored-by: dahn <daan.hoogland@gmail.com>

GaOrtiga · 2023-11-14T17:57:09Z

ai, conflicts, Can you have a look @GaOrtiga (cc @stephankruggg )

Done

DaanHoogland · 2023-11-15T08:11:22Z

@blueorangutan package

blueorangutan · 2023-11-15T08:14:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-11-15T14:50:30Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7762

DaanHoogland · 2023-11-15T15:16:00Z

@blueorangutan test

blueorangutan · 2023-11-15T15:18:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-11-16T04:38:11Z

[SF] Trillian test result (tid-8329)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46510 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6892-t8329-kvm-centos7.zip
Smoke tests completed. 116 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	556.04	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	680.24	test_kubernetes_clusters.py
test_08_migrate_vm	`Error`	44.84	test_vm_life_cycle.py

Co-authored-by: Stephan Krug <stephan.krug@scclouds.com.br> Co-authored-by: GaOrtiga <49285692+GaOrtiga@users.noreply.github.com> Co-authored-by: dahn <daan.hoogland@gmail.com>

boring-cyborg bot added component:agent component:kvm labels Nov 9, 2022

DaanHoogland approved these changes Nov 10, 2022

View reviewed changes

apache deleted a comment from blueorangutan Dec 1, 2022

rohityadavcloud approved these changes Dec 12, 2022

View reviewed changes

DaanHoogland added the status:needs-testing label Oct 31, 2023

github-actions bot added the status:has-conflicts label Nov 6, 2023

shwstppr closed this Nov 8, 2023

shwstppr reopened this Nov 8, 2023

Stephan Krug and others added 5 commits November 14, 2023 14:34

Externalize frequency

b8344d5

Fix code smell and move import

53d17ae

Externalize additional KVM HA properties

31e6df9

Apply suggestions from code review

6a86a7b

Co-authored-by: dahn <daan.hoogland@gmail.com>

Update agent/src/main/java/com/cloud/agent/properties/AgentProperties…

5b68672

….java Co-authored-by: dahn <daan.hoogland@gmail.com>

GaOrtiga force-pushed the externalize-KVM-HA-heartbeat-frequency branch from 46182fc to 5b68672 Compare November 14, 2023 17:51

github-actions bot removed the status:has-conflicts label Nov 14, 2023

DaanHoogland assigned blueorangutan Nov 15, 2023

DaanHoogland merged commit 267a457 into apache:main Nov 16, 2023

Externalize KVM HA heartbeat frequency #6892

Externalize KVM HA heartbeat frequency #6892

Uh oh!

Conversation

stephankruggg commented Nov 9, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

How Has This Been Tested?

Uh oh!

sonarqubecloud bot commented Nov 9, 2022

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Dec 1, 2022

Uh oh!

DaanHoogland commented Dec 1, 2022

Uh oh!

blueorangutan commented Dec 1, 2022

Uh oh!

blueorangutan commented Dec 1, 2022

Uh oh!

blueorangutan commented Dec 2, 2022

Uh oh!

blueorangutan commented Dec 2, 2022

Uh oh!

DaanHoogland commented Dec 2, 2022

Uh oh!

stephankruggg commented Dec 2, 2022

Uh oh!

DaanHoogland commented Dec 5, 2022

Uh oh!

blueorangutan commented Dec 14, 2022

Uh oh!

blueorangutan commented Dec 16, 2022

Uh oh!

DaanHoogland commented Dec 19, 2022

Uh oh!

blueorangutan commented Dec 19, 2022

Uh oh!

blueorangutan commented Dec 19, 2022

Uh oh!

blueorangutan commented Dec 19, 2022

Uh oh!

blueorangutan commented Oct 31, 2023

Uh oh!

blueorangutan commented Oct 31, 2023

Uh oh!

DaanHoogland commented Nov 1, 2023

Uh oh!

blueorangutan commented Nov 1, 2023

Uh oh!

blueorangutan commented Nov 1, 2023

Uh oh!

DaanHoogland commented Nov 2, 2023

Uh oh!

GaOrtiga commented Nov 3, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

DaanHoogland commented Nov 6, 2023

Uh oh!

GaOrtiga commented Nov 14, 2023

Uh oh!

DaanHoogland commented Nov 15, 2023

Uh oh!

blueorangutan commented Nov 15, 2023

Uh oh!

blueorangutan commented Nov 15, 2023

Uh oh!

DaanHoogland commented Nov 15, 2023

Uh oh!

blueorangutan commented Nov 15, 2023

Uh oh!

blueorangutan commented Nov 16, 2023

Uh oh!

Reviewers

Assignees