vmware: vm/volume inter-cluster migration failure during vm start operation #7311
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
server
CLOUDSTACK VERSION
4.15.2 or higher (and maybe lower as well)
CONFIGURATION
Advanced zone with VMware hypervisor
OS / ENVIRONMENT
N/A
SUMMARY
When VM is migrated across clusters during VM start operation, the migration which uses secondary storage and SSVM fails while deleting the original volume file at the source storage pool which is in a different cluster. Though VM eventually starts in the destination cluster on subsequent retries by the server, it leaves duplicate volume entries for the same volume in the DB, one with Destroy state. This is problematic in cases when VM is again manually migrated back to the original cluster/storage and storage GC expunges the volume in Destroy state. While expunging the volume entry in Destroy state, there could be a case that both the entries for volume point to the same VMDK file on the storage and it gets deleted.
The problem is that the code that tries to select the host for executing DeleteCommand returns the VM host ID which will be a host in the destination cluster at the point.
STEPS TO REPRODUCE
- Prepare a multi-cluster environment with CLUSTER scope storages
- Deploy a VM in cluster1, stop the VM after first boot
- Create a scenario that cluser1 is exceeding memory/CPU capacity if the VM is started and start the VM. CloudStack will initiate migration of the VM to cluster2.
- Observe, VM volume is migrated from source primary storage to secondary storage and then secondary storage to destination primary storage. But when CloudStack tries to expunge the original source volume entry it sends the command with a host from destination cluster and fails.
EXPECTED RESULTS
VM is migrated successfully and started on the first attempt
ACTUAL RESULTS
VM migration fails and VM start operation fails on the first attempt
Logs:
2023-03-05 12:30:21,019 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 12-1055249687688249398: Sending { Cmd , MgmtId: 32989308257179, via: 12(s-9-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b3c952c2-1110-4284-8c1b-8b3cd7f38b21","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.0.32.4/acs/secondary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-sec1","_role":"Image"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","path":"volumes/2/8/1054a78b938140f690c81b042d4c039b","volumeId":"8","vmName":"i-2-7-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/74c0c720111b4bb19a0a771fc3a992cb.vmdk"]}","format":"OVA","provisioningType":"THIN","id":"8","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"cdfe1462-6ce0-4949-9f43-93ffea720e1c","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1a8fc3ac-8e72-38a4-88d2-8538385b7d04","id":"2","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2/?ROLE=Primary&STOREUUID=1a8fc3ac-8e72-38a4-88d2-8538385b7d04","fullCloneFlag":"false","isManaged":"false"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","volumeId":"31","accountId":"2","format":"OVA","provisioningType":"THIN","id":"31","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"false","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:22,438 DEBUG [c.c.a.ApiServlet] (qtp961409111-21:ctx-3cf5e419) (logid:b97f0300) ===START=== 10.0.3.251 -- GET jobId=1ee8d65b-8c2f-42d2-8531-aa98327acd57&command=queryAsyncJobResult&response=json
2023-03-05 12:30:22,442 DEBUG [c.c.a.ApiServer] (qtp961409111-21:ctx-3cf5e419 ctx-57ee5987) (logid:b97f0300) CIDRs from which account 'Acct[2135d3cf-b9b5-11ed-b588-1e00ec00039b-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2023-03-05 12:30:22,451 DEBUG [c.c.a.ApiServlet] (qtp961409111-21:ctx-3cf5e419 ctx-57ee5987) (logid:b97f0300) ===END=== 10.0.3.251 -- GET jobId=1ee8d65b-8c2f-42d2-8531-aa98327acd57&command=queryAsyncJobResult&response=json
2023-03-05 12:30:25,171 DEBUG [c.c.a.t.Request] (AgentManager-Handler-18:null) (logid:) Seq 12-1055249687688249398: Processing: { Ans: , MgmtId: 32989308257179, via: 12, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"path":"25ebf915cac3437e85d8c0b05f6bcf29","accountId":"0","id":"0","directDownload":"false","deployAsIs":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:25,171 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 12-1055249687688249398: Received: { Ans: , MgmtId: 32989308257179, via: 12(s-9-VM), Ver: v1, Flags: 10, { CopyCmdAnswer } }
2023-03-05 12:30:25,184 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) getCommandHostDelegation: class org.apache.cloudstack.storage.command.DeleteCommand
2023-03-05 12:30:25,184 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) We are returning the default host to execute commands because the command is not of Copy type.
2023-03-05 12:30:25,186 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 12-1055249687688249399: Sending { Cmd , MgmtId: 32989308257179, via: 12(s-9-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b3c952c2-1110-4284-8c1b-8b3cd7f38b21","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.0.32.4/acs/secondary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-sec1","_role":"Image"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","path":"volumes/2/8/1054a78b938140f690c81b042d4c039b","volumeId":"8","vmName":"i-2-7-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/74c0c720111b4bb19a0a771fc3a992cb.vmdk"]}","format":"OVA","provisioningType":"THIN","id":"8","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:25,252 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) (logid:) Seq 12-1055249687688249399: Processing: { Ans: , MgmtId: 32989308257179, via: 12, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:25,252 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 12-1055249687688249399: Received: { Ans: , MgmtId: 32989308257179, via: 12(s-9-VM), Ver: v1, Flags: 10, { Answer } }
2023-03-05 12:30:25,288 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Updating resource Type = volume count for Account = 2 Operation = decreasing Amount = 1
2023-03-05 12:30:25,292 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Updating resource Type = primary_storage count for Account = 2 Operation = decreasing Amount = (20.00 GB) 21474836480
2023-03-05 12:30:25,435 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) getCommandHostDelegation: class org.apache.cloudstack.storage.command.DeleteCommand
2023-03-05 12:30:25,435 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) We are returning the default host to execute commands because the command is not of Copy type.
2023-03-05 12:30:25,438 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 2-7981222964630651033: Sending { Cmd , MgmtId: 32989308257179, via: 2(10.0.32.160), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9d530610-b7a2-379c-b41c-b7fd4184bd28","id":"5","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3/?ROLE=Primary&STOREUUID=9d530610-b7a2-379c-b41c-b7fd4184bd28","isManaged":"false"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","path":"74c0c720111b4bb19a0a771fc3a992cb","volumeId":"8","vmName":"i-2-7-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/74c0c720111b4bb19a0a771fc3a992cb.vmdk"]}","format":"OVA","provisioningType":"THIN","id":"8","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:25,439 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 2-7981222964630651033: Executing: { Cmd , MgmtId: 32989308257179, via: 2(10.0.32.160), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9d530610-b7a2-379c-b41c-b7fd4184bd28","id":"5","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3/?ROLE=Primary&STOREUUID=9d530610-b7a2-379c-b41c-b7fd4184bd28","isManaged":"false"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","path":"74c0c720111b4bb19a0a771fc3a992cb","volumeId":"8","vmName":"i-2-7-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/74c0c720111b4bb19a0a771fc3a992cb.vmdk"]}","format":"OVA","provisioningType":"THIN","id":"8","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2023-03-05 12:30:25,439 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-125:ctx-faa57283) (logid:9f3282fc) Seq 2-7981222964630651033: Executing request
2023-03-05 12:30:25,439 INFO [c.c.s.r.VmwareStorageProcessor] (DirectAgent-125:ctx-faa57283 10.0.32.160, job-201/job-202, cmd: DeleteCommand) (logid:1ee8d65b) Executing resource DeleteCommand: {"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9d530610-b7a2-379c-b41c-b7fd4184bd28","id":5,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri3/?ROLE=Primary&STOREUUID=9d530610-b7a2-379c-b41c-b7fd4184bd28","isManaged":false}},"name":"DATA-7","size":21474836480,"path":"74c0c720111b4bb19a0a771fc3a992cb","volumeId":8,"vmName":"i-2-7-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/74c0c720111b4bb19a0a771fc3a992cb.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":1,"hypervisorType":"VMware","directDownload":false,"deployAsIs":false}},"wait":0,"bypassHostMaintenance":false}
2023-03-05 12:30:25,501 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-125:ctx-faa57283 10.0.32.160, job-201/job-202, cmd: DeleteCommand) (logid:1ee8d65b) Unable to find datastore based on volume mount point 9d530610-b7a2-379c-b41c-b7fd4184bd28
2023-03-05 12:30:25,501 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-125:ctx-faa57283 10.0.32.160, job-201/job-202, cmd: DeleteCommand) (logid:1ee8d65b) [ignored]failed to get message for exception: Unable to find datastore based on volume mount point 9d530610-b7a2-379c-b41c-b7fd4184bd28
2023-03-05 12:30:25,501 ERROR [c.c.s.r.VmwareStorageProcessor] (DirectAgent-125:ctx-faa57283 10.0.32.160, job-201/job-202, cmd: DeleteCommand) (logid:1ee8d65b) delete volume failed due to Exception: java.lang.Exception
Message: Unable to find datastore based on volume mount point 9d530610-b7a2-379c-b41c-b7fd4184bd28
java.lang.Exception: Unable to find datastore based on volume mount point 9d530610-b7a2-379c-b41c-b7fd4184bd28
>---at com.cloud.storage.resource.VmwareStorageProcessor.deleteVolume(VmwareStorageProcessor.java:2611)
>---at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:143)
>---at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:63)
>---at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:569)
>---at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>---at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>---at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>---at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>---at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>---at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>---at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>---at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>---at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>---at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>---at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>---at java.base/java.lang.Thread.run(Thread.java:829)
...
...
2023-03-05 12:30:30,643 DEBUG [c.c.a.t.Request] (Work-Job-Executor-28:ctx-62b4ed71 job-201/job-202 ctx-35c751f4) (logid:1ee8d65b) Seq 2-7981222964630651035: Sending { Cmd , MgmtId: 32989308257179, via: 2(10.0.32.160), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":"7","name":"i-2-7-VM","bootloader":"HVM","state":"Starting","type":"User","cpus":"1","minSpeed":"500","maxSpeed":"1000","minRam":"(1.00 GB) 1073741824","maxRam":"(1.00 GB) 1073741824","hostName":"test0503","arch":"x86_64","os":"CentOS 5.3 (64-bit)","platformEmulator":"centos64Guest","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynamicallyScaleVm":"false","vncPassword":"nSl89lQnNow2MngNUwVPoA","params":{"dataDiskController":"osdefault","memoryOvercommitRatio":"1.0","Message.ReservedCapacityFreed.Flag":"false","nestedVirtualizationFlag":"false","cpuOvercommitRatio":"2.0","vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000","rootDiskController":"ide"},"uuid":"41b4eb0e-4cb3-4ed2-acc8-56bc2685ab0c","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6d65ee33-ed21-4827-814e-30fce5ec4155","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1a8fc3ac-8e72-38a4-88d2-8538385b7d04","id":"2","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2/?ROLE=Primary&STOREUUID=1a8fc3ac-8e72-38a4-88d2-8538385b7d04","isManaged":"false"}},"name":"ROOT-7","size":"(2.00 GB) 2147483648","path":"b98e7f2c7d534fa5b7c1cb7e52f3f570","volumeId":"30","vmName":"i-2-7-VM","accountId":"2","format":"OVA","provisioningType":"THIN","id":"30","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"diskSeq":"0","path":"b98e7f2c7d534fa5b7c1cb7e52f3f570","type":"ROOT","_details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","volumeSize":"(2.00 GB) 2147483648"}},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b3c952c2-1110-4284-8c1b-8b3cd7f38b21","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1a8fc3ac-8e72-38a4-88d2-8538385b7d04","id":"2","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-4587-v-M7-abhishek-kumar/ref-trl-4587-v-M7-abhishek-kumar-esxi-pri2/?ROLE=Primary&STOREUUID=1a8fc3ac-8e72-38a4-88d2-8538385b7d04","isManaged":"false"}},"name":"DATA-7","size":"(20.00 GB) 21474836480","path":"25ebf915cac3437e85d8c0b05f6bcf29","volumeId":"31","vmName":"i-2-7-VM","accountId":"2","format":"OVA","provisioningType":"THIN","id":"31","deviceId":"1","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false"}},"diskSeq":"1","path":"25ebf915cac3437e85d8c0b05f6bcf29","type":"DATADISK","_details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","volumeSize":"(20.00 GB) 21474836480"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"200","defaultNic":"true","pxeDisable":"false","nicUuid":"6c897ab1-f804-44fd-a9d1-a61b31fd6e79","details":{"MacAddressChanges":"true","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":"false","uuid":"ab619fb3-c43f-4da2-ad1d-ebb590a18b7c","ip":"10.1.1.220","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:0c:81:00:01","dns1":"10.0.32.1","dns2":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1225","isolationUri":"vlan://1225","isSecurityGroupEnabled":"false","name":"vSwitch1,,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"hostIp":"10.0.32.160","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
...
...
2023-03-05 12:30:38,722 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-9f6049b1 job-201 ctx-ef390c82) (logid:1ee8d65b) Complete async job-201, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"41b4eb0e-4cb3-4ed2-acc8-56bc2685ab0c","name":"test0503","displayname":"test0503","account":"admin","userid":"21372e1d-b9b5-11ed-b588-1e00ec00039b","username":"admin","domainid":"078a4cf2-b9b5-11ed-b588-1e00ec00039b","domain":"ROOT","created":"2023-03-05T09:41:39+0000","state":"Running","haenable":"false","zoneid":"af96e7c8-c675-4fae-a75b-2cfecd1e77da","zonename":"ref-trl-4587-v-M7-abhishek-kumar","hostid":"c2edaf6f-3e70-43ed-8d1d-4c4d6bc4e641","hostname":"10.0.32.160","templateid":"078ece4c-b9b5-11ed-b588-1e00ec00039b","templatename":"CentOS 5.3(64-bit) no GUI (vSphere)","templatedisplaytext":"CentOS 5.3(64-bit) no GUI (vSphere)","passwordenabled":"false","serviceofferingid":"a9c39fdb-9b5e-4f83-b445-b68df5e46e10","serviceofferingname":"Medium Instance","diskofferingid":"8fe596b1-287d-4d2a-81e8-33d9aa3f9163","diskofferingname":"Medium","cpunumber":"1","cpuspeed":"1000","memory":"1024","cpuused":"0%","networkkbsread":"(0 bytes) 0","networkkbswrite":"(0 bytes) 0","diskkbsread":"(2 bytes) 2","diskkbswrite":"(0 bytes) 0","memorykbs":"(1.00 MB) 1048576","memoryintfreekbs":"(0 bytes) 0","memorytargetkbs":"(1.00 MB) 1048576","diskioread":"(0 bytes) 0","diskiowrite":"(0 bytes) 0","guestosid":"07938c06-b9b5-11ed-b588-1e00ec00039b","rootdeviceid":"0","rootdevicetype":"ROOT","securitygroup":[],"nic":[{"id":"6c897ab1-f804-44fd-a9d1-a61b31fd6e79","networkid":"ab619fb3-c43f-4da2-ad1d-ebb590a18b7c","networkname":"l2","netmask":"255.255.255.0","gateway":"10.1.1.1","ipaddress":"10.1.1.220","isolationuri":"vlan://1225","broadcasturi":"vlan://1225","traffictype":"Guest","type":"Isolated","isdefault":"true","macaddress":"02:00:0c:81:00:01","secondaryip":[],"extradhcpoption":[]}],"hypervisor":"VMware","instancename":"i-2-7-VM","details":{"dataDiskController":"osdefault","memoryOvercommitRatio":"1.0","Message.ReservedCapacityFreed.Flag":"false","cpuOvercommitRatio":"2.0","rootDiskController":"ide"},"affinitygroup":[],"displayvm":"true","isdynamicallyscalable":"false","ostypeid":"07938c06-b9b5-11ed-b588-1e00ec00039b","osdisplayname":"CentOS 5.3 (64-bit)","tags":[],"jobid":"1ee8d65b-8c2f-42d2-8531-aa98327acd57","jobstatus":"0"}
In DB, volume entry with Destroy state:
MariaDB [cloud]> select id,pool_id,last_pool_id,name,path,volume_type,created,removed,state,chain_info from volumes where instance_id=7 and volume_type='ROOT' and removed is NULL;
+----+---------+--------------+--------+----------------------------------+-------------+---------------------+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | pool_id | last_pool_id | name | path | volume_type | created | removed | state | chain_info |
+----+---------+--------------+--------+----------------------------------+-------------+---------------------+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 7 | 5 | NULL | ROOT-7 | ROOT-7 | ROOT | 2023-03-05 09:41:39 | NULL | Destroy | {"diskDeviceBusName":"ide0:1","diskChain":["[9d530610b7a2379cb41cb7fd4184bd28] i-2-7-VM/ROOT-7.vmdk","[9d530610b7a2379cb41cb7fd4184bd28] 92516057597432de9e97770977467b39/92516057597432de9e97770977467b39.vmdk"]} |
| 30 | 2 | 5 | ROOT-7 | b98e7f2c7d534fa5b7c1cb7e52f3f570 | ROOT | 2023-03-05 12:18:15 | NULL | Ready | {"diskDeviceBusName":"ide0:1","diskChain":["[1a8fc3ac8e7238a488d28538385b7d04] i-2-7-VM/b98e7f2c7d534fa5b7c1cb7e52f3f570.vmdk"]} |
+----+---------+--------------+--------+----------------------------------+-------------+---------------------+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+