server: improve storage GC to skip expunging possible duplicate volumes #7313

shwstppr · 2023-03-06T08:59:33Z

Description

Storage GC deletes volumes that are in Destroy state. This PR adds a check in this action that is if two volume records (one in Destroy state and one in Ready state) point to exactly the same disk file then skip doing GC on such Destroy state volume record.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

shwstppr · 2023-03-06T14:05:33Z

@blueorangutan package

blueorangutan · 2023-03-06T14:06:03Z

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-03-06T14:54:27Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5683

DaanHoogland · 2023-03-06T15:12:44Z

@blueorangutan test matrix

blueorangutan · 2023-03-06T15:13:03Z

@DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-03-07T01:08:10Z

Trillian test result (tid-6262)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34389 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6262-xenserver-71.zip
Smoke tests completed. 99 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_06_deploy_vm_with_extraconfig_throws_exception_xenserver	`Error`	1.17	test_deploy_vm_extra_config_data.py
test_07_deploy_vm_with_extraconfig_xenserver	`Error`	1.15	test_deploy_vm_extra_config_data.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.12	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	1.15	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.17	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	1.12	test_vm_deployment_planner.py

blueorangutan · 2023-03-07T02:08:40Z

Trillian test result (tid-6263)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 38020 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6263-vmware-67u3.zip
Smoke tests completed. 99 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_04_deploy_vm_with_extraconfig_throws_exception_vmware	`Error`	1.21	test_deploy_vm_extra_config_data.py
test_05_deploy_vm_with_extraconfig_vmware	`Error`	1.20	test_deploy_vm_extra_config_data.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.19	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	1.23	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.22	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	1.20	test_vm_deployment_planner.py

blueorangutan · 2023-03-07T02:52:58Z

Trillian test result (tid-6264)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40558 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6264-kvm-centos7.zip
Smoke tests completed. 99 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_deploy_vm_with_extraconfig_kvm	`Error`	0.13	test_deploy_vm_extra_config_data.py
test_03_update_vm_with_extraconfig_kvm	`Error`	1.16	test_deploy_vm_extra_config_data.py
test_02_deploy_vm_on_specific_cluster	`Error`	1.12	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	1.16	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.16	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	1.15	test_vm_deployment_planner.py

rohityadavcloud

Didn't test this, LGTM - this would need testing (one concern that can be tested if, this doesn't introduce any regression in the GC logic such that valid volumes that must be GC'd and deleted are no longer deleted)

shwstppr · 2023-03-09T12:08:22Z

Changing base branch creates conflicts @rohityadavcloud @DaanHoogland

…icate Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2023-05-23T08:55:54Z

@blueorangutan package

blueorangutan · 2023-05-23T08:58:03Z

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-05-23T09:53:40Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6121

rohityadavcloud · 2023-05-25T09:59:11Z

@blueorangutan test

blueorangutan · 2023-05-25T10:00:03Z

@rohityadavcloud a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-05-25T10:02:05Z

Trillian Build Failed (tid-6589)

DaanHoogland · 2023-05-25T14:52:55Z

@blueorangutan LLtest

blueorangutan · 2023-05-25T14:54:03Z

@DaanHoogland a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-05-26T04:37:16Z

[LL]Trillian test result (tid-6544)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47715 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6544-kvm-centos7.zip
Smoke tests completed. 105 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	`Failure`	966.51	test_privategw_acl.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	356.44	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	303.64	test_vpc_redundant.py
test_01_redundant_vpc_site2site_vpn	`Failure`	516.61	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	1184.62	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	449.88	test_vpc_vpn.py

DaanHoogland · 2023-05-26T08:09:26Z

@blueorangutan test

blueorangutan · 2023-05-26T08:10:03Z

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-05-26T08:12:01Z

[SF] Trillian Build Failed (tid-6594)

DaanHoogland · 2023-05-26T08:41:01Z

@blueorangutan LLtest

blueorangutan · 2023-05-26T08:42:02Z

@DaanHoogland a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-05-26T22:45:06Z

[LL]Trillian test result (tid-6549)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47826 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6549-kvm-centos7.zip
Smoke tests completed. 106 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.03	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py
test_01_redundant_vpc_site2site_vpn	`Failure`	547.30	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	1181.42	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	465.71	test_vpc_vpn.py

DaanHoogland · 2023-05-30T08:02:48Z

@blueorangutan test matrix

blueorangutan · 2023-05-30T08:04:03Z

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

shwstppr · 2023-06-01T13:24:16Z

@blueorangutan package

blueorangutan · 2023-06-01T13:26:03Z

@shwstppr a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-06-01T14:39:48Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6067

DaanHoogland · 2023-06-02T10:39:55Z

@blueorangutan test matrix

blueorangutan · 2023-06-02T10:42:03Z

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-06-02T22:25:05Z

[SF] Trillian test result (tid-6664)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40805 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6664-xenserver-71.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-06-02T23:11:26Z

[SF] Trillian test result (tid-6665)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 43525 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6665-vmware-67u3.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-06-02T23:17:42Z

[SF] Trillian test result (tid-6666)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43842 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7313-t6666-kvm-centos7.zip
Smoke tests completed. 108 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-06-05T07:38:11Z

This seems ready to merge, I am just missing some validation and given the description it seems hard to reproduce/test.

shwstppr · 2023-06-05T07:47:23Z

This is just a defensive check and I don't think it would be simple to reproduce. At most, we can test that there is no regression in the GC logic. I've also tried to add multiple unit tests to prevent any unnecessary change in the behaviour

DaanHoogland · 2023-06-05T08:03:10Z

This is just a defensive check and I don't think it would be simple to reproduce. At most, we can test that there is no regression in the GC logic. I've also tried to add multiple unit tests to prevent any unnecessary change in the behaviour

Thanks, @shwstppr I think we did enough for regression testing, looking at the code change (scope), merging

boring-cyborg bot added the component:storage label Mar 6, 2023

shwstppr force-pushed the improve-storage-gc-vol-delete branch from 627eedc to c85e114 Compare March 6, 2023 12:13

DaanHoogland added this to the 4.18.0.0 milestone Mar 6, 2023

DaanHoogland added the Severity:Critical Critical bug label Mar 6, 2023

rohityadavcloud approved these changes Mar 9, 2023

View reviewed changes

shwstppr changed the base branch from 4.17 to main March 9, 2023 12:07

shwstppr changed the base branch from main to 4.17 March 9, 2023 12:07

DaanHoogland modified the milestones: 4.18.0.0, 4.18.1.0 Mar 10, 2023

shwstppr mentioned this pull request Mar 28, 2023

Fix ScaleVM to consider resize volume in any type of service offering #7359

Merged

12 tasks

shwstppr added 2 commits May 23, 2023 13:30

server: improve storage GC to not expunge volumes which could be dupl…

1b4a456

…icate Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

8a0564d

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr force-pushed the improve-storage-gc-vol-delete branch from c85e114 to 8a0564d Compare May 23, 2023 08:49

boring-cyborg bot added component:agent component:build component:integration labels May 23, 2023

shwstppr changed the base branch from 4.17 to 4.18 May 23, 2023 08:49

shwstppr marked this pull request as ready for review May 23, 2023 08:56

DaanHoogland approved these changes May 30, 2023

View reviewed changes

DaanHoogland merged commit 2d6a069 into apache:4.18 Jun 5, 2023

server: improve storage GC to skip expunging possible duplicate volumes #7313

server: improve storage GC to skip expunging possible duplicate volumes #7313

Uh oh!

Conversation

shwstppr commented Mar 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

shwstppr commented Mar 6, 2023

Uh oh!

blueorangutan commented Mar 6, 2023

Uh oh!

blueorangutan commented Mar 6, 2023

Uh oh!

DaanHoogland commented Mar 6, 2023

Uh oh!

blueorangutan commented Mar 6, 2023

Uh oh!

blueorangutan commented Mar 7, 2023

Uh oh!

blueorangutan commented Mar 7, 2023

Uh oh!

blueorangutan commented Mar 7, 2023

Uh oh!

rohityadavcloud left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Mar 9, 2023

Uh oh!

shwstppr commented May 23, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

blueorangutan commented May 23, 2023

Uh oh!

rohityadavcloud commented May 25, 2023

Uh oh!

blueorangutan commented May 25, 2023

Uh oh!

blueorangutan commented May 25, 2023

Uh oh!

DaanHoogland commented May 25, 2023

Uh oh!

blueorangutan commented May 25, 2023

Uh oh!

blueorangutan commented May 26, 2023

Uh oh!

DaanHoogland commented May 26, 2023

Uh oh!

blueorangutan commented May 26, 2023

Uh oh!

blueorangutan commented May 26, 2023

Uh oh!

DaanHoogland commented May 26, 2023

Uh oh!

blueorangutan commented May 26, 2023

Uh oh!

blueorangutan commented May 26, 2023

Uh oh!

DaanHoogland commented May 30, 2023

Uh oh!

blueorangutan commented May 30, 2023

Uh oh!

shwstppr commented Jun 1, 2023

Uh oh!

blueorangutan commented Jun 1, 2023

Uh oh!

blueorangutan commented Jun 1, 2023

Uh oh!

DaanHoogland commented Jun 2, 2023

Uh oh!

blueorangutan commented Jun 2, 2023

Uh oh!

blueorangutan commented Jun 2, 2023

shwstppr commented Mar 6, 2023 •

edited

Loading