Storage-based Snapshots for KVM VMs #3724

slavkap · 2019-11-28T13:32:23Z

Description

Cloudstack, with KVM as a hypervisor, provides live VM snapshots only with volumes whose image format is QCOW.
This is a limitation for storage providers with disks in RAW format.

link to ML: http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201911.mbox/%3cCAA6FghF7eaY-A3XGN5zSwKVvp7zrcuNzv5nFAQKaF+et3zH-ag@mail.gmail.com%3e#archives

With this feature

CloudStack users will be able to do live consistent snapshots of all disks attached to the VM
Is using respective storage provider plugin implementation to take the snapshot
For NFS/Local storage - now this functionality is limited until KVM disk-only based snapshot of volumes instead of taking VM's full snapshot and extracting disks #5297 gets merged
You can create a VM snapshot without its memory

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Documentation

apache/cloudstack-documentation#153

Prerequisites

guest agent enabled - used to freeze/thaw the VM

How Has This Been Tested?

Environment:

CloudStack - management and agent version - main
OS for management and hypervisor - CentOS7
Hypervisor - KVM
Primary storages - NFS, StorPool, Local storage and Ceph.

Enable the global configuration setting - kvm.vmstoragesnapshot.enabled

Take snapshot:

Take VM snapshot without memory on running virtual machine
We have tested this feature with these storage providers - NFS, StorPool, Local storage and Ceph.
There is one smoke test, which creates VM, installs qemu-guest-agent, takes, reverts and deletes VM snapshot.

Revert snapshot

Stop the virtual machine and revert the VM snapshot

DaanHoogland · 2019-11-29T14:22:08Z

@blueorangutan package

andrijapanicsb · 2020-01-04T14:28:01Z

@blueorangutan package

blueorangutan · 2020-01-04T14:28:30Z

@andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-04T14:49:29Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-533

andrijapanicsb · 2020-01-04T15:56:09Z

@blueorangutan test

blueorangutan · 2020-01-04T15:56:29Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-01-05T00:16:16Z

Trillian test result (tid-694)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28199 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3724-t694-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshot_kvm.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 76 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
runTest	`Error`	0.00	test_vm_snapshot_kvm.py
test_hostha_enable_ha_when_host_disabled	`Error`	3.63	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	302.75	test_hostha_kvm.py

weizhouapache · 2020-01-07T10:34:36Z

@slavkap @andrijapanicsb
this is a nice feature.
I will review and test it . It might take some days, depends on my availability.

slavkap · 2020-01-09T08:05:03Z

Thank you @weizhouapache that you will spend time on this! I am available if you have any comments or questions.

andrijapanicsb · 2020-01-13T20:49:58Z

@blueorangutan test

blueorangutan · 2020-01-13T20:50:12Z

@andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

andrijapanicsb · 2020-01-13T22:19:51Z

@slavkap - nice stuff as already commented!

I see "lots" of dependency which has to be documented properly in upstream documentation (preferably) but also I would like to see it documented in the description of the global setting - "...requires qemu 1.6+ and the qemu-guest-agent installed inside a VM"
Do you know how well qemu-guest-agent is supported/possible on Windows guests - lot's of people will be running Windows VMs as well - covering just Linux workloads would be suboptimal.
You are mentioning QCOW format and behaviour about it. Ceph uses RAW format from Qemu/Libvirt perspective, but possibly is marked as QCOW2 in DB (I recall some hassle around DB format in past) - can you please elaborate about this - how does it work with Ceph vs other "real" QCOW2 formats? Store
Currently, due to someone noticing bugs in QCOW2 snapshots (or restore of) where the volume/filesystem of the guest gets corrupted - volume snapshots have been disabled for running VMs for KVM. @weizhouapache can you comment on this? I'm wondering if we might hit the same issue here as well
Did you guys test this on the busy VMs (IO and CPU heavy workloads)?

thx

wido

I left some comments about executing commands from inside Java

...c/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtFreezeThawVMCommandWrapper.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/KVMStorageProcessor.java

andrijapanicsb · 2020-01-14T11:20:20Z

moving to 4.15, I'm afraid we won't be able to test in time.

weizhouapache · 2020-01-14T11:35:07Z

4. urrently, due to someone noticing bugs in QCOW2 snapshots (or restore of) where the volume/filesystem of the guest gets corrupted - volume snapshots have been disabled for running VMs for KVM. @weizhouapache can you comment on this? I'm wondering if we might hit the same issue here as well

@andrijapanicsb as far as I remember, vm is paused not only when we create snapshot but also delete a snapshot. I think corrupted volume issue should be fixed.

slavkap · 2020-02-05T08:44:32Z

Hello @andrijapanicsb, sorry for the late response! Bellow you can find answers of your questions

I see "lots" of dependency which has to be documented properly in upstream documentation (preferably) but also I would like to see it documented in the description of the global setting - "...requires qemu 1.6+ and the qemu-guest-agent installed inside a VM"

We'll document the functionality, but could you please tell me what should be included? Also I will update the global setting with:

I would like to see it documented in the description of the global setting - "...requires qemu 1.6+ and the qemu-guest-agent installed inside a VM"

Do you know how well qemu-guest-agent is supported/possible on Windows guests - lot's of people will be running Windows VMs as well - covering just Linux workloads would be suboptimal.

For Windows freeze/thaw of guest agent is included in versions from 2013

You are mentioning QCOW format and behaviour about it. Ceph uses RAW format from Qemu/Libvirt perspective, but possibly is marked as QCOW2 in DB (I recall some hassle around DB format in past) - can you please elaborate about this - how does it work with Ceph vs other "real" QCOW2 formats? Store

Ceph is marked in DB as RAW format, but this doesn't affect snapshot and revert of Ceph's volumes, because we are using its implementation for this. For each primary storage we are using appropriate plugin with take/revert snapshot implementation.

Did you guys test this on the busy VMs (IO and CPU heavy workloads)?

We've completed our tests including on busy VMs with different primary storages. They've been successfully reverted and it works as expected.

moving to 4.15, I'm afraid we won't be able to test in time.

Unfortunately we'll need time to implement qemu-agent and qemu-monitor commands functionality in libvirt java api, and don't know when it will be accepted.

nvazquez · 2022-03-17T16:26:35Z

@blueorangutan package

blueorangutan · 2022-03-17T16:27:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-17T17:12:27Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2921

nvazquez · 2022-03-17T17:34:53Z

@blueorangutan test

blueorangutan · 2022-03-17T17:37:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-18T02:32:21Z

Trillian test result (tid-3656)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30801 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3724-t3656-kvm-centos7.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

rp- · 2022-03-22T13:45:15Z

I tested this PR today with the Linstor plugin and creating/reverting and deleting works with Linstor.

nvazquez · 2022-03-22T14:50:20Z

Thanks @rp- can you please approve on the Files changed tab -> Review changes -> Submit review?

rp-

Tested with Linstor primary storage:
create/revert and delete worked.

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/utils/qemu/QemuCommand.java

...c/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtFreezeThawVMCommandWrapper.java

nvazquez · 2022-03-28T18:25:03Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManager.java

    public static final ConfigKey<Boolean> BackupSnapshotAfterTakingSnapshot = new ConfigKey<Boolean>(Boolean.class, "snapshot.backup.to.secondary",  "Snapshots", "true",
            "Indicates whether to always backup primary storage snapshot to secondary storage. Keeping snapshots only on Primary storage is applicable for KVM + Ceph only.", false, ConfigKey.Scope.Global, null);

+    public static final ConfigKey<Boolean> VMsnapshotKVM = new ConfigKey<>(Boolean.class, "kvm.vmstoragesnapshot.enabled", "Snapshots", "false", "For live snapshot of virtual machine instance on KVM hypervisor without memory. Requieres qemu version 1.6+ (on NFS or Local file system) and qemu-guest-agent installed on guest VM", true, ConfigKey.Scope.Global, null);


Could it be renamed to something like 'VMStorageSnapshotKVM'? Could the scope be reduced to zone?

@nvazquez, I will rename it, but why limit this to a Zone?

@slavkap I was thinking it would make sense for admins to enable/disable the feature for a zone and not for all of them at the same time - just a suggestion

nvazquez

Thanks @slavkap - code LGTM

nvazquez · 2022-03-30T14:29:28Z

Hi @wido @GabrielBrascher @svenvogel would it be possible for you to test this PR on Ceph or Solidfire?

nvazquez · 2022-03-30T16:18:27Z

@blueorangutan package

blueorangutan · 2022-03-30T16:19:04Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-30T16:54:56Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3027

nvazquez · 2022-04-04T10:46:14Z

@blueorangutan test

blueorangutan · 2022-04-04T10:48:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-04-04T20:01:33Z

Trillian test result (tid-3805)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31833 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3724-t3805-kvm-centos7.zip
Smoke tests completed. 93 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2022-04-08T00:42:05Z

Merging based on approvals and tests results

rohityadavcloud · 2022-04-08T08:20:41Z

Fantastic, this is finally merged! Thank for your work and patience @slavkap - do raise a documentation PR to as required - https://github.com/apache/cloudstack-documentation

slavkap · 2022-04-08T11:10:38Z

yes, @rohityadavcloud 🎉 ❤️
Thank you all for your help!
I'll update the PR in docs - 153 - with the changes I made

Documentation fo pull request - apache/cloudstack#3724

* [WIP] Storage-based VM snapshots on KVM Documentation fo pull request - apache/cloudstack#3724 * Removed information for QCOW2 support * Fix reference * Fix section

svenvogel added component:managed-storage type:new-feature component:primary-storage labels Nov 28, 2019

DaanHoogland requested a review from GabrielBrascher November 29, 2019 14:09

GabrielBrascher added this to the 4.14.0.0 milestone Nov 29, 2019

andrijapanicsb requested review from rohityadavcloud and weizhouapache January 4, 2020 14:28

andrijapanicsb requested a review from wido January 13, 2020 21:40

wido previously requested changes Jan 14, 2020

View reviewed changes

andrijapanicsb modified the milestones: 4.14.0.0, 4.15.0.0 Jan 14, 2020

rohityadavcloud closed this Jan 29, 2020

rohityadavcloud reopened this Jan 29, 2020

DaanHoogland marked this pull request as draft May 7, 2020 08:59

DaanHoogland added the component:kvm label May 7, 2020

Skip tests if there is pool with NFS/Local

9cbe148

rp- approved these changes Mar 22, 2022

View reviewed changes

nvazquez reviewed Mar 28, 2022

View reviewed changes

slavkap added 2 commits March 30, 2022 16:27

address comments

996cc45

Merge branch 'main' into feature_kvmsnapshot

4518956

nvazquez approved these changes Mar 30, 2022

View reviewed changes

nvazquez merged commit 2b075ed into apache:main Apr 8, 2022

slavkap added a commit to slavkap/cloudstack-documentation that referenced this pull request Apr 12, 2022

[WIP] Storage-based VM snapshots on KVM

9c0f1ea

Documentation fo pull request - apache/cloudstack#3724

rohityadavcloud mentioned this pull request Oct 10, 2023

storage: allow VM snapshots without memory for KVM when global setting allows #8062

Merged

6 tasks

slavkap mentioned this pull request Aug 22, 2024

KVM Disk-only VM Snapshots #9524

Closed

Storage-based Snapshots for KVM VMs #3724

Storage-based Snapshots for KVM VMs #3724

Uh oh!

Conversation

slavkap commented Nov 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Documentation

Prerequisites

How Has This Been Tested?

Uh oh!

DaanHoogland commented Nov 29, 2019

Uh oh!

andrijapanicsb commented Jan 4, 2020

Uh oh!

blueorangutan commented Jan 4, 2020

Uh oh!

blueorangutan commented Jan 4, 2020

Uh oh!

andrijapanicsb commented Jan 4, 2020

Uh oh!

blueorangutan commented Jan 4, 2020

Uh oh!

blueorangutan commented Jan 5, 2020

Uh oh!

weizhouapache commented Jan 7, 2020

Uh oh!

slavkap commented Jan 9, 2020

Uh oh!

andrijapanicsb commented Jan 13, 2020

Uh oh!

blueorangutan commented Jan 13, 2020

Uh oh!

andrijapanicsb commented Jan 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wido left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrijapanicsb commented Jan 14, 2020

Uh oh!

weizhouapache commented Jan 14, 2020

Uh oh!

slavkap commented Feb 5, 2020

Uh oh!

nvazquez commented Mar 17, 2022

Uh oh!

blueorangutan commented Mar 17, 2022

Uh oh!

blueorangutan commented Mar 17, 2022

Uh oh!

nvazquez commented Mar 17, 2022

Uh oh!

blueorangutan commented Mar 17, 2022

Uh oh!

blueorangutan commented Mar 18, 2022

Uh oh!

rp- commented Mar 22, 2022

Uh oh!

nvazquez commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rp- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvazquez Mar 28, 2022

Choose a reason for hiding this comment

Uh oh!

slavkap Mar 29, 2022

Choose a reason for hiding this comment

Uh oh!

nvazquez Mar 29, 2022

Choose a reason for hiding this comment

slavkap commented Nov 28, 2019 •

edited

Loading

andrijapanicsb commented Jan 13, 2020 •

edited

Loading

nvazquez commented Mar 22, 2022 •

edited

Loading

nvazquez commented Mar 30, 2022 •

edited

Loading