Allow additional configuration metadata to VMs #3510

anuragaw · 2019-07-23T10:19:53Z

Currently, CloudStack does not allow additional configuration metadata to guest VMs during deployment. This feature aims to provide operators the functionality to pass arbitrary additional configuration metadata that will be included everytime the guest VM is started.

FS document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enable+additional+configuration+metadata+to+virtual+machines

Description

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

anuragaw · 2019-07-23T10:33:05Z

@blueorangutan package

blueorangutan · 2019-07-23T10:34:08Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-23T10:54:20Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-174

borisstoyanov · 2019-07-23T14:02:17Z

@blueorangutan test matrix

blueorangutan · 2019-07-23T14:03:09Z

@borisstoyanov a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-07-24T00:09:48Z

Trillian test result (tid-206)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34298 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3510-t206-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_extra_config_data.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 77 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_deploy_vm__with_extraconfig_kvm	`Error`	0.27	test_deploy_vm_extra_config_data.py

blueorangutan · 2019-07-24T05:05:22Z

Trillian test result (tid-207)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52206 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3510-t207-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 74 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_reboot_router	`Error`	356.40	test_network.py
test_02_list_snapshots_with_removed_data_store	`Error`	1.19	test_snapshots.py
test_01_volume_usage	`Error`	10.45	test_usage.py
test_05_rvpc_multi_tiers	`Failure`	587.76	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	630.44	test_vpc_redundant.py

rohityadavcloud

Did not test it, lgtm.

wido · 2019-07-24T10:03:23Z

This seems to be Xen specific, right?

What can this do in addition to user-data through cloud-init?

borisstoyanov · 2019-07-24T11:53:29Z

This is not just Xen specific but for vmware and kvm as well @wido, it only adds extra configuration to a VM it has nothing to do with user data and cloud-init.

wido · 2019-07-24T11:56:57Z

@borisstoyanov I got confused by some keys I noticed.

I see it also works for KVM, but what kind of keys would somebody want to configure with KVM?

borisstoyanov

@wido whatever is not managed by cloudstack :) numa, hugepages etc..
@anuragaw can you have a look on the failing tests or you want me to?

anuragaw · 2019-07-29T05:29:33Z

I'll take a look @borisstoyanov

svenvogel · 2019-08-10T10:38:42Z

@anuragaw Hi. Small question. how does this work in practise? do we speak about the systemvm?

borisstoyanov

@anuragaw can you address the merge conflicts pls

anuragaw · 2019-09-11T19:20:17Z

@blueorangutan package

blueorangutan · 2019-09-11T19:21:02Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-11T20:20:54Z

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-273

anuragaw · 2019-09-11T22:20:46Z

@blueorangutan package

blueorangutan · 2019-09-11T22:21:01Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-11T22:41:35Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-275

anuragaw · 2019-09-12T17:41:16Z

@blueorangutan test

blueorangutan · 2019-09-12T17:42:04Z

@anuragaw a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

…tails

anuragaw · 2019-12-21T15:49:27Z

Needed rebasing after merge conflicts.

@blueorangutan package

blueorangutan · 2019-12-21T15:50:28Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-12-21T16:09:33Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-500

borisstoyanov · 2019-12-31T09:37:13Z

@blueorangutan test

blueorangutan · 2019-12-31T09:38:11Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-12-31T20:11:41Z

Trillian test result (tid-676)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36005 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3510-t676-kvm-centos7.zip
Smoke tests completed. 78 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

anuragaw · 2020-01-03T11:05:54Z

@DaanHoogland - can you please take a look at the last commit for code review. Thanks!!

@blueorangutan package

blueorangutan · 2020-01-03T11:06:25Z

@anuragaw a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-03T11:25:58Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-521

anuragaw · 2020-01-03T11:26:57Z

@blueorangutan test

blueorangutan · 2020-01-03T11:27:25Z

@anuragaw a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-01-03T19:31:14Z

Trillian test result (tid-682)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27456 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3510-t682-kvm-centos7.zip
Smoke tests completed. 78 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

andrijapanicsb

LGMT.

Tested "in-house" and seems to be working fine for all 3 hypervisors.

andrijapanicsb · 2020-01-08T16:53:11Z

Test Name	Steps	Expected Result	Status
Add configuration when not enabled	Make sure that enable.additional.vm.configuration is false and add extra configuration following the KVM example bellow.	VM Details are not being updated (localcloud) SBCM5> > update virtualmachine id=26f88815-5e08-4ab1-ba0b-15658110ab40 extraconfig='%3CmemoryBacking%3E%0A%20%20%3Chugepages%2F%3E%0A%3C%2FmemoryBacking%3E' filter=details, {}	Passed
Enable configuration	Set enable.additional.vm.configuration to True and add extra configuration following the KVM example bellow.	User is able to set configuration (localcloud) SBCM5> > update virtualmachine id=26f88815-5e08-4ab1-ba0b-15658110ab40 extraconfig='%3CmemoryBacking%3E%0A%20%20%3Chugepages%2F%3E%0A%3C%2FmemoryBacking%3E' filter=details, { “details": { "extraconfig-1": "\n \n" },	Passed
KVM extra config	Deploy a VM and set extra config with the following api call: deploy virtualmachine … extraconfig='%3CmemoryBacking%3E%0A%20%20%3Chugepages%2F%3E%0A%3C%2FmemoryBacking%3E'	Dump xml the domain on the kvm host and verify that the extra settings are there virsh dumpxml 2 … 524288 524288 …	Passed
VMware extra config	Deploy a VM Stop the VM Update the VM with the following api command and start it update virtualmachine id=[vm-id] extraconfig=' hypervisor.cpuid.v0%3DFALSE'	Cat the vmx file on the primary storage and observe that the specific setting is there cat "$(esxcli vm process list \| grep i-2-8-VM \| tail -1 \| awk '{print $3}')" \| grep -w 'hypervisor.cpuid.v0 = "FALSE"' hypervisor.cpuid.v0 = "FALSE"	Passed
XenServer extra config	Deploy a VM Stop the VM Update the VM with the following api command and start it update virtualmachine id=[vm-id] extraconfig=' HVM-boot-policy%3D%0APV-bootloader%3Dpygrub%0APV-args%3Dhvc0'	On the Xen host us the ‘xe vm-param-get param-name’ command to get the specific parameter and observe it’s been set xe vm-param-get param-name=PV-bootloader uuid=f5b6ea53-b9dd-d118-a404-582675cc4345 pygrub xe vm-param-get param-name=PV-args uuid=f5b6ea53-b9dd-d118-a404-582675cc4345 hvc0	Passed
KVM key list restriction	Remove all items from allow.additional.vm.configuration.list.kvm Deploy a VM and set extra config with the following api call: deploy virtualmachine … extraconfig='%3CmemoryBacking%3E%0A%20%20%3Chugepages%2F%3E%0A%3C%2FmemoryBacking%3E'	Exception appears that the item is not allowed. Any of the child records of that xml node were not added as well CloudstackAPIException: Execute cmd: deployvirtualmachine failed, due to: errorCode: 530, errorText:Extra config is not on the list of allowed keys for KVM hypervisor hosts	Passed
VMware key list restriction	Remove all items from allow.additional.vm.configuration.list.vmware global setting Deploy a VM Stop the VM Update the VM with the following api command and start it update virtualmachine id=[vm-id] extraconfig='hypervisor.cpuid.v0%3DFALSE'	Exception appears that the item is not allowed. CloudstackAPIException: Execute cmd: deployvirtualmachine failed, due to: errorCode: 530, errorText:Extra config hypervisor.cpuid.v0=FALSE is not on the list of allowed keys for VMware hypervisor hosts.	Passed
XenServer key list restriction	Remove all items from allow.additional.vm.configuration.list.xenserver global setting Deploy a VM Stop the VM Update the VM with the following api command and start it update virtualmachine id=[vm-id] extraconfig='HVM-boot-policy%3D%0APV-bootloader%3Dpygrub%0APV-args%3Dhvc0'	Exception appears that the item is not allowed. CloudstackAPIException: Execute cmd: deployvirtualmachine failed, due to: errorCode: 530, errorText:Extra config HVM-boot-policy= is not on the list of allowed keys for XenServer hypervisor hosts.	Passed
XenServer key mapping negative	Pass an argument that is not valid as per the given pattern: Test value: HVM-boot-policy= PV-bootloader=pygrub=failure PV-args=hvc0	Cloudstack throws an exception when validating the map: (localcloud) SBCM5> > update virtualmachine id=afeb87cf-5a0a-4807-9c69-8644544ccdf9 extraconfig=HVM-boot-policy%3D%0APV-bootloader%3Dpygrub%3Dfailure%0APV-args%3Dhvc0 🙈 Error: (HTTP 530, error code 4250) The passed extra config string 'HVM-boot-policy= PV-bootloader=pygrub=failure PV-args=hvc0' contains an invalid key/value pair pattern	Passed
Clean extra config data	Stop the VM Execute update VM with ‘cleanupdetails=true’ argument	Any additional config has been removed (localcloud) SBCM5> > list virtualmachines id=9207b8b4-6d94-4a93-ba44-20ad5cd39f76 filter=details, { "virtualmachine": [ "details": { "deployvm": "true", "extraconfig-1": "\n \n" } (localcloud) SBCM5> > update virtualmachine id=9207b8b4-6d94-4a93-ba44-20ad5cd39f76 cleanupdetails=true { "virtualmachine": {...} (localcloud) SBCM5> > list virtualmachines id=9207b8b4-6d94-4a93-ba44-20ad5cd39f76 filter=details, { "virtualmachine": [ "details": {} }	Passed
KVM – Nested keys not allowed	Create an xml that is containing a not allowed key inside an allowed key and try submitting it.	(localcloud) SBCM5> > update virtualmachine id=9d5b6561-ee7b-4711-94c7-c79f268cfa24 extraconfig='%3CmemoryBacking%3E%0A%20%20%3Chugepages%2F%3E%0A%3C%2FmemoryBacking%3E' 🙈 Error: (HTTP 530, error code 4250) Extra config hugepages is not on the list of allowed keys for KVM hypervisor hosts (localcloud) SBCM5> > list configurations name=allow.additional.vm.configuration.list.kvm { "name": "allow.additional.vm.configuration.list.kvm", "value": "memoryBacking" }	Passed
Apply configuration when it’s disabled	Set enable.additional.vm.configuration to false Apply any configurations to a VM.	(localcloud) SBCM5> > update virtualmachine id=9d5b6561-ee7b-4711-94c7-c79f268cfa24 extraconfig='%3CmemoryBacking%3E%0A%3C%2FmemoryBacking%3E' 🙈 Error: (HTTP 431, error code 4350) attempted setting extraconfig but enable.additional.vm.configuration is disabled	Passed
Update some of the enabled configurations	Have a list of enabled keys, but pass just one of them	(localcloud) SBCM5> > update virtualmachine id=9d5b6561-ee7b-4711-94c7-c79f268cfa24 extraconfig='%3CmemoryBacking%3E%0A%3C%2FmemoryBacking%3E' { "virtualmachine": { … "details": { "Message.ReservedCapacityFreed.Flag": "true", "extraconfig-1": "\n" } … (localcloud) SBCM5> > list configurations name=allow.additional.vm.configuration.list.kvm { "name": "allow.additional.vm.configuration.list.kvm", "value": "memoryBacking, hugepages" }	Passed

* Suqash commits to a single commit and rebase against master Update marvin tests to use white list * * Fix marvin test failure * Add new marvin negative tests cases * Remove hard-coded hypervisor types in marvin tests * Fix build error after rebase and add hugepagesless * Fix readability of python code * Fix failing test * Adding cleanup of vms for negative tests * Bug fixes - change config checks properly and block extraconfig in details * Trim to compare the keys * CR comments * Don't skip extraconfig without exception Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>

harikrishna-patnala · 2020-12-02T07:24:13Z

@pavanaravapalli please open a github issue when you see anything not working with appropriate information and steps. Thanks

pavanaravapalli · 2020-12-02T07:34:30Z

@harikrishna-patnala ok i will do necessary if required.

anuragaw changed the title ~~Allow additional config to vms~~ [WIP DO NOT MERGE] Allow additional config to vms Jul 23, 2019

rohityadavcloud added this to the 4.14.0.0 milestone Jul 24, 2019

rohityadavcloud approved these changes Jul 24, 2019

View reviewed changes

rohityadavcloud closed this Jul 24, 2019

rohityadavcloud reopened this Jul 24, 2019

borisstoyanov reviewed Jul 24, 2019

View reviewed changes

svenvogel added type:enhancement type:new-feature labels Aug 10, 2019

borisstoyanov requested changes Aug 15, 2019

View reviewed changes

anuragaw force-pushed the allow-additional-config-to-vms branch from d0fcd97 to 4427f14 Compare September 11, 2019 19:19

Anurag Awasthi and others added 5 commits December 21, 2019 19:09

Fix failing test

162f1f5

Adding cleanup of vms for negative tests

98e0ee1

Bug fixes - change config checks properly and block extraconfig in de…

8274ecd

…tails

Trim to compare the keys

501653b

CR comments

1d22a0a

anuragaw force-pushed the allow-additional-config-to-vms branch from ddc1689 to 1d22a0a Compare December 21, 2019 15:48

Don't skip extraconfig without exception

c00f80f

DaanHoogland approved these changes Jan 3, 2020

View reviewed changes

andrijapanicsb approved these changes Jan 8, 2020

View reviewed changes

andrijapanicsb changed the title ~~[WIP DO NOT MERGE] Allow additional config to vms~~ Allow additional configuration metadata to VMs Jan 8, 2020

andrijapanicsb merged commit 70fcf75 into apache:master Jan 8, 2020

PaulAngus mentioned this pull request Jan 20, 2020

New Feature: KVM Agent Hooks #3823

Closed

nvazquez mentioned this pull request Feb 13, 2020

FEATURE-3823: kvm agent hooks #3839

Merged

5 tasks

weizhouapache mentioned this pull request Aug 10, 2023

UI: deploy and update vm instances with extraconfig #7849

Closed

Allow additional configuration metadata to VMs #3510

Allow additional configuration metadata to VMs #3510

Uh oh!

Conversation

anuragaw commented Jul 23, 2019 • edited by andrijapanicsb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

anuragaw commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

borisstoyanov commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 23, 2019

Uh oh!

blueorangutan commented Jul 24, 2019

Uh oh!

blueorangutan commented Jul 24, 2019

Uh oh!

rohityadavcloud left a comment

Choose a reason for hiding this comment

Uh oh!

wido commented Jul 24, 2019

Uh oh!

borisstoyanov commented Jul 24, 2019

Uh oh!

wido commented Jul 24, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

anuragaw commented Jul 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

svenvogel commented Aug 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

anuragaw commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 11, 2019

Uh oh!

anuragaw commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 11, 2019

Uh oh!

anuragaw commented Sep 12, 2019

Uh oh!

blueorangutan commented Sep 12, 2019

Uh oh!

anuragaw commented Dec 21, 2019

Uh oh!

blueorangutan commented Dec 21, 2019

Uh oh!

blueorangutan commented Dec 21, 2019

Uh oh!

borisstoyanov commented Dec 31, 2019

Uh oh!

blueorangutan commented Dec 31, 2019

Uh oh!

blueorangutan commented Dec 31, 2019

Uh oh!

anuragaw commented Jan 3, 2020

Uh oh!

blueorangutan commented Jan 3, 2020

Uh oh!

blueorangutan commented Jan 3, 2020

Uh oh!

anuragaw commented Jul 23, 2019 •

edited by andrijapanicsb

Loading

anuragaw commented Jul 29, 2019 •

edited

Loading

svenvogel commented Aug 10, 2019 •

edited

Loading

harikrishna-patnala commented Dec 2, 2020 •

edited

Loading