Enable Direct Download for systemVM templates #3731

shwstppr · 2019-12-02T09:00:45Z

Description

As of now provisioning of System VMs with direct download enabled template was not supported. Even after setting such templates as SYSTEM template, while provisioning SSVM and CPVM the server would return an error message like,

System vm template is not ready at data center 1, wait until it is ready to launch secondary storage vm

This behaviour has been changed and it allows the server to provision SSVM and CPVM even with direct download system templates.
Additionally, changes have been made to make different timeouts for direct download functionality configurable using global settings. Following global configurations have been added for this,
direct.download.connect.timeout - Connection establishment timeout in milliseconds for direct download
direct.download.socket.timeout - Socket timeout (SO_TIMEOUT) in milliseconds for direct download
direct.download.connection.request.timeout - Requesting a connection from connection manager timeout in milliseconds for direct download

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

New global settings,

How Has This Been Tested?

Admin registers a direct download private USER or ROUTING type template,

> register template url=http://staging.yadav.xyz/systemvmtemplate/4.14/systemvmtemplate-4.14.0-kvm.qcow2.bz2 name=NewTemplate displaytext=NewTemplate zoneids=94ecae8c-923a-4f32-9d32-4067a99e76e2 format=QCOW2 isextractable=false passwordEnabled=false isdynamicallyscalable=false ostypeid=528a9a3d-4e13-11ea-8448-1e006e01076c hypervisor=KVM ispublic=false requireshvm=true isfeatured=false isrouting=true details[0].rootDiskController=osdefault directdownload=true
{
  "count": 1,
  "template": [
    {
      "account": "admin",
      "bits": 0,
      "created": "2020-02-13T09:02:45+0000",
      "crossZones": false,
      "details": {
        "rootDiskController": "osdefault"
      },
      "directdownload": true,
      "displaytext": "NewTemplate",
      "domain": "ROOT",
      "domainid": "527d6f21-4e13-11ea-8448-1e006e01076c",
      "format": "QCOW2",
      "hypervisor": "KVM",
      "id": "4cdf991c-4ed8-46f5-be3b-9810c9fce17d",
      "isdynamicallyscalable": false,
      "isextractable": false,
      "isfeatured": false,
      "ispublic": false,
      "isready": true,
      "name": "NewTemplate",
      "ostypeid": "528a9a3d-4e13-11ea-8448-1e006e01076c",
      "ostypename": "Debian GNU/Linux 5.0 (64-bit)",
      "passwordenabled": false,
      "requireshvm": true,
      "size": 328521792,
      "sshkeyenabled": false,
      "status": "Bypassed Secondary Storage",
      "tags": [],
      "templatetype": "ROUTING",
      "zoneid": "94ecae8c-923a-4f32-9d32-4067a99e76e2",
      "zonename": "pr3731-t985-kvm-centos7"
    }
  ]
}

For testing databse is updated manually to make newly added template to the SYSTEM type template. This can be done using upgrade script during upgrade,

MariaDB [cloud]> update vm_template set type='SYSTEM' where uuid='4cdf991c-4ed8-46f5-be3b-9810c9fce17d';
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1  Changed: 0  Warnings: 0

MariaDB [cloud]> select * from vm_template where uuid='4cdf991c-4ed8-46f5-be3b-9810c9fce17d'\G;
*************************** 1. row ***************************
                  id: 207
         unique_name: 207-2-b5793aa9-d8d1-33bd-b1c5-d144f98a6719
                name: NewTemplate
                uuid: 4cdf991c-4ed8-46f5-be3b-9810c9fce17d
              public: 0
            featured: 0
                type: SYSTEM
                 hvm: 1
                bits: 64
                 url: http://staging.yadav.xyz/systemvmtemplate/4.14/systemvmtemplate-4.14.0-kvm.qcow2.bz2
              format: QCOW2
             created: 2020-02-13 09:02:45
             removed: NULL
          account_id: 2
            checksum: NULL
        display_text: NewTemplate
     enable_password: 0
       enable_sshkey: 0
         guest_os_id: 15
            bootable: 1
         prepopulate: 0
         cross_zones: 0
         extractable: 0
     hypervisor_type: KVM
  source_template_id: NULL
        template_tag: NULL
            sort_key: 0
                size: 328521792
               state: Active
        update_count: 0
             updated: NULL
dynamically_scalable: 0
  parent_template_id: NULL
     direct_download: 1
1 row in set (0.00 sec)

Existing SSVM and CPVM are destroyed for testing new template and server can provision new VMs with newly added template.
New VMs get created,

Database values for template_id verified in vm_instance,
MariaDB [cloud]> select id,name,vm_template_id from vm_instance where removed IS NULL\G;

*************************** 1. row ***************************
            id: 5
          name: VM-17980201-ff14-48c5-ba8a-d90e1be13490
vm_template_id: 4
*************************** 2. row ***************************
            id: 6
          name: r-6-VM
vm_template_id: 203
*************************** 3. row ***************************
            id: 12
          name: s-12-VM
vm_template_id: 207
*************************** 4. row ***************************
            id: 13
          name: v-13-VM
vm_template_id: 207
4 rows in set (0.00 sec)

To check direct download template for VR router.template.* global config is updated and VR is recreated or new network is created.

> list routers filter=id,name
{
  "count": 1,
  "router": [
    {
      "id": "e003704c-c98e-4e84-9a6d-8bfeb45b10cb",
      "name": "r-15-VM"
    }
  ]
}

Verified in DB,

MariaDB [cloud]> select id,name,vm_template_id from vm_instance where uuid='e003704c-c98e-4e84-9a6d-8bfeb45b10cb'\G;
*************************** 1. row ***************************
            id: 15
          name: r-15-VM
vm_template_id: 207
1 row in set (0.00 sec)

nvazquez · 2019-12-23T12:40:26Z

core/src/main/java/org/apache/cloudstack/agent/directdownload/HttpDirectDownloadCommand.java

-    public HttpDirectDownloadCommand(String url, Long templateId, PrimaryDataStoreTO destPool, String checksum, Map<String, String> headers) {
+    public HttpDirectDownloadCommand(String url, Long templateId, PrimaryDataStoreTO destPool, String checksum, Map<String, String> headers, int connectTimeout, int soTimeout) {
        super(url, templateId, destPool, checksum, headers);
+        setConnectTimeout(connectTimeout);


Can these be set on the DrectDownloadCommand constructor?

nvazquez · 2019-12-23T12:47:11Z

server/src/main/java/org/apache/cloudstack/direct/download/DirectDownloadManagerImpl.java

+                DownloadProtocol.HTTPS.equals(protocol) ||
+                DownloadProtocol.METALINK.equals(protocol)) {
+            try {
+                connectTimeout = Integer.parseInt(configDao.getValue(DirectDownloadConnectTimeout.key()));


Please do not use this and use the config key value() method, same below

nvazquez · 2019-12-23T13:16:15Z

server/src/main/java/com/cloud/template/TemplateManagerImpl.java


    private void updateDirectDownloadTemplate(long templateId, long zoneId) {
-        TemplateDataStoreVO templateStore = _tmplStoreDao.findByTemplateZone(templateId, zoneId, DataStoreRole.Image);
+        TemplateDataStoreVO templateStore = _tmplStoreDao.findByTemplate(templateId, DataStoreRole.Image);


So we are updating every reference in every zone for the template?

@nvazquez as we do not store store_id in cloud.template_store_ref table for direct download template, we cannot find the entry in the table for a zone while update. Continued with same behaviour from user direct download templates. Do we need to change this for system direct download templates? updateDirectDownloadTemplate is new method added in this PR only

nvazquez · 2019-12-23T13:41:59Z

Thanks @shwstppr, left some comments after reviewing

shwstppr · 2019-12-24T07:58:32Z

@blueorangutan package

blueorangutan · 2019-12-24T07:58:40Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-12-24T08:18:02Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-503

rohityadavcloud · 2020-01-28T05:28:10Z

Is it ready @shwstppr @Spaceman1984 ?

shwstppr · 2020-01-28T05:30:02Z

No @rhtyd. This is based on another PR so waiting for it to get merge, then rebase.

DaanHoogland · 2020-01-29T12:11:40Z

No @rhtyd. This is based on another PR so waiting for it to get merge, then rebase.

#3748

blueorangutan · 2020-02-05T10:00:20Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-751

blueorangutan · 2020-02-05T10:07:28Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-752

blueorangutan · 2020-02-05T10:42:23Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-753

blueorangutan · 2020-02-06T12:47:26Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-772

rohityadavcloud · 2020-02-12T08:02:37Z

@blueorangutan package

blueorangutan · 2020-02-12T08:03:14Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-02-12T08:58:18Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-812

blueorangutan · 2020-02-12T12:15:33Z

Packaging result: ✖centos6 ✖centos7 ✔debian. JID-829

blueorangutan · 2020-02-20T06:11:52Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-912

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

DaanHoogland · 2020-02-20T20:43:40Z

@shwstppr is this still WIP? if no, please remove tag from the title.

borisstoyanov · 2020-02-21T09:03:10Z

@blueorangutan package

blueorangutan · 2020-02-21T09:04:06Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

shwstppr · 2020-02-21T09:31:58Z

@DaanHoogland yes it still needs some fixing related to the listing of direct download system template after removing adding new zones. Working on it

blueorangutan · 2020-02-21T09:33:57Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-929

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2020-02-21T13:02:59Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-932

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2020-02-21T17:05:06Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-933

shwstppr · 2020-02-22T09:57:11Z

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

                }

+                // delete template refs for this zone
+                templateZoneDao.deleteByZoneId(zoneId);


@DaanHoogland please review if this is the correct place to cleanup template_zone_ref entries for a zone to be removed? Or should this call be made after the successful removal of DataCenterVO in success block below? Functionality-wise it is working fine.
cc @rhtyd

@shwstppr absolutely only on success. I don't think the zone is still usable if deletion of the zoneVO, but it is not right to do an automatic clean-up of items (templates) in a container (zone) if the container is not disposed of.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2020-02-23T10:50:23Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-937

shwstppr · 2020-02-23T13:19:03Z

@blueorangutan test

blueorangutan · 2020-02-23T13:19:05Z

@shwstppr a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-02-23T21:12:36Z

Trillian test result (tid-1091)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27035 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3731-t1091-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 74 look OK, 6 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	178.68	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	191.76	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	279.47	test_privategw_acl.py
test_11_migrate_vm	`Error`	43.66	test_vm_life_cycle.py
test_13_attachAndDetach_iso	`Failure`	1511.69	test_vm_life_cycle.py
test_06_download_detached_volume	`Error`	9.31	test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py

borisstoyanov · 2020-02-24T07:44:09Z

@blueorangutan test

blueorangutan · 2020-02-24T07:45:08Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-02-24T20:58:03Z

Trillian test result (tid-1108)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29372 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3731-t1108-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 79 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	184.54	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	177.04	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	246.52	test_privategw_acl.py

borisstoyanov

LGTM, test failures does not seem related

andrijapanicsb · 2020-02-26T12:37:03Z

LGTM after internal review.

andrijapanicsb · 2020-02-26T12:38:26Z

Merging based on 4 x approvals and regression test results (and of course manual testing)

svenvogel added type:enhancement component:agent labels Dec 2, 2019

GabrielBrascher assigned shwstppr Dec 5, 2019

rohityadavcloud added this to the 4.14.0.0 milestone Dec 7, 2019

shwstppr force-pushed the feature-system-vm-direct-download branch 2 times, most recently from 02b3802 to 9d7f71d Compare December 20, 2019 08:37

nvazquez reviewed Dec 23, 2019

View reviewed changes

shwstppr changed the title ~~[WIP DO NOT MERGE] System VM Template Direct Download Support~~ [WIP DO NOT MERGE] Enable Direct Download for System VMs Jan 13, 2020

rohityadavcloud closed this Jan 29, 2020

rohityadavcloud reopened this Jan 29, 2020

DaanHoogland mentioned this pull request Jan 29, 2020

Systemvm template api #3748

Closed

5 tasks

shwstppr force-pushed the feature-system-vm-direct-download branch from 76fe17e to c66a81c Compare February 5, 2020 07:03

shwstppr force-pushed the feature-system-vm-direct-download branch from 1afa879 to c66a81c Compare February 12, 2020 11:47

connection request timeout as hidden configuration

6c08e4a

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix template zone ref cleanup on zone deletion

31d28b9

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 2 commits February 21, 2020 21:53

fix previous commit test error, change implementation

8b98528

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Merge branch 'master' into feature-system-vm-direct-download

0bb6ee7

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr commented Feb 22, 2020

View reviewed changes

DaanHoogland removed their request for review February 22, 2020 15:12

refactored zone template cleanup

96b0b70

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

borisstoyanov approved these changes Feb 25, 2020

View reviewed changes

DaanHoogland changed the title ~~[WIP DO NOT MERGE] Enable Direct Download for System VMs~~ Enable Direct Download for System VMs Feb 25, 2020

andrijapanicsb approved these changes Feb 26, 2020

View reviewed changes

andrijapanicsb merged commit 0ad2370 into apache:master Feb 26, 2020

andrijapanicsb changed the title ~~Enable Direct Download for System VMs~~ Enable Direct Download for systemVM templates Apr 12, 2020

shwstppr mentioned this pull request Jun 2, 2020

kvm: direct download templates for system VM apache/cloudstack-documentation#134

Merged

Enable Direct Download for systemVM templates #3731

Enable Direct Download for systemVM templates #3731

Uh oh!

Conversation

shwstppr commented Dec 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez Dec 23, 2019

Choose a reason for hiding this comment

Uh oh!

shwstppr Dec 24, 2019

Choose a reason for hiding this comment

Uh oh!

nvazquez Dec 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shwstppr Dec 24, 2019

Choose a reason for hiding this comment

Uh oh!

nvazquez Dec 23, 2019

Choose a reason for hiding this comment

Uh oh!

shwstppr Dec 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Dec 23, 2019

Uh oh!

shwstppr commented Dec 24, 2019

Uh oh!

blueorangutan commented Dec 24, 2019

Uh oh!

blueorangutan commented Dec 24, 2019

Uh oh!

rohityadavcloud commented Jan 28, 2020

Uh oh!

shwstppr commented Jan 28, 2020

Uh oh!

DaanHoogland commented Jan 29, 2020

Uh oh!

blueorangutan commented Feb 5, 2020

Uh oh!

blueorangutan commented Feb 5, 2020

Uh oh!

blueorangutan commented Feb 5, 2020

Uh oh!

blueorangutan commented Feb 6, 2020

Uh oh!

rohityadavcloud commented Feb 12, 2020

Uh oh!

blueorangutan commented Feb 12, 2020

Uh oh!

blueorangutan commented Feb 12, 2020

Uh oh!

blueorangutan commented Feb 12, 2020

Uh oh!

blueorangutan commented Feb 20, 2020

Uh oh!

DaanHoogland commented Feb 20, 2020

Uh oh!

borisstoyanov commented Feb 21, 2020

Uh oh!

blueorangutan commented Feb 21, 2020

Uh oh!

shwstppr commented Feb 21, 2020

Uh oh!

blueorangutan commented Feb 21, 2020

Uh oh!

blueorangutan commented Feb 21, 2020

Uh oh!

blueorangutan commented Feb 21, 2020

Uh oh!

shwstppr Feb 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

shwstppr commented Dec 2, 2019 •

edited

Loading

nvazquez Dec 23, 2019 •

edited

Loading

shwstppr Dec 24, 2019 •

edited

Loading

shwstppr Feb 22, 2020 •

edited

Loading