marvin: make deployDataCenter.py script compatible with python 2 and 3 #5139

Pearl1594 · 2021-06-22T12:04:51Z

Description

This PR fixes a compatibility issue noticed with deployDataCenter.py script when run with python3 - the script reports the following exception:

Exception Occurred  while persisting DC Settings: ['Traceback (most recent call last):\n', '  File "tools/marvin/marvin/deployDataCenter.py", line 76, in __persistDcConfig\n    pickle.dump(self.__cleanUp, file_to_write)\n', 'TypeError: write() argument must be str, not bytes\n']

And hence doesn't create the dc_entries.obj file

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

Ran the deployDataCenter script using python2 and python3 - it runs successfully and generates the dc_entries.obj file in human readable format (which is controlled by the protocol number passed)

weizhouapache · 2021-06-22T13:47:39Z

@blueorangutan package

blueorangutan · 2021-06-22T13:48:03Z

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-06-22T14:41:45Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 324

weizhouapache · 2021-06-22T14:44:23Z

@blueorangutan test kvm-ubuntu20 kvm-ubuntu20

blueorangutan · 2021-06-22T14:45:02Z

@weizhouapache unsupported parameters provided. Supported mgmt server os are: centos6, centos7, centos8, ubuntu18, ubuntu20. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-centos8, kvm-ubuntu18, kvm-ubuntu20, xenserver-71, xenserver-65sp1, vmware-70u1, vmware-67u3, vmware-65u2, vmware-60u2, vmware-55u3, xcpng76, xcpng80, xcpng81, xenserver-74, xcpng74

weizhouapache · 2021-06-22T14:45:25Z

@blueorangutan test ubuntu20 kvm-ubuntu20

blueorangutan · 2021-06-22T14:46:03Z

@weizhouapache a Trillian-Jenkins test job (ubuntu20 mgmt + kvm-ubuntu20) has been kicked to run smoke tests

weizhouapache · 2021-06-22T14:46:50Z

@blueorangutan test

blueorangutan · 2021-06-22T14:48:03Z

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-06-22T15:18:05Z

Trillian Build Failed (tid-1042)

blueorangutan · 2021-06-23T03:06:35Z

Trillian test result (tid-1041)
Environment: kvm-ubuntu20 (x2), Advanced Networking with Mgmt server u20
Total time taken: 42515 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5139-t1041-kvm-ubuntu20.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 87 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	528.61	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	528.63	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	540.18	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	510.48	test_vpc_redundant.py

rohityadavcloud · 2021-06-23T04:57:17Z

@blueorangutan test

blueorangutan · 2021-06-23T04:58:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-06-24T00:31:39Z

Trillian test result (tid-1043)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 67119 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5139-t1043-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 77 look OK, 11 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_nic	`Error`	154.29	test_nic.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	555.81	test_kubernetes_clusters.py
test_01_isolate_network_FW_PF_default_routes_egress_true	`Failure`	341.47	test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	430.94	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	`Failure`	406.04	test_routers_network_ops.py
ContextSuite context=TestVMWareStoragePolicies>:setup	`Error`	0.00	test_storage_policy.py
test_02_create_template_with_checksum_sha1	`Error`	65.58	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.44	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.50	test_templates.py
test_05_create_template_with_no_checksum	`Error`	65.45	test_templates.py
test_04_extract_template	`Failure`	128.49	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_volume_usage	`Error`	194.53	test_usage.py
test_10_attachAndDetach_iso	`Failure`	1510.29	test_vm_life_cycle.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible	`Error`	1511.20	test_hostha_kvm.py

Pearl1594 · 2021-06-24T03:45:47Z

@blueorangutan test

blueorangutan · 2021-06-24T03:46:02Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland

looks good, have to find some time to test this but low risk so I'm ok with merge.
(if travis passes)

rohityadavcloud · 2021-06-29T07:14:33Z

Travis is passing except that one job
@blueorangutan test

blueorangutan · 2021-06-29T07:15:02Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

rohityadavcloud · 2021-07-05T06:18:17Z

@blueorangutan package

blueorangutan · 2021-07-05T06:19:04Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-07-05T06:46:24Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 466

Pearl1594 · 2021-07-05T06:47:19Z

@blueorangutan test

blueorangutan · 2021-07-05T06:48:02Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-06T11:22:36Z

Trillian test result (tid-1199)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 101255 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5139-t1199-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_async_job.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vms_with_varied_deploymentplanners.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_nested_virtualization.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_nic_adapter_type.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_portforwardingrules.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 63 look OK, 25 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestInternalLb>:setup	`Error`	0.00	test_internal_lb.py
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	3.93	test_accounts.py
test_forceDeleteDomain	`Failure`	3.92	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	4.95	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
test_01_create_iso_with_checksum_sha1	`Error`	66.49	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	66.70	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	66.70	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	66.37	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	69.68	test_iso.py
test_01_create_iso	`Failure`	1512.87	test_iso.py
ContextSuite context=TestISO>:setup	`Error`	3023.52	test_iso.py
ContextSuite context=TestAsyncJob>:setup	`Error`	0.00	test_async_job.py
ContextSuite context=TestLoadBalance>:setup	`Error`	0.00	test_loadbalance.py
test_list_clusters_metrics	`Error`	1518.67	test_metrics_api.py
test_list_vms_metrics	`Error`	0.13	test_metrics_api.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
ContextSuite context=TestDeployVMFromISO>:setup	`Error`	0.00	test_deploy_vm_iso.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup	`Error`	0.00	test_deploy_vms_with_varied_deploymentplanners.py
ContextSuite context=TestNetworkACL>:setup	`Error`	0.00	test_network_acl.py
test_delete_account	`Error`	1523.48	test_network.py
test_delete_network_while_vm_on_it	`Error`	1.09	test_network.py
test_deploy_vm_l2network	`Error`	1.10	test_network.py
test_l2network_restart	`Error`	2.15	test_network.py
ContextSuite context=TestPortForwarding>:setup	`Error`	3.27	test_network.py
ContextSuite context=TestPublicIP>:setup	`Error`	0.96	test_network.py
test_reboot_router	`Failure`	0.03	test_network.py
test_releaseIP	`Error`	0.41	test_network.py
ContextSuite context=TestRouterRules>:setup	`Error`	0.44	test_network.py
ContextSuite context=TestAdapterTypeForNic>:setup	`Error`	0.00	test_nic_adapter_type.py
ContextSuite context=TestDomainsServiceOfferings>:setup	`Error`	1514.44	test_domain_service_offerings.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_06_deploy_invalid_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	`Error`	60.53	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
test_nic_secondaryip_add_remove	`Error`	1515.49	test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup	`Error`	0.00	test_nested_virtualization.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup	`Error`	0.00	test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup	`Error`	0.00	test_portforwardingrules.py
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
ContextSuite context=TestProjectSuspendActivate>:setup	`Error`	1518.98	test_projects.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	`Failure`	341.80	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	`Failure`	336.27	test_routers_network_ops.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	520.23	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	520.25	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	531.46	test_vpc_redundant.py

Pearl1594 · 2021-07-06T11:24:15Z

@blueorangutan test

blueorangutan · 2021-07-06T11:25:02Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-06T21:26:14Z

Trillian test result (tid-1207)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34453 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5139-t1207-kvm-centos7.zip
Smoke tests completed. 88 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez

LGTM

nvazquez · 2021-07-13T01:52:36Z

Merging based on approvals and test results

marvin: make deployDataCenter.py script py2/py3 compatible

82fc80b

Pearl1594 requested a review from DaanHoogland June 22, 2021 12:04

rohityadavcloud approved these changes Jun 22, 2021

View reviewed changes

DaanHoogland approved these changes Jun 24, 2021

View reviewed changes

rohityadavcloud added this to the 4.16.0.0 milestone Jul 5, 2021

nvazquez approved these changes Jul 13, 2021

View reviewed changes

nvazquez merged commit 2e376f2 into apache:main Jul 13, 2021

nvazquez added component:marvin Python Warning... Python code Ahead! labels Oct 14, 2021

marvin: make deployDataCenter.py script compatible with python 2 and 3 #5139

marvin: make deployDataCenter.py script compatible with python 2 and 3 #5139

Uh oh!

Conversation

Pearl1594 commented Jun 22, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

How Has This Been Tested?

Uh oh!

weizhouapache commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

weizhouapache commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

weizhouapache commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

weizhouapache commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 22, 2021

Uh oh!

blueorangutan commented Jun 23, 2021

Uh oh!

rohityadavcloud commented Jun 23, 2021

Uh oh!

blueorangutan commented Jun 23, 2021

Uh oh!

blueorangutan commented Jun 24, 2021

Uh oh!

Pearl1594 commented Jun 24, 2021

Uh oh!

blueorangutan commented Jun 24, 2021

Uh oh!

DaanHoogland left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohityadavcloud commented Jun 29, 2021

Uh oh!

blueorangutan commented Jun 29, 2021

Uh oh!

rohityadavcloud commented Jul 5, 2021

Uh oh!

blueorangutan commented Jul 5, 2021

Uh oh!

blueorangutan commented Jul 5, 2021

Uh oh!

Pearl1594 commented Jul 5, 2021

Uh oh!

blueorangutan commented Jul 5, 2021

Uh oh!

blueorangutan commented Jul 6, 2021

Uh oh!

Pearl1594 commented Jul 6, 2021

Uh oh!

blueorangutan commented Jul 6, 2021

Uh oh!

blueorangutan commented Jul 6, 2021

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Jul 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

DaanHoogland left a comment •

edited

Loading