kvm: pre-add 32 PCI controller for hot-plug issue on ARM64/aarch64 #5193

rohityadavcloud · 2021-07-08T08:12:35Z

On newer libvirt/qemu it seems PCI hot-plugging could be an issue as
seen in:

https://www.suse.com/support/kb/doc/?id=000019383
https://bugs.launchpad.net/nova/+bug/1836065

This was found to be true on ARM64/aarch64 platform (tested on
RaspberryPi4). As per the default machine doc, it advises to
pre-allocate PCI controllers on the machine and pcie-to-pci-bridge based
controller for legacy PCI models:
https://libvirt.org/pci-hotplug.html#x86_64-q35

This patch introduces the concept as a workaround until a proper fix is
done (ideally in the upstream libvirt/qemu projects). Until then client
code can add 32 PCI controllers and a pcie-to-pci-bridge controller for
aarch64 platforms.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Tested on RaspberryPi4+Ubuntu 20.04 with: http://download.cloudstack.org/rpi4/4.15/
Old PR: #4182

On newer libvirt/qemu it seems PCI hot-plugging could be an issue as seen in: https://www.suse.com/support/kb/doc/?id=000019383 https://bugs.launchpad.net/nova/+bug/1836065 This was found to be true on ARM64/aarch64 platform (tested on RaspberryPi4). As per the default machine doc, it advises to pre-allocate PCI controllers on the machine and pcie-to-pci-bridge based controller for legacy PCI models: https://libvirt.org/pci-hotplug.html#x86_64-q35 This patch introduces the concept as a workaround until a proper fix is done (ideally in the upstream libvirt/qemu projects). Until then client code can add 32 PCI controllers and a pcie-to-pci-bridge controller for aarch64 platforms. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

rohityadavcloud · 2021-07-08T08:13:28Z

@blueorangutan package

blueorangutan · 2021-07-08T08:14:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-07-08T08:39:31Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 499

rohityadavcloud · 2021-07-08T11:21:35Z

@blueorangutan test

blueorangutan · 2021-07-08T11:22:02Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-08T20:47:41Z

Trillian test result (tid-1218)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32321 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5193-t1218-kvm-centos7.zip
Smoke tests completed. 87 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

cltgm, seems to do what it says on the tin. you are still the only r-pi user @rhtyd so i don't think we'll find a third party tester :(

rohityadavcloud · 2021-07-13T08:32:22Z

@DaanHoogland I think there are few other users but yes, let's accept based on smoketests and my manual tests on my toy RPi4 env (I can give access if anybody wants - https://yadav.cloud/client)

rohityadavcloud · 2021-07-15T07:36:32Z

Domain xml of a functioning working VM:

root@pikvm2:~# virsh list
 Id   Name         State
----------------------------
 1    s-236-VM     running
 2    v-237-VM     running
 3    i-2-183-VM   running

root@pikvm2:~# virsh dumpxml i-2-183-VM
<domain type='kvm' id='3'>
  <name>i-2-183-VM</name>
  <uuid>f973d4ab-043f-41eb-93c5-c14657cec3bd</uuid>
  <description>Other Ubuntu (64-bit)</description>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <shares>2000</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Apache Software Foundation</entry>
      <entry name='product'>CloudStack KVM Hypervisor</entry>
      <entry name='uuid'>f973d4ab-043f-41eb-93c5-c14657cec3bd</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='aarch64' machine='virt-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/i-2-183-VM_VARS.fd</nvram>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <gic version='2'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='kvmclock'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source file='/mnt/fd7d94b4-1672-3337-89f2-7dbc82e716f2/4353bb8e-b7a0-474c-a379-0f6e85a52858' index='2'/>
      <backingStore type='file' index='3'>
        <format type='qcow2'/>
        <source file='/mnt/fd7d94b4-1672-3337-89f2-7dbc82e716f2/21b0501b-2946-4cd8-ab02-8d4117116c46'/>
        <backingStore/>
      </backingStore>
      <target dev='sda' bus='scsi'/>
      <serial>4353bb8eb7a0474ca379</serial>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hdc' bus='scsi'/>
      <readonly/>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='4'/>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x12'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x13'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x14'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x15'/>
      <alias name='pci.14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='15' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='15' port='0x16'/>
      <alias name='pci.15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='16' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='16' port='0x17'/>
      <alias name='pci.16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='17' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='17' port='0x18'/>
      <alias name='pci.17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='18' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='18' port='0x19'/>
      <alias name='pci.18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='19' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='19' port='0x1a'/>
      <alias name='pci.19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='20' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='20' port='0x1b'/>
      <alias name='pci.20'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='21' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='21' port='0x1c'/>
      <alias name='pci.21'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='22' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='22' port='0x1d'/>
      <alias name='pci.22'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='pci' index='23' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='23' port='0x1e'/>
      <alias name='pci.23'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/>
    </controller>
    <controller type='pci' index='24' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='24' port='0x1f'/>
      <alias name='pci.24'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='pci' index='25' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='25' port='0x20'/>
      <alias name='pci.25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='26' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='26' port='0x21'/>
      <alias name='pci.26'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='pci' index='27' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='27' port='0x22'/>
      <alias name='pci.27'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='28' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='28' port='0x23'/>
      <alias name='pci.28'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/>
    </controller>
    <controller type='pci' index='29' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='29' port='0x24'/>
      <alias name='pci.29'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x4'/>
    </controller>
    <controller type='pci' index='30' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='30' port='0x25'/>
      <alias name='pci.30'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x5'/>
    </controller>
    <controller type='pci' index='31' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='31' port='0x26'/>
      <alias name='pci.31'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x6'/>
    </controller>
    <controller type='pci' index='32' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='32' port='0x27'/>
      <alias name='pci.32'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='pci' index='33' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.33'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='02:00:0f:af:00:2b'/>
      <source bridge='breth0-200'/>
      <bandwidth>
        <inbound average='25600' peak='25600'/>
        <outbound average='25600' peak='25600'/>
      </bandwidth>
      <target dev='vnet6'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target type='system-serial' port='0'>
        <model name='pl011'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/i-2-183-VM.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
      <address type='usb' bus='0' port='2'/>
    </input>
    <input type='mouse' bus='usb'>
      <alias name='input2'/>
      <address type='usb' bus='0' port='3'/>
    </input>
    <graphics type='vnc' port='5902' autoport='yes' listen='192.168.1.12'>
      <listen type='address' address='192.168.1.12'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='i6300esb' action='none'>
      <alias name='watchdog0'/>
      <address type='pci' domain='0x0000' bus='0x21' slot='0x01' function='0x0'/>
    </watchdog>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

shwstppr

LGTM base on code change and given working domain XML

rohityadavcloud added this to the 4.15.2.0 milestone Jul 8, 2021

rohityadavcloud mentioned this pull request Jul 8, 2021

kvm: pre-add 32 PCI controller for hot-plug issue #4182

Closed

5 tasks

rohityadavcloud added component:kvm platform:arm64 platform:raspberrypi labels Jul 8, 2021

DaanHoogland approved these changes Jul 13, 2021

View reviewed changes

rohityadavcloud requested review from nvazquez and sureshanaparti July 15, 2021 07:10

shwstppr approved these changes Jul 15, 2021

View reviewed changes

rohityadavcloud merged commit 5228fae into apache:4.15 Jul 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

kvm: pre-add 32 PCI controller for hot-plug issue on ARM64/aarch64 #5193

kvm: pre-add 32 PCI controller for hot-plug issue on ARM64/aarch64 #5193

Uh oh!

rohityadavcloud commented Jul 8, 2021

Uh oh!

rohityadavcloud commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

rohityadavcloud commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

DaanHoogland left a comment

Uh oh!

rohityadavcloud commented Jul 13, 2021

Uh oh!

rohityadavcloud commented Jul 15, 2021

Uh oh!

shwstppr left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kvm: pre-add 32 PCI controller for hot-plug issue on ARM64/aarch64 #5193

kvm: pre-add 32 PCI controller for hot-plug issue on ARM64/aarch64 #5193

Uh oh!

Conversation

rohityadavcloud commented Jul 8, 2021

Types of changes

Uh oh!

rohityadavcloud commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

rohityadavcloud commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

blueorangutan commented Jul 8, 2021

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

rohityadavcloud commented Jul 13, 2021

Uh oh!

rohityadavcloud commented Jul 15, 2021

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants