Skip to content

kvm: pre-add 32 PCI controller for hot-plug issue #4182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
rohityadavcloud wants to merge 1 commit into from
Closed

kvm: pre-add 32 PCI controller for hot-plug issue #4182

rohityadavcloud wants to merge 1 commit into from

Conversation

@rohityadavcloud
Copy link
Member

@rohityadavcloud rohityadavcloud commented Jun 27, 2020

On newer libvirt/qemu it seems PCI hot-plugging could be an issue as
seen in:

https://www.suse.com/support/kb/doc/?id=000019383
https://bugs.launchpad.net/nova/+bug/1836065

As per the default machine doc, it advises to pre-allocate PCI
controllers on the machine and pcie-to-pci-bridge based controller
for legacy PCI models:
https://libvirt.org/pci-hotplug.html#x86_64-q35

This patch introduces the concept as a workaround until a proper fix is
done (ideally in the upstream libvirt/qemu projects). Until then client
code can add 32 PCI controllers and a pcie-to-pci-bridge controller.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

On this env:

# virsh version
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.0

I was unable to add nic to a VM (beyond 3 nics) and got the error "No more available PCI slots". After this workaround/fix I was able to add nic and disks to the VM. (The VM was using virtio-scsi for disks and virtio based nics on PCI controller)

@rohityadavcloud
kvm: pre-add 32 PCI controller for hot-plug issue
104bb17 
On newer libvirt/qemu it seems PCI hot-plugging could be an issue as
seen in:

https://www.suse.com/support/kb/doc/?id=000019383
https://bugs.launchpad.net/nova/+bug/1836065

As per the default machine doc, it advises to pre-allocate PCI
controllers on the machine and pcie-to-pci-bridge based controller
for legacy PCI models:
https://libvirt.org/pci-hotplug.html#x86_64-q35

This patch introduces the concept as a workaround until a proper fix is
done (ideally in the upstream libvirt/qemu projects). Until then client
code can add 32 PCI controllers and a pcie-to-pci-bridge controller.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
(cherry picked from commit 532b234)
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@rohityadavcloud rohityadavcloud added this to the 4.13.2.0 milestone Jun 27, 2020
@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jun 27, 2020

@kiwiflyer @weizhouapache @wido @GabrielBrascher @nvazquez @andrijapanicsb
This is a workaround and would allow upto 64 PCI/legacy-pcie nics/disks to be added. But this needs discussion, preferably a better solution than having to pre-allocate controllers (than use a hardcoded or settings-define number of controllers). Thanks.

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jun 27, 2020

@blueorangutan package

@blueorangutan
Copy link

blueorangutan commented Jun 27, 2020

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

blueorangutan commented Jun 27, 2020

Packaging result: ✔centos7 ✔debian. JID-1474

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jun 28, 2020

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jun 28, 2020

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jul 1, 2020

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jul 1, 2020

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@GabrielBrascher
Copy link
Member

GabrielBrascher commented Jul 1, 2020 •
edited
Loading

@rhtyd I have no objection to this implementation as a work-around.

I researched some alternatives, one would be Using PCI Hotplug Support. But that requires Hotplug drivers installed on all KVM nodes; and then use qemu monitor to add the devices on demand.

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jul 4, 2020

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jul 4, 2020

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jul 6, 2020

@blueorangutan test

@blueorangutan
Copy link

blueorangutan commented Jul 6, 2020

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jul 7, 2020

On hold - tests are failing, needs investigation and explore best case solution

@rohityadavcloud rohityadavcloud marked this pull request as draft July 8, 2020 05:52
@rohityadavcloud rohityadavcloud removed this from the 4.13.2.0 milestone Aug 12, 2020
@rohityadavcloud rohityadavcloud mentioned this pull request Jul 8, 2021
Merged
5 tasks
@rohityadavcloud
Copy link
Member Author

rohityadavcloud commented Jul 8, 2021

Closing this, moved to #5193

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wido wido Awaiting requested review from wido

@GabrielBrascher GabrielBrascher Awaiting requested review from GabrielBrascher

@nvazquez nvazquez Awaiting requested review from nvazquez

@kiwiflyer kiwiflyer Awaiting requested review from kiwiflyer

@andrijapanicsb andrijapanicsb Awaiting requested review from andrijapanicsb

@weizhouapache weizhouapache Awaiting requested review from weizhouapache

Assignees

No one assigned

Labels

component:kvm platform:raspberrypi status:wip type:bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rohityadavcloud @blueorangutan @GabrielBrascher @ACSGitBot