We have found (bug 928463) that kvm drastically improves test run time on the Android x86 emulators. To use that, we will need to install kvm on the hosts that run Android x86 emu tests. Generic instructions are here: https://help.ubuntu.com/community/KVM/Installation Running on the "ix loaner", I found this was sufficient: sudo apt-get install cpu-checker sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils sudo adduser cltbld libvirtd I did not install virt-manager; I did not need to chown anything. To verify installation: [cltbld@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]$ kvm-ok INFO: /dev/kvm exists KVM acceleration can be used [cltbld@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]$ virsh -c qemu:///system list Id Name State ---------------------------------- [cltbld@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]$ ls -l /dev/kvm crw-rw----+ 1 root kvm 10, 232 Oct 18 10:30 /dev/kvm

I will be reading the documentation tomorrow.

Comment on attachment 821272 [details] [diff] [review] [wip] first patch to install kvm packages I'd rather break that out into a few packages, since really only qemu-kvm of that list can be described as "kvm". We've made exceptions in the past for closely-related packages, so if libvert and/or libvirt-devel were also included here, I'd be comfortable including those under a single pacakges::libvirt class. Based on the name here, though, I suspect all five should be distinct packages::* classes.

dustin, how do I limit it to just Ubuntu x64 bit machines?

You'll need to add another conditional in modules/talos/manifests/init.pp, using $hardwaremodel which is 'x86_64' on those hosts.

It seems that I made progress but I might be introducing a bug: > Error: /Stage[main]//Resources[firewall]: Failed to generate additional resources using 'generate': Invalid address from IPAddr.new: ! 17:34 dustin: those iptables rules are added when the VM is started, I think 17:34 dustin: that might be problematic :( 17:35 dustin: armenzg: https://groups.google.com/forum/#!msg/puppet-users/4VwOUxnKxx4/W1WdDvDv8asJ 17:35 dustin: and https://github.com/puppetlabs/puppetlabs-firewall/issues/125 17:36 armenzg: dustin: what do you mean? 17:36 armenzg: is it produced by my patches? 17:37 armenzg: or is it unrelated? 17:37 dustin: I mean running puppet on a host where libvirt is also running may be a problem 17:37 dustin: it's produced by running libvirt on that machine

I will look into the iptables issue on Thursday. How does this look so far?

BTW, I tried using a pull request, however, I was not allowed to do so since I believe you have not forked from https://github.com/mozilla/build-puppet; is this a correct assesment? (I'm trying to learn to collaborate through github).

gbrown: why do we need libvirt-bin? IIUC, it only seems to be recommended/suggested: [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# apt-cache show libvirt-bin | grep "kvm" Suggests: policykit-1, apparmor, pm-utils, qemu-kvm | qemu (>= 0.9.1), radvd Enhances: qemu, qemu-kvm, xen Suggests: policykit-1, apparmor, qemu-kvm | qemu (>= 0.9.1), radvd Enhances: qemu, qemu-kvm, xen [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# apt-cache show qemu-kvm | grep "libvirt" [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# apt-cache show cpu-checker | grep "libvirt" [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# apt-cache show ubuntu-vm-builder | grep "libvirt" [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# apt-cache show bridge-utils | grep "libvirt" On another note, is this the version of QEMU-kvm we want to use? [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# kvm --version QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard This is what I see when I run with the debug flag: Debug: Puppet::Type::Firewall::ProviderIptables: [instances] Debug: Executing '/sbin/iptables-save' Error: /Stage[main]//Resources[firewall]: Failed to generate additional resources using 'generate': Invalid address from IPAddr.new: !

gbrown: could you please try uninstalling libvirt-bin, rebooting the machine and try to run the tests again? Assuming that we don't need it. I would like to know if we have that option open or not and what we gain from libvirt-bin.

Without it I get this: [root@talos-linux64-ix-100.test.releng.scl3.mozilla.com ~]# virsh -c qemu:///system list The program 'virsh' is currently not installed. You can install it by typing: apt-get install libvirt-bin

Comment on attachment 821987 [details] [diff] [review] kvm.puppet.diff PuppetAgain isn't set up for github processes - it's an hg repo and uses the usual patch-on-Bugzilla process. I think you need to remove modules/packages/manifests/kvm.pp, and also refer to each of the packages::* individually in modules/talos/manifests/init.pp. As for the iptables issue, the problem is that site.pp is set up to flush all firewall rules. If that *worked*, then Puppet and libvirt would be at war with one another, removing and re-adding the rules. However, it doesn't work - puppet's code to parse iptables output fails to parse libvirt's rules correctly, with the result that the puppet run fails. So there are two options here: 1. Fix puppet to parse the rules correctly, then fix puppet to generate and expect those rules, so that puppet and libvirt agree on what iptables rules should exist; or 2. Change the puppet manifests to not try to manage iptables at all on hosts that do not install firewall rules (currently only signing servers use firewall rules). I think #2 is probably easier. It would involve moving the firewall-related stuff from site.pp into modules/fw/manifests/init.pp, and including that from fw::port. That will need to be tested on the signing servers. It probably makes sense to break that work out into a separate bug blocking this one.

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #8) > gbrown: why do we need libvirt-bin? > > IIUC, it only seems to be recommended/suggested: I think you are correct. I was blindly following the instructions at https://help.ubuntu.com/community/KVM/Installation, but now I notice it says: "libvirt-bin provides libvirtd which you need to administer qemu and kvm instances using libvirt". As you noted, it also seems to provide virsh...but I don't know that we need to run that. I removed libvirt-bin from the loaner, rebooted, and started a set of tests...will let you know how that goes. > On another note, is this the version of QEMU-kvm we want to use? > [root@talos-linux64-ix-001.test.releng.scl3.mozilla.com ~]# kvm --version > QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Yes, that's what I see on the loaner.

(In reply to Geoff Brown [:gbrown] from comment #12) > I removed libvirt-bin from the loaner, rebooted, and started a set of > tests...will let you know how that goes. The tests seem to run fine without libvirt-bin -- let's not install libvirt-bin.

Woohoo!

Comment on attachment 822468 [details] [diff] [review] add kvm packages It might be wrong. Let me make sure it works.

Comment on attachment 822468 [details] [diff] [review] add kvm packages Review of attachment 822468 [details] [diff] [review]: ----------------------------------------------------------------- "Working" is overrated! This looks fine, so r+ conditioned on it working in your tests. I assume that not installing libvirt-bin means no spurious iptables entries?

I had a bunch of syntax issues. I fixed it now. No iptables show up on the logs. Carrying forward r+. I will land it on Monday. I'm running a bunch of desktop and emulator jobs on staging to be sure first. AFAIK, we only run talos on these machines + the emulators.

Comment on attachment 822481 [details] [diff] [review] add kvm packages https://hg.mozilla.org/build/puppet/rev/fe403a29f338

Comment on attachment 822481 [details] [diff] [review] add kvm packages https://hg.mozilla.org/build/puppet/rev/56c0f57e7d9a Backing out as it also affected the ec2 x86-64 machines.

We have landed a change that will remove libvirt-bin from the Linux x86-64 hosts. It will take a couple of hours for this to settle down. I will hope to look at this again by Wednesday once I bring buildduty under control.

FYI libvirt-bin got installed on both type of machines. Or at least, virsh got installed on both type of machines. Not just the ec2 machines; I should re-test everything.

I backed out the temp change that would uninstall libvirt-bin from every machine: https://hg.mozilla.org/build/puppet/rev/9962e72f34b5

I will be resuming this today (now that buildduty is out of the way).

Do we get everything that we need by just installing qemu-kvm? FTR ubuntu-vm-builder installed libvirt-bin. Do we need ubuntu-vm-builder? cpu-checker and bridge-utils seem to be OK.

I still have to test it with an ec2 test slave to be sure that we're good to go.

Comment on attachment 827505 [details] [diff] [review] kvm.diffadd kvm packages (without ubuntu-vm-builder) It works on ec3 as well. This does not install virsh. I would like to deploy this tomorrow if possible. I assume I have to merge default to production, correct?

Comment on attachment 827505 [details] [diff] [review] kvm.diffadd kvm packages (without ubuntu-vm-builder) This looks fine, but remember that installing virsh is not what broke things - it was the addition of iptables rules. Does this not add iptables rules?

And yes, you'll need to merge to production.

That's correct. The puppet runs were clean. It's the daemon that gets started with virsh IIUC.

Comment on attachment 827505 [details] [diff] [review] kvm.diffadd kvm packages (without ubuntu-vm-builder) Live. https://hg.mozilla.org/build/puppet/rev/16bc4953bcff

It seems it has made it this time around.