Closed Bug 574190 Opened 14 years ago Closed 12 years ago

Update pulseaudio packages on linux32+64 test machines to 0.9.21-5

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Platform:
x86
Linux
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kinetik, Assigned: rail)

References

Details

(Whiteboard: [puppet][unittest])

Attachments

(2 files)

rpm deps
(deleted), text/plain
Details
exec rpm -U them
(deleted), patch
dustin
: review+
rail
: checked-in+
Details | Diff | Splinter Review 
One of the more frequent random orange timeouts the content/media mochitests are suffering from is caused by a hang writing audio to ALSA.  I've investigated this over in bug 573924 and found what I believe is the primary cause--a bug in the version of PulseAudio shipped with Fedora 12.

Fortunately, the necessary fix appears to have been shipped in a Fedora 12 update.  If we get this installed on the F12 test machines, I expect to see a significant reduction in the number of random timeouts in the media tests.

Update steps:
- Enable Fedora 12 updates
- yum install -y pulseaudio
  (it'll pull in a few pulseaudio packages)
- Kill all running instances of PulseAudio
  (or reboot the machine, if possible)

PulseAudio runs per-use, and is set up to automatically restart the first time an application tries to use it.  Killing all running copies is a fairly low-interruption way to make sure the update has taken.  It will affect any current running apps that are using PulseAudio (probably only running instances of Firefox on the test machines), but any subsequent app that tries to run PulseAudio will cause a new instance to start automatically.
Blocks: 573924
> per-use

per-user, I mean.
we can use yumdownloader --resolve pulseaudio.  I am not sure if yumdownloader can use the --enablerepo, but we could temporarily enable updates.  We would then use puppet to deploy the package.
Priority: -- → P4
Whiteboard: [puppet]
This bug is heading towards being two months old.  Is there anything I can do to get this moving?  It's blocking fixes for frequent random orange on the Linux build machines.
Also, please make sure that alsa-plugins-pulseaudio is updated to at least 1.0.22-1 (from Fedora updates) at the same time.
Can we get some action here please? Can we get pulseaudio updated on the Linux build machines? We're still getting frequent test failures in the media test on Linux due to bugs in pulseaudio.
(In reply to comment #5)
> Can we get some action here please? Can we get pulseaudio updated on the Linux
> build machines? We're still getting frequent test failures in the media test on
> Linux due to bugs in pulseaudio.

There is no pulseaudio on the build machines.  As I understand it, we are using the alsa plugin on the mini test machines.

From looking at this bug, we need to do

yum install pulseaudio alsa-plugins-pulseaudio .  Is this correct?
(In reply to comment #6)
> There is no pulseaudio on the build machines.  As I understand it, we are using
> the alsa plugin on the mini test machines.

Right, we only care about the machines the tests run on.

> From looking at this bug, we need to do
> 
> yum install pulseaudio alsa-plugins-pulseaudio .  Is this correct?

If the updates repository is enabled, yes.  Otherwise you need to enable the updates repo first.  And after updating, any running PulseAudio instances need to be killed for the update to take effect (per comment 0).
as we want to ensure that every machine has the same copy of the libraries, we won't be using yum.  Instead, we will download the rpms using yumdownloader --resolve.  If the repository is disabled, we can use the --enablerepo option to enable the updates repository.
Blocks: 571798
Does this look OK ?
pulseaudio-0.9.21-5.fc12.i686.rpm
pulseaudio-libs-0.9.21-5.fc12.i686.rpm
pulseaudio-libs-glib2-0.9.21-5.fc12.i686.rpm
pulseaudio-module-bluetooth-0.9.21-5.fc12.i686.rpm
pulseaudio-module-gconf-0.9.21-5.fc12.i686.rpm
pulseaudio-module-x11-0.9.21-5.fc12.i686.rpm
pulseaudio-utils-0.9.21-5.fc12.i686.rpm

That's 'yumdownloader --resolve alsa-plugins-pulseaudio pulseaudio' with the 0.9.19-2 packages from the existing install removed. (They're stashed at talos-r3-fed001:/root/pulseaudio/). 

Do the Fedora12x64 minis need this too ?
That looks fine if you can confirm that alsa-plugins-pulseaudio is already installed at 1.0.22-1, otherwise it looks like that's not showing up as an update.  Koji is showing it available in F12 updates: http://koji.fedoraproject.org/koji/buildinfo?buildID=150927

And yeah, the same change is wanted on the 12x64 machines too, please.
Oops, paste fail. Meant to say alsa-plugins-pulseaudio-1.0.22-1.fc12.i686.rpm there.
Summary: Update pulseaudio packages on Fedora 12 machines to 0.9.21-5 from Fedora 12 updates → Update pulseaudio packages on Fedora 12 machines (32 & 64bit) to 0.9.21-5 from Fedora 12 updates
Is it possible to get an ETA on this so I don't have to nag in the bug every couple of weeks, please?  I'm not sure if it's scheduled or just keeps falling off of the radar.
Can we get pulseaudio updated on the Linux mochitest machines please? We're getting a significant number of mochitest hangs due to known bugs in our out-of-date pulseaudio libraries.
Attached file rpm deps (deleted) — 
I poked around with 'rpm --test -U <rpm>' for the list in comment #10 and #12 and there's a circular dependency issue when you try to install the rpms individually, as puppet would with 'provider => rpm' (see the attachment). Perhaps the best way to do this is a single Exec of 'rpm -U <list of rpms>' ? I don't think we should set up a yum repo to deal with this, and we don't want to hit an external repo from each machine.
Assignee: nobody → nrthomas
Status: NEW → ASSIGNED
(In reply to comment #15)
> Created attachment 481416 [details]
> rpm deps
> 
> I poked around with 'rpm --test -U <rpm>' for the list in comment #10 and #12
> and there's a circular dependency issue when you try to install the rpms
> individually, as puppet would with 'provider => rpm' (see the attachment).
> Perhaps the best way to do this is a single Exec of 'rpm -U <list of rpms>' ? I
> don't think we should set up a yum repo to deal with this, and we don't want to
> hit an external repo from each machine.

The consensus in #puppet seems to be that you can't deal with circular dependencies using the rpm package provider, so that sounds like a fine workaround to me. You'll need to do something clever with the onlyif parameter or use some other way of preventing the exec {} from being executed on every run. Might be able to use something similar to:
http://hg.mozilla.org/build/puppet-manifests/file/tip/includes/functions.pp#l11
Priority: P4 → P2
I came up with a workaround for the problem and checked it in a few days ago.  Bug 573924 has the details.  There's probably no benefit to upgrading PA on our hosts now, so closing as WONTFIX.
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → WONTFIX
Resurrecting this, because I suspect we're hitting problems again in bug 620721, plus I'd like a newer version available for when bug 623444 lands to avoid even further debugging pain.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
It looks like puppet will be able to deal with the rpm misfeature of circular dependencies in an upcoming version -- see http://projects.puppetlabs.com/issues/2198

So until then we can hack around this with an exec and an onlyif.
Whiteboard: [puppet] → [puppet][unittest]
I spoke to kinetik about this today. Doing the upgrade would fix some of the orange we hit in the media tests, mostly to do with test hangs which sometimes lead to testsuite hangs. The test to do is loop over mochitest-1 (where 80% of the media tests live) on an upgraded slave and capture logs. kinetik offered to help sort through the orange in those to determine how much it helps, and to check for regressions.

If it's a win we'll figure out the puppet deployment.
Any updates?
Ping.
(In reply to Matthew Gregan [:kinetik] from comment #21)
> Any updates?

(In reply to Matthew Gregan [:kinetik] from comment #22)
> Ping.

Sorry for the delayed response. We're focused on upgrading pulseaudio on the linux builders first, as this helps us reshuffle build machines across all OS as well as move linux builds to AWS - all of which improves build wait times for all developers. Details are in bug#662417, and the most interesting work is being done in dep bug#772446. 

If this priority feels wrong, please let us know.
(In reply to John O'Duinn [:joduinn] from comment #23)
> (In reply to Matthew Gregan [:kinetik] from comment #21)
> > Any updates?
> 
> (In reply to Matthew Gregan [:kinetik] from comment #22)
> > Ping.
> 
> Sorry for the delayed response. We're focused on upgrading pulseaudio on the
> linux builders first, as this helps us reshuffle build machines across all
> OS as well as move linux builds to AWS - all of which improves build wait
> times for all developers. Details are in bug#662417, and the most
> interesting work is being done in dep bug#772446. 
> 
> If this priority feels wrong, please let us know.

Matthew Gregan [:kinetik]

Hey there. Now that bug#662417 is complete, we're looking at this again.

1) From comment#0, we cannot just enable updates for fedora, as that would update other libraries, with unknown fallout for other tests. Instead we'll use puppet to deploy specific version of this pulse audio lib to all our linux32/linux64 test machines.

2) This bug has been open for long time, so can you please verify that the version of pulse you want is still 0.9.21-5? We can then verify the dependency chain to make sure there's no surprises included.

3) Once we install pulseaudio on one test machine, can you verify that everything works as expected, and tests are passing green, before we deploy this out to the entire production pool?

4) Headsup that we are building out ubuntu32/64 test machines in AWS to offload some of the overloaded fedora12 32/64 test machines. We will verify the audio libraries on these new ubuntu instances in AWS, and check with you if they meet your needs. If you need specific hardware machines for these audio tests, we can schedule your test jobs only on the linux hardware we have inhouse. More news as we have it.

5) Anything else we need to watch for?
Depends on: 662417
Summary: Update pulseaudio packages on Fedora 12 machines (32 & 64bit) to 0.9.21-5 from Fedora 12 updates → Update pulseaudio packages on linux32+64 test machines to 0.9.21-5
Ubuntu test machines has these packages already installed:

ii  gstreamer0.10-pulseaudio               0.10.31-1ubuntu1                        GStreamer plugin for PulseAudio
ii  pulseaudio                             1:1.1-0ubuntu15                         PulseAudio sound server
ii  pulseaudio-module-bluetooth            1:1.1-0ubuntu15                         Bluetooth module for PulseAudio sound server
ii  pulseaudio-module-gconf                1:1.1-0ubuntu15                         GConf module for PulseAudio sound server
ii  pulseaudio-module-x11                  1:1.1-0ubuntu15                         X11 module for PulseAudio sound server
ii  pulseaudio-utils                       1:1.1-0ubuntu15                         Command line tools for the PulseAudio sound server

Do we need anything else?
Assignee: nthomas → rail
(In reply to John O'Duinn [:joduinn] from comment #24)

> Hey there. Now that bug#662417 is complete, we're looking at this again.

Thanks!

> 2) This bug has been open for long time, so can you please verify that the
> version of pulse you want is still 0.9.21-5? We can then verify the
> dependency chain to make sure there's no surprises included.

Whatever the newest version available from the distribution's repository are.  It looks like that's now 0.9.21-6.

> 3) Once we install pulseaudio on one test machine, can you verify that
> everything works as expected, and tests are passing green, before we deploy
> this out to the entire production pool?

As I understand it, PA is already installed and running, so this will just be updating the existing install, not installing it from new.

I'm happy to verify the test machine, absolutely.

> 4) Headsup that we are building out ubuntu32/64 test machines in AWS to
> offload some of the overloaded fedora12 32/64 test machines. We will verify
> the audio libraries on these new ubuntu instances in AWS, and check with you
> if they meet your needs. If you need specific hardware machines for these
> audio tests, we can schedule your test jobs only on the linux hardware we
> have inhouse. More news as we have it.

I'm not sure how ALSA/PA will behave with no audio hardware, I'll test this locally in a VM so I know what to expect.  Which version of Ubuntu is being rolled out?

> 5) Anything else we need to watch for?

Not that I'm aware of.

Thanks!
(In reply to Matthew Gregan [:kinetik] from comment #26)
> Whatever the newest version available from the distribution's repository
> are.  It looks like that's now 0.9.21-6.

...and the newest alsa-plugins-pulseaudio is still 1.0.22-1.

(In reply to Rail Aliiev [:rail] from comment #25)
> Ubuntu test machines has these packages already installed:

Is that Ubuntu 12.04LTS?

> Do we need anything else?

libasound2-plugins (1.0.25-1ubuntu1), but that's probably already installed if PA is.
(In reply to Matthew Gregan [:kinetik] from comment #26)
>Which version of Ubuntu is being rolled out?

12.04(In reply to Matthew Gregan [:kinetik] from comment #27)
> (In reply to Matthew Gregan [:kinetik] from comment #26)
> > Whatever the newest version available from the distribution's repository
> > are.  It looks like that's now 0.9.21-6.
> 
> ...and the newest alsa-plugins-pulseaudio is still 1.0.22-1.
> 
> (In reply to Rail Aliiev [:rail] from comment #25)
> > Ubuntu test machines has these packages already installed:
> 
> Is that Ubuntu 12.04LTS?

Correct, Ubuntu 12.04 LTS, 32 and 64 bit.
 
> > Do we need anything else?
> 
> libasound2-plugins (1.0.25-1ubuntu1), but that's probably already installed
> if PA is.

Yup:

ii  libasound2                             1.0.25-1ubuntu10                        shared library for ALSA applications
ii  libasound2-plugins                     1.0.25-1ubuntu1                         ALSA library additional plugins
BTW, do we still need pulseaudio-0.9.21-6 on Fedora slaves (since the summary has changed)?
If the Ubuntu machines are completely replacing the Fedora machines, the answer is no.  If we're going to keep a mixture around, then the Fedora machines will need the update applied.

If we're losing the Fedora machines completely, I definitely need to chase up:

(In reply to John O'Duinn [:joduinn] from comment #24)
> 4) [...] If you need specific hardware machines for these
> audio tests, we can schedule your test jobs only on the linux hardware we
> have inhouse. More news as we have it.

(In reply to Matthew Gregan [:kinetik] from comment #26)
> I'm not sure how ALSA/PA will behave with no audio hardware, I'll test this
> locally in a VM so I know what to expect.

...but that's a separate issue.
(In reply to Matthew Gregan [:kinetik] from comment #30)
> If the Ubuntu machines are completely replacing the Fedora machines, the
> answer is no.  If we're going to keep a mixture around, then the Fedora
> machines will need the update applied.

We're going to have a mix because some tests will only run on machines with real hardware.
Which test suites are affected by this? If they run properly under the VMs, then it's much better to run then there. Easier to create more VMs in Amazon than to order more physical machines!
The bulk of the media tests are in content/media/test and run in the Mochitest-1 group.  The media mochitests should run fine without audio hardware present (they have in the past, I think).  There are a few other media-using tests spread around the tree, and I'm not sure about WebRTC.

My concern is that running the tests on machines with no audio hardware will reduce test coverage of real-world configurations, because we've seen quite a few bugs that were dependent on the behaviour of audio playback.  If it's still going to be possible to run specific tests on a machine with audio hardware, that'll be fine.  Longer term, the answer is probably to improve our media tests.
Attached patch exec rpm -U them (deleted) — — Splinter Review 
Let's just upgrade the packages on Fedora as well, just to be sure. It worked fine in staging.
Attachment #708532 - Flags: review?(dustin)
Attachment #708532 - Flags: review?(dustin) → review+
Comment on attachment 708532 [details] [diff] [review]
exec rpm -U them

http://hg.mozilla.org/build/puppet-manifests/rev/e80cd7750506
Attachment #708532 - Flags: checked-in+
Most of the Fedora tests slaves (32 and 64 bit) have the upgraded version now. I will verify the remaining ones later today.
All done here. Some of the slaves didn't get the updated yet - some waiting for hard reboot, some of them needs buildduty love (in their tracked bugs).
Status: REOPENED → RESOLVED
Closed: 14 years ago → 12 years ago
Resolution: --- → FIXED
Blocks: 837971
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: