AMO add-ons get reviewed and tested by experts every time they are updated. Non-AMO add-ons get no such testing. We should test the most popular ones. The top 26 have 3 million or more users, so let's round up to the top 30. (For more details about the genesis of this bug, see bug 730737 comment 40 and onwards.) A test day would be a good start for this, and then AMO editors could hopefully review any that haven't been looked at.

Sorry for the delayed response. I haven't forgotten about this. I'll try to organize something once I get back from Blackhat/Defcon.

Bug 730737 comment 52 seems to indicate this was completed already, or am I mistaken?

(In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #2) > Bug 730737 comment 52 seems to indicate this was completed already, or am I > mistaken? No, we simply gave up.

(In reply to Justin Lebar [:jlebar] from comment #3) > (In reply to Anthony Hughes, Mozilla QA (:ashughes) from comment #2) > > Bug 730737 comment 52 seems to indicate this was completed already, or am I > > mistaken? > > No, we simply gave up. I'd like some more detail on the attached etherpad archive. What is the context of that etherpad? Was some top non-AMO add-on testing coordinated and completed? Is there still a need for QA to organize some sort of top-30 non-AMO add-ons testing? I'm still committed to working with you to deliver this event but I don't want to waste your time if it's not something you need any longer. We can discuss this in a real-time format (skype, vidyo, phone, etc) if you want -- it might be more productive then going back and forth in bugs.

> I'd like some more detail on the attached etherpad archive. What is the context of that etherpad? > Was some top non-AMO add-on testing coordinated and completed? It was top add-on testing, irrespective of AMO status. (You'll observe that Greasemonkey, for example, is in the list. As is AVG.) > Is there still a need for QA to organize some sort of top-30 non-AMO add-ons testing? Yes, because as you can see from the etherpad (and from the fact that Nick filed this bug in the first place), we did not make much headway in our testing.

Thanks for clearing that up, Justin -- and thank you for your patience. Jorge, do we have a list of the top-30 non-AMO add-ons somewhere? I'd like to get this testday coordinated as soon as possible.

One additional data point: I understand that the add-ons folks have some plan in the work for testing popular add-ons. But I don't know anything more than that.

Still working out some details here via email. I'd like to get this event up and running very soon.

I sincerely apologize this has taken so long. I've set up my first draft of a testplan here: https://wiki.mozilla.org/User:Ashughes/Testdays/Memshrink_Addons Can someone please review it and suggest improvements? I'd like to get this done in Q1.

> Can someone please review it and suggest improvements? This is a great start, esp. having the list of the add-ons and where to get them. My main comment is that it's focused entirely on memory consumption and zombie compartments. This is understandable, because we've certainly talked about them a lot in the past (e.g. bug 730737). However, zombie compartments in add-ons are now (happily) a much smaller problem than they were because we've eliminated their main cause (see https://blog.mozilla.org/nnethercote/2012/07/19/firefox-15-plugs-the-add-on-leaks/ and http://blog.kylehuey.com/post/21892343371/fixing-the-memory-leak). My intention when I filed this bug was that the top 30 non-AMO add-ons would be tested not just for zombie compartments, but just for their general behaviour. Much like AMO add-ons are tested when they are reviewed. Checking for zombie compartments could be part of that -- because they can still happen even though we eliminated the main cause of them -- but ideally other aspects would be tested. Some ideas for that: - effect on start-up time - effect on general performance - general quality -- does the add-on do what it's advertised to do? - does the add-on affect a user's set-up in undesirable ways (e.g. changing the default search provider) https://developer.mozilla.org/en-US/docs/Addons/Add-on_guidelines would be a good place to look to flesh this out more. This broadens the scope of the testing significantly, and will be likely to produce much more useful results, IMO. Does that make sense?

I'd add GC/CC times to that list. Especially since bug 713216 was filed, I've suspected that GC pauses are one of the more severe causes of UI responsiveness issues. I'd like some data to back that up or disprove it. I know they're the main cause of performance problems for one of my own extensions.

This is going nowhere.