Discovery Pane has changed lately and some of the panes from the Collection Panel were removed Those related mozmill tests are failing due to lack of data to test on, so they should be urgently removed from hg. We are currently working on a new testing strategy for the Collections Panel automation tests The tests I am talking about are * bug 664019 * bug 657492

Please feel free to invalidate this bug if considering this is not a proper place for this request

Requesting feedback from Henrik. Can we simply backout the changesets from those respective bugs?

Where are the bugs which cover the removal of those panels? I would like to see those first in the dependency list before I give any statement regarding their removal.

(In reply to Henrik Skupin (:whimboo) from comment #3) > Where are the bugs which cover the removal of those panels? I would like to > see those first in the dependency list before I give any statement regarding > their removal. Well, my suggestion would be to simply close this bug and take care of the backouts on the respective bugs (since that's where the patches live). Vlad, can you identify ALL the tests/bugs which need backout?

No, this bug is perfectly valid for this task but we have to reference the bugs for removal of those pages.

(In reply to Henrik Skupin (:whimboo) from comment #5) > No, this bug is perfectly valid for this task but we have to reference the > bugs for removal of those pages. Alright then. We have two bugs in comment 0. Vlad are there any other bugs?

No. I've provided all the needed data

I do not talk about the bugs which cover the test implementation, but the bugs for the removal of those panels on the product side. Also I don't see any failure for the pick of the month test. Can you please list those failing results?

I can't find the bugs for removal on the product side. I'm sure I am not looking in the right place. I've spoken with Krupa about it and she said that marketing department handles those pane and that there isn't a specific development schedule for those. I also don't see errors from Pick of the Month pane test, but keep in mind that the test is skipped on aurora and default branch - it only works for beta and release - and I don't see those branches in the report as well Either way, those panes do not exist anymore in the current Discovery Pane.

So bug 661651 is about some changes of the pick of the month panel. But I don't follow it completely. Even with an admin tool now in place, where does this selected add-on now appear? Can you please get an answer for that question? If it simply has another location we only would have to update the current test. Well with 7.0 it fails: http://mozmill-release.brasstacks.mozilla.com/#/remote/report/cbba999432ff4d1adda4c1946a122e6f

It seems that panels reappear in the disc pane, i believe the marketing team decides which panels are active and when - we should not back out the tests just come up with some kind of a runtime decision algorithm in the API which would tell us if a panel is active/visible or not - then do the test, if not, skip test. wontfix then

If marketing is going to decide when and what panes will be visible without a moment's notice I suggest we shouldn't have tests for this feature. It's going to make them far too unstable. On the other hand, if there will be a set of baseline panes which will always be there it should be safe to have tests for that. Vlad can you please find out which panels will never change and which panels will change? We should then back out the tests for changing panels and scrap them. Reopening for now so we can do some more investigation.

I'll get with Krupa in one of the webqa meetings see if we can get someone to help asap. My answer will come in the bugs and also on the e-mails

Well there seem to be no specific rule in to how or which the panes change. Our disc pane tests will fail if there are not day-to-day monitored, and I don't think anyone has time to do that right now.

This is a wontfix. the best decision is to disable the tests because the panes may reappear again and we can re-enable and make usage of the disc pane remote tests. A good example of a pane coming back is "Pick of the Month" add-on pane Wontfix here and continue this on bug 732353

Thanks Vlad, I agree with this assessment.