Andreea Matei [:AndreeaMatei] Reporter
	Description • 10 years ago

This fails on OS X, with latest Nightly. It must be some regression, I can reproduce on my mac but not on Linux.

Will investigate further.

This is the affected line:
http://hg.mozilla.org/qa/mozmill-tests/file/default/firefox/tests/addons/ide%40seleniumhq.org/lib/selenium.js#l117

Report:
http://mozmill-daily.blargon7.com/#/addons/report/4e9e123ff3e2bbcb949b21929201a084

	Andrei Eftimie Assignee
	Comment 1 • 10 years ago

Reducing the regression range

	Andrei Eftimie Assignee
	Comment 2 • 10 years ago

Regressor is bug 1008455 with https://hg.mozilla.org/integration/mozilla-inbound/rev/b42ec325e34d

Looks like it might have broken selenium-ide.
We are using an older version: 2.3.0.
I've tested with the latest stable 2.5.0 and we still get the same error.

As per bug bug 1008455 comment 29 this addon looks to be affected.
Jorge, what should we expect here? Will the addon need to be updated?

	Andrei Eftimie Assignee
	Comment 3 • 10 years ago

Also tagging Jonathan here.

	Jonathan Watt [:jwatt]
	Comment 4 • 10 years ago

What is that Selenium code actually doing? In other words, what gecko functionality is it relying on that's behaving differently now and causing it to break?

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 5 • 10 years ago

(In reply to Jonathan Watt [:jwatt] from comment #4)
> What is that Selenium code actually doing? In other words, what gecko
> functionality is it relying on that's behaving differently now and causing
> it to break?

Dave might be able to shed some light on it, or at least point us to the current maintainer.

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 6 • 10 years ago

The current maintainer for Selenium IDE is Samit Badle (CC'd). I can replicate an issue with Selenium IDE 2.5.0 against Firefox Nightly. The Selenium IDE window is blank, and the Browser Console shows "TypeError: this.treebox is undefined" pointing to https://github.com/SeleniumHQ/selenium/blob/68b415adb11acca4286be45124325cc36cfc4a51/ide/main/src/content/treeView.js#L305

	Jorge Villalobos [:jorgev] (he/him)
	Comment 7 • 10 years ago

It's likely they are injecting XUL code into an HTML document, in which case they need to modify their code for it to continue working.

	Samit Badle
	Comment 8 • 10 years ago

In this instance, there is no XUL being injected into HTML. Has there been any changes to nsiTreeView, especially the setTree() method?

	Jorge Villalobos [:jorgev] (he/him)
	Comment 9 • 10 years ago

(In reply to Samit Badle from comment #8)
> In this instance, there is no XUL being injected into HTML. Has there been
> any changes to nsiTreeView, especially the setTree() method?

:jwatt is probably the right person to ask about this.

	Cosmin Malutan, [:cosmin-malutan]
	Comment 10 • 10 years ago

This affects Aurora too and it fails constantly, we should skip until it get's fixed. 
http://mozmill-daily.blargon7.com/#/addons/report/c69af0ad3436e0d472c1ca8a06e5d913

	Cosmin Malutan, [:cosmin-malutan]
	Comment 11 • 10 years ago

Here is the skip patch, I have tested it on nightly and aurora.

	Andrei Eftimie Assignee
	Comment 12 • 10 years ago

Comment on attachment 8437617 [details] [diff] [review]
skip.patch

Review of attachment 8437617 [details] [diff] [review]:
-----------------------------------------------------------------

Disabled:
https://hg.mozilla.org/qa/mozmill-tests/rev/8351f61b162b (default)
https://hg.mozilla.org/qa/mozmill-tests/rev/930bde25e81d (mozilla-aurora)

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 13 • 10 years ago

Landing that skip patch was not that helpful, given that you are running an empty testrun now, which bails out. What is the next step of action here?

	Samit Badle
	Comment 14 • 10 years ago

I think I have a solution. I will try to pass on a beta version tomorrow. Now for 4 hours of sleep...

	Cosmin Malutan, [:cosmin-malutan]
	Comment 16 • 10 years ago

Comment on attachment 8438239 [details] [diff] [review]
dummy_test.patch

I propose we add a dummy test, similar as we have on remote suite, so we can ran that in case everything else is broken.

	Cosmin Malutan, [:cosmin-malutan]
	Comment 17 • 10 years ago

I added the test as a follow up so we don't have to backout the skip patch first.

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 18 • 10 years ago

This is actually a bug in Jenkins that has been fixed. Upgrading should prevent the build failure. See https://issues.jenkins-ci.org/browse/JENKINS-7970

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 19 • 10 years ago

(In reply to Dave Hunt (:davehunt) from comment #18)
> This is actually a bug in Jenkins that has been fixed. Upgrading should
> prevent the build failure. See
> https://issues.jenkins-ci.org/browse/JENKINS-7970

Well, it's not that bug as filed here, but you might be right in terms of Jenkins and handling zero junit reports. We could try out how Jenkins behaves on staging soon, once I upgraded it to 1.554.2. If it's not fixed we should file an issue on mozmill-ci.

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 20 • 10 years ago

(In reply to Henrik Skupin (:whimboo) from comment #19)
> (In reply to Dave Hunt (:davehunt) from comment #18)
> > This is actually a bug in Jenkins that has been fixed. Upgrading should
> > prevent the build failure. See
> > https://issues.jenkins-ci.org/browse/JENKINS-7970
> 
> Well, it's not that bug as filed here, but you might be right in terms of
> Jenkins and handling zero junit reports. We could try out how Jenkins
> behaves on staging soon, once I upgraded it to 1.554.2. If it's not fixed we
> should file an issue on mozmill-ci.

Yes, sorry for not being clear. I was referring to Jenkins treating all skipped tests as a build failure.

	Andrei Eftimie Assignee
	Comment 21 • 10 years ago

Comment on attachment 8438252 [details] [diff] [review]
followup_dummy_test.patch

Review of attachment 8438252 [details] [diff] [review]:
-----------------------------------------------------------------

Landed:
https://hg.mozilla.org/qa/mozmill-tests/rev/a5bf6e119076 (default)

	Andrei Eftimie Assignee
	Comment 22 • 10 years ago

Also transplanted on mozilla-aurora:
https://hg.mozilla.org/qa/mozmill-tests/rev/f172cbe94bdf (mozilla-aurora)

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 23 • 10 years ago

(In reply to Dave Hunt (:davehunt) from comment #20)
> Yes, sorry for not being clear. I was referring to Jenkins treating all
> skipped tests as a build failure.

And that has been fixed on staging now. Thanks for the side-node Dave!

Samit, whenever you have a new XPI for Selenium IDE please let us know, so we can give it a go for testing. Thanks.

	Andrei Eftimie Assignee
	Comment 24 • 10 years ago

Hi Samit,

Any updates on the fix for Selenium IDE?

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 25 • 10 years ago

Andrei, I'm not sure why you have removed the needinfo flag from jwatt. I have seen it just now. This should not have happened, given that we still not know what could have been caused this. So comment 9 still applies.

	Jonathan Watt [:jwatt]
	Comment 26 • 10 years ago

I don't have any guesses as to what's going on. Right now all I've got to work with is a large and complicated piece of code (selenium-ide) that is doing stuff with XUL things that I don't know anything about (nsITreeView) and which I don't know how to set up and debug. It would take me a long time to get to the bottom of this, and I don't have that time right now. If someone can create a reduced testcase (that can be run with the pref dom.allow_XUL_XBL_for_file set to true) I'll take a look at that.

	Andrei Eftimie Assignee
	Comment 27 • 10 years ago

Hi Samit,
we're still not testing the Selenium IDE here.

I will unassign since there's no action we can take here.

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 28 • 10 years ago

Samit, we strongly think about to stop add-on tests, given that we only have tests for Selenium IDE and those are not getting run due to this breakage. We would appreciate your help in getting this fixed. If it wont happen in the next weeks, we might wanna take the tests offline. Thanks.

	Samit Badle
	Comment 29 • 10 years ago

Hi Henrik,

I have sent you an internal build by email to see if this issue is fixed. If the new build fixes the issue, I can make a new release this weekend.

Cheers,
Samit

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 30 • 10 years ago

Great and thank you! Andrei will get this tested, so we will know soon if that bug can finally be marked as fixed soon.

	Andrei Eftimie Assignee
	Comment 31 • 10 years ago

Mozmill (actually the runner itself) won't install this particular addon build.
I can manually install it, but I'm unable to run our tests against it.

I've tracked the issue down to mozrunner, we pass everything correctly along until there.
Will have to dive deeper to see why this fails, but I probably won't have time today for this.

I'll leave the needinfo flag on to revisit this a bit later.

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 32 • 10 years ago

Which failure do you get? That would be important to know.

	Andrei Eftimie Assignee
	Comment 33 • 10 years ago

We don't get the addon icon in the toolbar. This is the actual failure that we can't find the addon "menu entry".

If I try installing manually this works (we do have the addon icon in the toolbar). In the testrun we do download to the correct location and pass everything from mozmill-automation to mozmill and to mozrunner (up to this point everything is identical - in the python objects passed around - to the previously used version).

	Dave Hunt [:davehunt] [he/him] ⌚BST
	Comment 34 • 10 years ago

It could be because Selenium IDE is a multi-package xpi. Try just using the editor xpi.

	Andrei Eftimie Assignee
	Comment 35 • 10 years ago

Thanks for the tip Dave, that did help indeed!
I've extracted the selenium-ide.xpi from this multi-package and it work.

Samit, this build has fixed the original issue.

We have however some other failures like:
> 'Test case passed' should equal 'echo: final command'
See the report: http://mozmill-crowd.blargon7.com/#/addons/report/12a8568e34c97b929089c7a61fb19640

Are those 'echo:...' debug statements?

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 36 • 10 years ago

Samit, best would be if you would run the tests on your own. See here how to do it:
https://developer.mozilla.org/en-US/docs/Mozmill_Tests/Addon_tests

That way we could sort out what's remaining to be fixed.

	Samit Badle
	Comment 37 • 10 years ago

I am glad this fixed the original issue. if the 'Test case passed ..." is the only failure, it should not be a big issue. This is a new feature request and the test should be updated to check for that instead. I am not able to access the http://mozmill-crowd.blargon7.com/#/addons/report/12a8568e34c97b929089c7a61fb19640 to check for any more failures. Do I need to be in the Mozilla network to access this?

I will try to run the tests myself today and see how it goes.

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 38 • 10 years ago

(In reply to Samit Badle from comment #37)
> I am not able to access the
> http://mozmill-crowd.blargon7.com/#/addons/report/
> 12a8568e34c97b929089c7a61fb19640 to check for any more failures. Do I need
> to be in the Mozilla network to access this?

You should be able to access. It's publicly available. Maybe you tried too fast yesterday and couchdb was still indexing all the new reports. Please check again.

	Samit Badle
	Comment 40 • 10 years ago

You were right. I was able to access it now. I have made some changes to the tests and attached a zip file of the ide@seleniumhq.org folder. I was however not able to create the right environment to test them. If this does not work, I will try to setup the environment as soon as possible and look into it deeper. Thanks a lot for your patience.

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 41 • 10 years ago

Thanks Samit. Would you mind to attach a diff of all the files? Then you can request review. Please see our docs how to do that:

https://developer.mozilla.org/en-US/docs/Mozilla/QA/Mozmill_tests#The_review_process

	Andrei Eftimie Assignee
	Comment 42 • 10 years ago

Here's a patch from Samit's changes. 
I've cleaned it up a bit.

(Note that we have changes to file permissions. I've checked and we have 644 on all other tests, so the change here is welcomed).

Here's a report (all GREEN):
http://mozmill-crowd.blargon7.com/#/addons/report/c456e942dfe0fd25bc60cb52771d6884

WHile this is all good, this has been run with the new build, we'll have to wait for an official release and update addon.ini to point to the new version.

	Samit Badle
	Comment 43 • 10 years ago

I was fixing some bugs. The new release should be out this weekend.

	Samit Badle
	Comment 44 • 10 years ago

The new version of Selenium IDE 2.7.0 has been released. You can get the main editor directly from here: http://release.seleniumhq.org/selenium-ide/editor/2.7.0/selenium-ide-editor-2.7.0.xpi

This is the editor which is included in the multi-xpi.

Please let me know if this fixes the issues.

Thanks a lot.

	Andrei Eftimie Assignee
	Comment 45 • 10 years ago

Thanks Samit for all the work here.

This patch updates the version of Selenium IDE to 2.7.0 and includes all fixes that Samit sent earlier. All tests are passing on every os:
http://mozmill-crowd.blargon7.com/#/addons/report/ee6ae3df35a164c4abc5cec1a397ebb1
http://mozmill-crowd.blargon7.com/#/addons/report/ee6ae3df35a164c4abc5cec1a396778f
(mozmill-dashboard doesn't want me to show the linux report, but all is green there as well).

I've left the patch author as Samit since he did all work, I just cleaned it up a bit.

	Andreea Matei [:AndreeaMatei] Reporter
	Comment 46 • 10 years ago

Comment on attachment 8489351 [details] [diff] [review]
fix2.patch

Review of attachment 8489351 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, just need to update the commit message. Great that we can re-enable this. Henrik, would you have a final look please?

	Henrik Skupin [:whimboo][⌚️UTC+2]
	Comment 47 • 10 years ago

Comment on attachment 8489351 [details] [diff] [review]
fix2.patch

Review of attachment 8489351 [details] [diff] [review]:
-----------------------------------------------------------------

I think that update is fine. There is one nit, which can be fixed in a follow-up. Thanks Samit!

r=me, and please get this landed!

::: firefox/tests/addons/ide@seleniumhq.org/lib/selenium.js
@@ +206,5 @@
> +      if (message.substr(0,5) == 'echo:') {
> +        return message;
> +      }
> +    }
> +    return 'No echo info log message found';

If nothing gets found we should really return null. Otherwise it is hard to verify. But that shouldn't be a blocker for this landing. You might wanna improve it in a follow-up bug if wanted.

	Andrei Eftimie Assignee
	Comment 48 • 10 years ago

Updated the nit and landed:
https://hg.mozilla.org/qa/mozmill-tests/rev/841d9d95194e (default)

Will backport carefully during the following days.

	Andrei Eftimie Assignee
	Comment 49 • 10 years ago

Everything ran fine on Nightly.
Aurora is all green.

Transplanted:
https://hg.mozilla.org/qa/mozmill-tests/rev/eec7fc8c7220 (mozilla-aurora)

	Andrei Eftimie Assignee
	Comment 50 • 10 years ago

Beta works fine as well: http://mozmill-crowd.blargon7.com/#/addons/report/bfd52839a01858de2f33d25f768119d5

Transplanted: https://hg.mozilla.org/qa/mozmill-tests/rev/bcd7ff416375 (mozilla-beta)

	Andrei Eftimie Assignee
	Comment 51 • 10 years ago

And release:
http://mozmill-crowd.blargon7.com/#/addons/report/bfd52839a01858de2f33d25f768268df

Transplanted:
https://hg.mozilla.org/qa/mozmill-tests/rev/00ef74568bf8 (mozilla-release)

	Andrei Eftimie Assignee
	Comment 52 • 10 years ago

ESR31 is actually unaffected, and the tests aren't disabled.
This is FIXED.

Thanks Samit for updating the addon and fixing the tests.