Paul Rouget [:paul] Reporter
	Description • 12 years ago

We are running the devtools tests with the toolbox docked. It would be great to re-run the test with the toolbox undocked as well.

If we re-run all the tests, that will multiply by two the time spend testing the tools. Maybe we could flag some tests as "test-once" (maybe the one less likely to be affected by the undock mechanism).

	Paul Rouget [:paul] Reporter
	Comment 1 • 12 years ago

In Makefile.in, we could define the tools to run twice, the one to run once, run all the tests, and then do some magic to change the pref for the second run.

	Girish Sharma [:Optimizer]
	Comment 2 • 12 years ago

An idea: Have an extra test at the end of the list of tests in the makefile. That test toggles the specific preference that will be used in the tests to have the toolbox docked or undocked. Since that test will also run 2 times, the preference will be restored to default state after all tests have completed.

	Joe Walker [:jwalker]
	Comment 3 • 12 years ago

I'm currently removing all the hardcoded hosts from the unit tests so we should be able to flip where the toolbox is located fairly easily.

Aside from specific tests of toolbox hosting, we don't currently know how much we need to run normal tests in different locations.

Once we can simply run the tests with different hostings, we should be in a position to know how important it is to do this routinely.

	Joe Walker [:jwalker]
	Comment 4 • 12 years ago

This patch isn't a candidate for central, but should inform us of how likely tests are to break in non-default hostings.

	Joe Walker [:jwalker]
	Comment 5 • 12 years ago

https://tbpl.mozilla.org/?tree=Try&rev=c4dbe85e3dfd

	Paul Rouget [:paul] Reporter
	Comment 6 • 12 years ago

I looked at the raw logs. I only see debugger failures. This is surprising. Am I missing something?

	Victor Porof [:vporof][:vp]
	Comment 7 • 12 years ago

(In reply to Paul Rouget [:paul] from comment #6)
> I looked at the raw logs. I only see debugger failures. This is surprising.
> Am I missing something?

Maybe the toolbox window needs to be focused?

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 8 • 12 years ago

2 of the Mac failures are probably caused by bug 726616, judging by the screenshots. For the other platforms, I think we can blame the leaktest failure to the delayed GCLI test cleanups, but we need to investigate the watch expressions that come up next.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 9 • 12 years ago

I have fixes for some of these and working on the others. Taking.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 10 • 12 years ago

This was a largely mechanical change to make sure debugger tests do not rely on events propagating to or from the tab window.

Try run with Joe's patch to launch the toolbox in a separate window:
https://tbpl.mozilla.org/?tree=Try&rev=c226a50ba9ce

I should note that tests pass locally with both a detached and an attached toolbox.

	Victor Porof [:vporof][:vp]
	Comment 11 • 12 years ago

(In reply to Panos Astithas [:past] from comment #10)

There's a bit of carnage in that try run. Not sure if it's because Joe's patch, or this one, but the changes look good in principle.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 12 • 12 years ago

I took a look at the oranges and it looks like some things regressed in the last two weeks after Joe's try push. There is only one failing debugger test in Windows and this is while the test has finished successfully, so it looks like the test just takes longer in those cases with a detached toolbox.

I can reproduce the first failures locally even when I only run the framework tests, without interference from previously run debugger tests. It should also be noted that this patch only touches tests, not product code, so I feel confident that the new failures are not caused by this patch.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 13 • 12 years ago

Verified that without my patch, the other failures still occur. Landed the debugger test fixes:
https://hg.mozilla.org/integration/fx-team/rev/a030d7a75344

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 14 • 12 years ago

https://hg.mozilla.org/mozilla-central/rev/a030d7a75344

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 15 • 12 years ago

Debugger tests are OK now, so I'm giving this back to Joe.

	Joe Walker [:jwalker]
	Comment 16 • 11 years ago

Comment on attachment 698416 [details] [diff] [review]
Test fixes

This is a very old patch. Do we know it applies etc?
I took checkin+ off in case this was a mistake. We can easily add it back if it wasn't.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 17 • 11 years ago

Comment on attachment 698416 [details] [diff] [review]
Test fixes

checkin+ indicates that the patch has landed (see comment 13), not that I'm asking someone to land it (that's checkin?). What I should have explained is that I added this so that bugmotodo.org stops showing this in my "To Check In" list :-)

	Joe Walker [:jwalker]
	Comment 18 • 11 years ago

(In reply to Panos Astithas [:past] from comment #17)
> Comment on attachment 698416 [details] [diff] [review]
> Test fixes
> 
> checkin+ indicates that the patch has landed (see comment 13), not that I'm
> asking someone to land it (that's checkin?). What I should have explained is
> that I added this so that bugmotodo.org stops showing this in my "To Check
> In" list :-)

I see. I didn't want it to be seen as "permission to land", this is such an old bug that I was surprised by the traffic. Thanks.

	Joe Walker [:jwalker]
	Comment 19 • 11 years ago

I won't have time to work on this.

	Panos Astithas (he/him) [:past] (please ni?)
	Comment 20 • 10 years ago

I ran all of our tests again with attachment 693344 [details] [diff] [review] applied and everything works. I'm closing the bug, but feel free to reopen if you disagree.