Closed
Bug 1043699
Opened 10 years ago
Closed 10 years ago
Add Mulet support to b2gdesktop Reftest runner
Categories
(Testing :: Reftest, defect)
Testing
Reftest
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla35
People
(Reporter: jgriffin, Assigned: armenzg)
References
Details
Attachments
(8 files, 19 obsolete files)
(deleted),
patch
|
ochameau
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jgriffin
:
review+
|
Details | Diff | Splinter Review |
(deleted),
patch
|
jlund
:
review+
|
Details | Diff | Splinter Review |
We want to be able to run reftests on Mulet the same way as we do for b2gdesktop...i.e., in side the test-container gaia app.
The current b2gdesktop runner completely fails with Mulet at present. It looks like there are several reasons for this:
- There are some gaia services that make non-local network connections, so MOZ_DISABLE_NONLOCAL_CONNECTIONS causes us to crash
- Launching the test-container app by setting b2g.system_startup_url fails; we're going to need to launch it via Marionette, which also probably means having to kill the FTU and lockscreen via Marionette
- Mulet doesn't recognize the --screen 800x1000 argument
Reporter | ||
Comment 1•10 years ago
|
||
WIP, this allows the runner to at least launch Mulet with Gaia running in it.
Reporter | ||
Updated•10 years ago
|
Assignee: nobody → jgriffin
Reporter | ||
Comment 2•10 years ago
|
||
I don't know how to cope with the fact that Mulet doesn't honor the --screen 800x1000 argument. Alexandre, can we add support for this? Reftests require this resolution and many will give false positives or negatives when run at the normal Gaia screensize.
Flags: needinfo?(poirot.alex)
Comment 3•10 years ago
|
||
The code to handle --screen is in Mulet, but the -chrome chrome://b2g/content/shell.html
arguments goes through another startup codepath which prevent window.arguments[0]
to be set to the nsICommandLine object! It ends up breaking b2g code implementing --screen argument.
Updated•10 years ago
|
Flags: needinfo?(poirot.alex)
Reporter | ||
Updated•10 years ago
|
Attachment #8476005 -
Flags: review?(gavin.sharp)
Comment 4•10 years ago
|
||
Comment on attachment 8476005 [details] [diff] [review]
Ensure that -chrome URI doesn't prevent window.arguments[0] to be nsICommandLine
This almost certainly breaks Firefox, which doesn't expect its windows to receive a reference to the cmdLine object. If b2g expects that, it needs to adjust, or we need a special case.
Attachment #8476005 -
Flags: review?(gavin.sharp) → review-
Comment 5•10 years ago
|
||
Given comment 4, it looks like window.arguments[0] being nsICommandLine
is an unexpected behavior...
So instead of fixing that to an expected behavior, I'm now working around this issue.
Attachment #8476005 -
Attachment is obsolete: true
Attachment #8480521 -
Flags: review?(fabrice)
Comment 6•10 years ago
|
||
Updated•10 years ago
|
Attachment #8480521 -
Flags: review?(fabrice) → review+
Comment 7•10 years ago
|
||
Forgot to add CommandLine.js in package-manifest.in...
https://tbpl.mozilla.org/?tree=Try&rev=e45bd7303b25
Attachment #8480521 -
Attachment is obsolete: true
Attachment #8480704 -
Flags: review+
Comment 9•10 years ago
|
||
Keywords: checkin-needed
Comment 10•10 years ago
|
||
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
Reporter | ||
Comment 11•10 years ago
|
||
Forgot to 'leave-open'; there's still more work to be done.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Reporter | ||
Updated•10 years ago
|
Blocks: firefox-mulet
Assignee | ||
Updated•10 years ago
|
Assignee: jgriffin → armenzg
Assignee | ||
Comment 12•10 years ago
|
||
With jgriffin's patch we get it started, however, we have the lock screen.
Can marionette help me unlock the screen?
For reference, here's what it shows without the patch:
http://people.mozilla.org/~armenzg/sattap/86644eb8.png
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette/marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=/home/armenzg/moz/tmp/mulet/firefox/firefox-bin --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/alissy@mozilla.com-b1950fdbd196/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip reftest/tests/layout/reftests/reftest.list
Attachment #8462225 -
Attachment is obsolete: true
Reporter | ||
Comment 13•10 years ago
|
||
Yes, you can use Marionette to unlock the lock screen.
There is already a script executed via Marionette for B2G reftests: http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/b2g_start_script.js
I think you could adjust this to include code to unlock the lockscreen, namely the 'unlock' method from here: https://github.com/mozilla-b2g/gaia/blob/master/tests/atoms/gaia_lock_screen.js
Assignee | ||
Comment 14•10 years ago
|
||
I added gaia_lock_screen.js temporarily by b2g_desktop.py and I called it like this:
+ self.marionette.import_script(os.path.abspath(
+ os.path.join(__file__, os.path.pardir, "gaia_lock_screen.js")))
+ self.marionette.switch_to_frame()
+ self.marionette.execute_async_script('GaiaLockScreen.unlock()')
I was hoping to try to emulate this code:
https://github.com/mozilla-b2g/gaia/blob/master/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L682
There is also this that uses executeScript() instead
https://github.com/mozilla-b2g/gaia/blob/master/tests/js-marionette/helper.js#L7
My apologies it is clear on how to proceed. I've not deal much with JavaScript in the past. I should be able to make progress with some pointers.
However, I'm getting this:
1410556331093 Marionette DEBUG accepted connection on 127.0.0.1:44664
1410556336101 Marionette DEBUG accepted connection on 127.0.0.1:44665
1410556336127 Marionette INFO loaded marionette-listener.js
1410556336127 Marionette INFO sendToClient: {"from":"0","value":{"browserName":"B2G","browserVersion":"35.0a1","platformName":"LINUX","platformVersion":"35.0a1","handlesAlerts":false,"nativeEvents":false,"rotatable":true,"secureSsl":false,"takesElementScreenshot":true,"takesScreenshot":true,"platform":"LINUX","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","appBuildId":"20140911230945","device":"desktop","version":"35.0a1","b2g":true}}, {ec6b34c4-cc4a-4228-9fba-e05353186962}, {ec6b34c4-cc4a-4228-9fba-e05353186962}
1410556336130 Marionette DEBUG Got request: setContext, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"setContext","parameters":{"value":"chrome"}}, id: {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336130 Marionette INFO sendToClient: {"from":"0","ok":true}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}, {4e2aa646-d696-44dd-8f1c-091b9416e8e0}
1410556336132 Marionette INFO sendToClient: {"from":"0","ok":true}, {268a247d-8b5b-42d9-90f6-b13573756a84}, {268a247d-8b5b-42d9-90f6-b13573756a84}
1410556336133 Marionette DEBUG Got request: switchToFrame, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"switchToFrame","parameters":{"id":null,"focus":true}}, id: {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336234 Marionette INFO sendToClient: {"from":"0","ok":true}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}, {5ecb34f3-5a79-429e-b5ec-ac340cbebf07}
1410556336238 Marionette DEBUG Got request: executeWithCallback, data: {"to":"0","sessionId":{"rotatable":true,"browserVersion":"35.0a1","takesScreenshot":true,"appBuildId":"20140911230945","XULappId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","secureSsl":false,"platform":"LINUX","browserName":"B2G","version":"35.0a1","device":"desktop","b2g":true,"nativeEvents":false,"platformVersion":"35.0a1","takesElementScreenshot":true,"platformName":"LINUX","handlesAlerts":false},"name":"executeAsyncScript","parameters":{"scriptTimeout":null,"specialPowers":false,"script":"GaiaLockScreen.unlock()","newSandbox":true,"args":[],"filename":"b2g_desktop.py","line":44}}, id: {7585ad9a-0d1c-454c-9805-631b56ea66a0}
1410556336242 Marionette INFO sendToClient: {"from":"0","error":{"message":"TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function","status":17,"stacktrace":"execute_async_script @b2g_desktop.py, line 44\ninline javascript, line 10\nsrc: \"undefined\""}}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}, {7585ad9a-0d1c-454c-9805-631b56ea66a0}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmps6jzGQ/minidumps/1b6fce7d-f369-4d0f-22db530f-198a27dd.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
File "reftest/runreftestb2g.py", line 611, in <module>
sys.exit(main())
File "reftest/runreftestb2g.py", line 606, in main
return run_desktop_reftests(parser, options, args)
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 191, in run_desktop_reftests
sys.exit(reftest.run_tests(args[0], options))
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 99, in run_tests
self.run_marionette_script()
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 44, in run_marionette_script
self.marionette.execute_async_script('GaiaLockScreen.unlock()')
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1253, in execute_async_script
filename=os.path.basename(frame[0]))
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
return func(*args, **kwargs)
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
self._handle_error(response)
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: TypeError: window.wrappedJSObject.SettingsListener.getSettingsLock is not a function
stacktrace:
execute_async_script @b2g_desktop.py, line 44
inline javascript, line 10
src: "undefined"
Attachment #8488673 -
Attachment is obsolete: true
Reporter | ||
Comment 15•10 years ago
|
||
Comment on attachment 8488849 [details] [diff] [review]
mulet.diff
Review of attachment 8488849 [details] [diff] [review]:
-----------------------------------------------------------------
I think this is very close, just a chrome/content mismatch here! Nice work!
::: ../mulet.original/reftest/b2g_desktop.py
@@ +36,5 @@
> def run_marionette_script(self):
> self.marionette = Marionette(**self.marionette_args)
> assert(self.marionette.wait_for_port())
> self.marionette.start_session()
> self.marionette.set_context(self.marionette.CONTEXT_CHROME)
The lockscreen is accessible only in content context, so you'll need to remove this line that sets the context to CONTEXT_CHROME (context defaults to content), and then add this line back after you execute the GaiaLockScreen.unlock() method.
@@ +92,5 @@
> log.info("%s | Application pid: %d",
> os.path.basename(__file__),
> self.runner.process_handler.pid)
> + import time
> + time.sleep(15)
If this is still necessary after fixing the context problem above, we'll have to add code to wait for the homescreen. See http://mxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#233 for an example of how this can be done.
Assignee | ||
Comment 16•10 years ago
|
||
I was loosing hope and was going to study deeper!
It works (I see the screen being unlocked) and it seems the sleep is not needed.
I'm now hitting:
1410557694318 Marionette INFO sendToClient: {"from":"0","error":{"message":"NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 100\nsrc: \"Cu.import(\"chrome://reftest/content/reftest.jsm\", reftest);\""}}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}, {ebb7e00c-beeb-449b-80a3-059b38009f8a}
PROCESS-CRASH | b2g_desktop.py | application crashed [unknown top frame]
Crash dump filename: /tmp/tmpNLLpCD/minidumps/05be21c9-af6e-d77c-2a4fd4ca-79da497b.dmp
No symbols path given, can't process dump.
MINIDUMP_STACKWALK not set, can't process dump.
Traceback (most recent call last):
File "reftest/runreftestb2g.py", line 611, in <module>
sys.exit(main())
File "reftest/runreftestb2g.py", line 606, in main
return run_desktop_reftests(parser, options, args)
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 196, in run_desktop_reftests
sys.exit(reftest.run_tests(args[0], options))
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 104, in run_tests
self.run_marionette_script()
File "/home/armenzg/moz/tmp/mulet/reftest/b2g_desktop.py", line 57, in run_marionette_script
self.marionette.execute_script(self.test_script)
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 1205, in execute_script
filename=os.path.basename(frame[0]))
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/decorators.py", line 35, in _
return func(*args, **kwargs)
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 638, in _send_message
self._handle_error(response)
File "/home/armenzg/moz/tmp/mulet/marionette/marionette/marionette.py", line 686, in _handle_error
raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: JavascriptException: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIXPCComponents_Utils.import]
stacktrace:
execute_script @b2g_desktop.py, line 57
inline javascript, line 100
src: "Cu.import("chrome://reftest/content/reftest.jsm", reftest);"
Attachment #8488849 -
Attachment is obsolete: true
Assignee | ||
Comment 17•10 years ago
|
||
I have managed to run reftests. [1]
We now have to figure out where we should put the gaia lock screen JS code and meanwhile I will work locally on my build environment to generate reftest.jsm when building mulet.
I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the right place.
[1]
* I downloaded https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140912061053/gaia-tests.zip
* I copied reftest.jsm from it
cp b2g_tests/reftest/reftest/chrome/reftest/content/reftest.jsm mulet/reftest/reftest/chrome/reftest/content/
Reporter | ||
Comment 18•10 years ago
|
||
> I put the gaia lock screen JS code beside the b2g_desktop.py code. Let me know if that would be the
> right place.
Yes, I think that's fine.
Assignee | ||
Comment 19•10 years ago
|
||
armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name reftest.js
obj-dir-mulet/dist/bin/chrome/reftest/content/reftest.js
armenzg-thinkpad mozilla-central hg:[default!] $ find obj-dir-mulet/ -name gaia_lock_screen.js
obj-dir-mulet/_tests/reftest/gaia_lock_screen.js
Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=ae607740c4f4
Attachment #8488862 -
Attachment is obsolete: true
Attachment #8489634 -
Flags: feedback?(jgriffin)
Assignee | ||
Comment 20•10 years ago
|
||
This is the proper patch. I pushed something wrong:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=69fa463786c1
Attachment #8489634 -
Attachment is obsolete: true
Attachment #8489634 -
Flags: feedback?(jgriffin)
Comment 21•10 years ago
|
||
Comment on attachment 8490106 [details] [diff] [review]
mulet_reftests.diff
Review of attachment 8490106 [details] [diff] [review]:
-----------------------------------------------------------------
::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
> env = self.buildBrowserEnv(options, self.profile.profile)
> +
> + # FIXME
> + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
This will need to be removed for the jobs to be unhidden, seeing as this is a new job, per:
https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
Assignee | ||
Comment 22•10 years ago
|
||
The build seems to generate something that works.
You can reproduce it locally by following the steps below [1]
With regards to mozharness changes, for now it seems that I need to:
1) install marionette
2) use runreftestb2g.py.
3) append --desktop
4) append --profile
The Firefox's mozharness reftest config is not good as is.
On another note, are the mulet mochitest jobs calling the right thing?
They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]
[1]
Check out a local clone of gaia and make a gaia profile (by running 'make' in
the gaia dir)
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/jsshell-linux-x86_64.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
unzip -q firefox-35.0a1.en-US.linux-x86_64.tests.zip
tar xvfj firefox-35.0a1.en-US.linux-x86_64.tar.bz2
virtualenv temp_venv
source temp_venv/bin/activate
pip install -e marionette
python reftest/runreftestb2g.py --desktop --profile=/home/armenzg/repos/branches/gaia/profile --appname=firefox/firefox-bin reftest/tests/layout/reftests/reftest.list
[2]
python scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --reftest-suite reftest --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --jsshell-url ~/moz/tmp/mulet_latest/jsshell-linux-x86_64.zip
python', '-u', '/home/armenzg/repos/mozharness/build/tests/reftest/runreftestb2g.py', '--appname=/home/armenzg/repos/mozharness/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=/home/armenzg/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', 'tests/reftest/tests/layout/reftests/reftest.list'
[3]
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/try_ubuntu64_vm_mulet_test-mochitest-1-bm115-tests1-linux64-build149.txt.gz
10:05:45 INFO - Calling ['/builds/slave/test/build/venv/bin/python', '-u', '/builds/slave/test/build/tests/mochitest/runtests.py', '--total-chunks', '5', '--this-chunk', '1', '--appname=/builds/slave/test/build/application/firefox/firefox', '--utility-path=tests/bin', '--extra-profile-file=tests/bin/plugins', '--symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip', '--certificate-path=tests/certs', '--autorun', '--close-when-done', '--console-level=INFO', '--setpref=webgl.force-enabled=true', '--quiet', '--log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log', '--use-test-media-devices', '--chunk-by-dir=4'] with output_timeout 1000
Reporter | ||
Comment 23•10 years ago
|
||
> On another note, are the mulet mochitest jobs calling the right thing?
> They seem to be calling mochitest/runtests.py instead of mochitest/runtestsb2g.py [3]
Probably they're not. Those jobs were setup with people unfamiliar with the way we run mochitests on B2G, so what we have running is the desktop flavor. This is fine, except that those tests don't run OOP, which we probably want. So, I think we'll need to move to B2G-style mochitests for Mulet.
Assignee | ||
Comment 24•10 years ago
|
||
Jordan, I have managed to the most extent be able to use desktop_unittest.py to run mulet reftests, however, there's one step that does not fit properly inside of desktop_unittest.py.
The step requires to checkout gaia and run a "make" command.
What do you suggest doing?
I'm thinking of creating a mulet_unittest.py and have a simplified version of desktop_unittest.py that also checkout gaia.
I'm also considering a few days experiment where the script and config to deal with mulet lives in the tree and the mozharness code does a very setup and lets the script from the tree do the rest.
This would lay ground for a data point for next week's mozharness conversations.
What do you think?
Flags: needinfo?(jlund)
Reporter | ||
Comment 25•10 years ago
|
||
Can we use b2g_desktop_unittest instead of desktop_unittest?
Reporter | ||
Comment 26•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #25)
> Can we use b2g_desktop_unittest instead of desktop_unittest?
The way these tests run is much more similar to b2gdesktop than it is to Firefox.
Assignee | ||
Comment 27•10 years ago
|
||
Alexandre,
To get this running we need to checkout gaia and make a profile.
Which branch should we test against as we move through the release channels?
Does this look correct?
* m-i, m-c, fx-team, try et trunk trees:
** http://hg.mozilla.org/integration/gaia-central
* m-a
** http://hg.mozilla.org/integration/gaia-2_1
* Mozilla-B2g32-v2.0
** http://hg.mozilla.org/integration/gaia-2_0
and on...
I'm not saying that we will enable it yet on those branches but I want to make sure that we want to follow the right versions versus sticking to gaia-central across trees.
Flags: needinfo?(poirot.alex)
Reporter | ||
Comment 28•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
>
> Which branch should we test against as we move through the release channels?
> Does this look correct?
>
> * m-i, m-c, fx-team, try et trunk trees:
> ** http://hg.mozilla.org/integration/gaia-central
> * m-a
> ** http://hg.mozilla.org/integration/gaia-2_1
> * Mozilla-B2g32-v2.0
> ** http://hg.mozilla.org/integration/gaia-2_0
> and on...
>
> I'm not saying that we will enable it yet on those branches but I want to
> make sure that we want to follow the right versions versus sticking to
> gaia-central across trees.
See e.g., http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/scripts/gaia_unit.py. We have a GaiaTest mixin that automatically clones the correct branch of gaia: http://hg.mozilla.org/build/mozharness/file/0f41dc44292e/mozharness/mozilla/testing/gaia_test.py It also has a 'make_gaia' method that builds a gaia profile.
Assignee | ||
Comment 29•10 years ago
|
||
With this patch and command I can get to checkout gaia and make the profile.
python scripts/mulet_unittest.py --cfg b2g/desktop_automation_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url ~/moz/tmp/mulet_latest/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg b2g/mulet_config.py --cfg developer_config.py
More next week.
Attachment #8490106 -
Attachment is obsolete: true
Comment 30•10 years ago
|
||
> The step requires to checkout gaia and run a "make" command.
>
> What do you suggest doing?
> I'm thinking of creating a mulet_unittest.py and have a simplified version
> of desktop_unittest.py that also checkout gaia.
inheriting from GaiaTest should allow you to do that. however I'm not sure if you need all that it comes with. it looks like there is a GaiaMixin that may do exactly what you want: clone/make
marionette already avails of that directly: http://mxr.mozilla.org/build/source/mozharness/scripts/marionette.py#31
GaiaMixin: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/gaia.py#12
so you can do what marionette did and add it in to desktop_unittest.py. maybe b2g_desktop_unittest.py is more appropriate like jgriffin suggested?
>
> I'm also considering a few days experiment where the script and config to
> deal with mulet lives in the tree and the mozharness code does a very setup
> and lets the script from the tree do the rest.
> This would lay ground for a data point for next week's mozharness
> conversations.
sure. depends on how we end up wanting to implement this but it could be a cool experiment :)
does this help?
Flags: needinfo?(jlund)
Comment 31•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #27)
> Alexandre,
> To get this running we need to checkout gaia and make a profile.
>
> Which branch should we test against as we move through the release channels?
We should be doing what we are already doing for other b2g projects (b2g-desktop/device/emulator).
(It looks like you already found the code that does that...)
> Does this look correct?
Otherwise, yes at first sight it looks correct.
(Just note that m-a won't stick to 2.1 and will switch to 2.2 someday)
Flags: needinfo?(poirot.alex)
Assignee | ||
Comment 32•10 years ago
|
||
I have tried various way on how to re-use the right class, however, there are a bunch of issues with the various Gaia related classes.
I'm going to try not do any re-factoring, however, it has to be dealt in the future (bug 1071219).
Assignee | ||
Comment 33•10 years ago
|
||
This can be run like this:
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
hg clone http://hg.mozilla.org/integration/gaia-central gaia
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list
Known issues:
* The gaia checkout is never clobbered
* It does not yet checkout gaia by itself
I'm requesting for feedback so at least I can get an idea that I'm not too off from your expectations.
NOTE: I have tried subclassing many other classes with much pain and little luck. I hope this can meet the expectations.
Attachment #8492358 -
Attachment is obsolete: true
Attachment #8493994 -
Flags: feedback?(jlund)
Assignee | ||
Comment 34•10 years ago
|
||
With the previous patch we can "run_file_names" from the tree.
Assignee | ||
Comment 35•10 years ago
|
||
This patch can now checkout gaia for me. It will need to test other scripts that make use of GaiaTest and GaiaMixin.
Instructions on how to run this locally [1]
Known issues:
* The gaia checkout and the gaia profile are never clobbered
* [FIXED] It does not yet checkout gaia by itself
[1]
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tests.zip
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.tar.bz2
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-69fa463786c1/try-linux64-mulet/firefox-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --blob-upload-branch mozilla-central --download-symbols ondemand --test-url firefox-35.0a1.en-US.linux-x86_64.tests.zip --installer-url firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --cfg developer_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --gaia-repo http://hg.mozilla.org/integration/gaia-central
Attachment #8493994 -
Attachment is obsolete: true
Attachment #8493994 -
Flags: feedback?(jlund)
Attachment #8494540 -
Flags: feedback?(jlund)
Assignee | ||
Comment 36•10 years ago
|
||
Attachment #8494686 -
Flags: review?(jlund)
Comment 37•10 years ago
|
||
Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash
Review of attachment 8494686 [details] [diff] [review]:
-----------------------------------------------------------------
lgtm. suppose we will do a tbpl patch after?
Attachment #8494686 -
Flags: review?(jlund) → review+
Comment 38•10 years ago
|
||
Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff
Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------
lrgtm :D :D
one high level question is how come you did not do to B2GDesktopTest what MarionetteTest did? I know you mentioned that you were having problems but could you not have just added GaiaMixin to B2GDesktopTest, overwritten make_gaia like you did in mulet_unittest.py, and extended the other mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet edge case.
On one argument, classes are getting overloaded with logic, on the other, we have too many mixins and subclasses. Not sure I know what we should do so just polling your opinion.
Looks great :)
::: mozharness/mozilla/gaia.py
@@ +17,5 @@
> + }],
> + [["--gaia-repo"],
> + {"action": "store",
> + "dest": "gaia_repo",
> + "default": None,
I don't think the 'default' lines are needed. if not given, I think it will default to None
@@ +239,5 @@
> + # gaia_branch by default is set to default
> + self.fatal("You're trying to run this outside of buildbot, " \
> + "therefore, you need to specify --gaia-repo.")
> +
> + def pull(self, **kwargs):
looks like marionette and GaiaTest have their own pull action. In which case the preflight_pull here will be called. I guess that's OK? Maybe even a good thing
@@ +250,5 @@
> + dirs = self.query_abs_dirs()
> + dest = dirs['abs_gaia_dir']
> + repo = {}
> +
> + if self.buildbot_config is not None:
same as 'if self.buildbot_config'.
I think we normally: 'self.buildbot_config and 'properties' in self.buildbot_config:'
@@ +261,5 @@
> + else:
> + repo = {
> + 'repo_path': self.config['gaia_repo'],
> + 'revision': 'default',
> + 'branch': self.config['gaia_branch']
should we add 'gaia_branch to preflight_pull like gaia_repo?
::: mozharness/mozilla/testing/gaia_test.py
@@ +76,5 @@
> "help": "Number of this chunk",
> }
> ]] + copy.deepcopy(testing_config_options) + \
> + copy.deepcopy(blobupload_config_options) + \
> + copy.deepcopy(gaia_config_options)
good call :)
::: mozharness/mozilla/testing/testbase.py
@@ +113,5 @@
> create_parent_dir=create_parent_dir,
> error_level=error_level,
> exit_code=exit_code)
>
> + def query_value(self, key):
I suspect that as you go, this method will make it closer and closer to the internals of mozharness. :)
::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
> {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
> {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
> ]
>
> + def __init__(self, config=None, require_config_file=False):
I think we need to add GaiaMixin to this class no?
@@ +75,2 @@
> super(B2GDesktopTest, self).__init__(
> + config_options=self.config_options + copy.deepcopy(config),
if config is None, it will error trying to concat to a list
@@ +99,5 @@
> self.test_url = c.get('test_url')
> self.test_manifest = c.get('test_manifest')
>
> + suite = self.config['test_suite']
> + if suite not in self.test_suites:
we want to support tree_config['suite'] or self.config['test_suite'] right? so if it's the former on a given script run, this test will always fatal right?
That is unless we pass suite to self.config *and* self.tree_config every time. but if we are moving to tree_config, I think we should drop self.config['suite'] altogether no?
::: scripts/mulet_unittest.py
@@ +18,5 @@
> +from mozharness.base.transfer import TransferMixin
> +from mozharness.mozilla.gaia import GaiaMixin, gaia_config_options
> +from b2g_desktop_unittest import B2GDesktopTest
> +
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
since pull is defined in GaiaMixin, I think we should add GaiaMixin to its definition as I mentioned above
@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> + test_suites = ('reftest',)
> +
> + def __init__(self):
> + super(MuletUnittest, self).__init__(config=gaia_config_options)
not sure if config is the best name for this. could we add options to the name?
@@ +52,5 @@
> +
> + def preflight_run_tests(self):
> + if self.config.get("binary_path"):
> + self.binary_path = self.config["binary_path"]
> + assert self.binary_path, \
asserts will skip log placement. I think self.fatal would be better as this is something we can only test at runtime
Attachment #8494540 -
Flags: feedback?(jlund) → feedback+
Assignee | ||
Comment 39•10 years ago
|
||
Comment on attachment 8494686 [details] [diff] [review]
[checked-in] add mulet reftests on Ash
https://hg.mozilla.org/build/buildbot-configs/rev/93e40fcc9464
Attachment #8494686 -
Attachment description: add mulet reftests on Ash → [checked-in] add mulet reftests on Ash
Assignee | ||
Comment 40•10 years ago
|
||
Comment on attachment 8494540 [details] [diff] [review]
wip - mozharness_mulet.diff
Review of attachment 8494540 [details] [diff] [review]:
-----------------------------------------------------------------
> lrgtm :D :D
> one high level question is how come you did not do to B2GDesktopTest what
> MarionetteTest did? I know you mentioned that you were having problems but
> could you not have just added GaiaMixin to B2GDesktopTest, overwritten
> make_gaia like you did in mulet_unittest.py, and extended the other
> mulet_unittest.py methods (run_test and query_abs_dirs) to support the mulet
> edge case.
>
I believe my original thought was not to mix mulet jobs with b2g jobs too much, however, as you can see we have already set a dependency on b2g desktop.
I would like to request for a review for my first approach so we can land it while I give a shot again at not needing mulet_unittest.py.
> On one argument, classes are getting overloaded with logic, on the other, we
> have too many mixins and subclasses. Not sure I know what we should do so
> just polling your opinion.
In my opinion, the long term solution is moving to writing scripts from the tree.
On the mean time, we should try to have code that makes it easy to write different scripts.
For instance, the marionette script had to redo a lot of functionality since it was not inside of GaiaMixin.
I think what would help here is if we had a clear separation of what goes into a Mixin and move it there.
For instance, GaiaTest should have defined whatever actions is overwriting and put everything else into GaiaMixin.
I did this a bit in my patch but not completely (e.g. query_abs_dir, extract_xre).
I'm also convinced that we should start thinking about using the concept of super reviewers to ensure that we have consistency and increase knowledge to make better architectural designs.
It is an area that we need due to the complexity of mozharness and one that not too many want to go to (more review load).
If we had mozharness modularized (as in proposal) we would have less need for actions, subclassing and mixins which would drop the need for super reviews.
::: mozharness/mozilla/gaia.py
@@ +261,5 @@
> + else:
> + repo = {
> + 'repo_path': self.config['gaia_repo'],
> + 'revision': 'default',
> + 'branch': self.config['gaia_branch']
gaia_branch has a default value and that is why I did not query, however, I would not mind adding it to be explicit (we never know if the default would change).
::: scripts/b2g_desktop_unittest.py
@@ +75,2 @@
> super(B2GDesktopTest, self).__init__(
> + config_options=self.config_options + copy.deepcopy(config),
I fixed this in a following patch.
@@ +99,5 @@
> self.test_url = c.get('test_url')
> self.test_manifest = c.get('test_manifest')
>
> + suite = self.config['test_suite']
> + if suite not in self.test_suites:
In this case, all we're trying to do is make sure that --test-suite $suite is inside of the list of approved suites. In the case of MuletUnittests we currently only support reftests.
::: scripts/mulet_unittest.py
@@ +22,5 @@
> +class MuletUnittest(B2GDesktopTest, GaiaMixin, TransferMixin):
> + test_suites = ('reftest',)
> +
> + def __init__(self):
> + super(MuletUnittest, self).__init__(config=gaia_config_options)
Sure.
@@ +52,5 @@
> +
> + def preflight_run_tests(self):
> + if self.config.get("binary_path"):
> + self.binary_path = self.config["binary_path"]
> + assert self.binary_path, \
Fixed in next patch.
Assignee | ||
Comment 41•10 years ago
|
||
This patch works on Ash, including for b2g desktop and gaia unit jobs.
As I mentioned, I would like to land as-is and do a bit of work to refactor it properly.
Attachment #8494540 -
Attachment is obsolete: true
Attachment #8495356 -
Flags: review?(jlund)
Assignee | ||
Comment 42•10 years ago
|
||
jgriffin, what do we need to do with regards to this comment?
(In reply to Ed Morley [:edmorley] from comment #21)
> Comment on attachment 8490106 [details] [diff] [review]
> mulet_reftests.diff
>
> Review of attachment 8490106 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +61,5 @@
> > env = self.buildBrowserEnv(options, self.profile.profile)
> > +
> > + # FIXME
> > + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
>
> This will need to be removed for the jobs to be unhidden, seeing as this is
> a new job, per:
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
Flags: needinfo?(jgriffin)
Reporter | ||
Comment 43•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #42)
> jgriffin, what do we need to do with regards to this comment?
>
> (In reply to Ed Morley [:edmorley] from comment #21)
> > Comment on attachment 8490106 [details] [diff] [review]
> > mulet_reftests.diff
> >
> > Review of attachment 8490106 [details] [diff] [review]:
> > -----------------------------------------------------------------
> >
> > ::: layout/tools/reftest/b2g_desktop.py
> > @@ +61,5 @@
> > > env = self.buildBrowserEnv(options, self.profile.profile)
> > > +
> > > + # FIXME
> > > + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> >
> > This will need to be removed for the jobs to be unhidden, seeing as this is
> > a new job, per:
> > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
Good question. The issue is that Gaia makes some remote http requests on startup that we have no way to disable. AFAICT, these don't impact the tests, but who knows for sure. I'll pass this back to Ed...Ed, if you feel strongly about this, we'll need to file bugs for devs to implement ways to skip these.
Flags: needinfo?(jgriffin) → needinfo?(emorley)
Assignee | ||
Comment 44•10 years ago
|
||
I attached an old patch.
Attachment #8495356 -
Attachment is obsolete: true
Attachment #8495356 -
Flags: review?(jlund)
Attachment #8495506 -
Flags: review?(jlund)
Comment 45•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #43)
> > > > + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > > > + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
> > >
> > > This will need to be removed for the jobs to be unhidden, seeing as this is
> > > a new job, per:
> > > https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#8.
> > > 29_Must_avoid_patterns_known_to_cause_non_deterministic_failures
>
> Good question. The issue is that Gaia makes some remote http requests on
> startup that we have no way to disable. AFAICT, these don't impact the
> tests, but who knows for sure. I'll pass this back to Ed...Ed, if you feel
> strongly about this, we'll need to file bugs for devs to implement ways to
> skip these.
Remote requests on startup can still affect the reliability of the job - and worse, by having this disabled we won't detect future regressions for requests not at startup. Given that opt B2G mochitest jobs are successfully running without this disabled (http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtestsb2g.py#163) I would like to see at least the opt builds running without undefining MOZ_DISABLE_NONLOCAL_CONNECTIONS, if that's ok?
Flags: needinfo?(emorley)
Reporter | ||
Comment 46•10 years ago
|
||
Understood. In that case, can you remove those bits Armen, and file a bug for whichever request it is that causes the test to crash? I can help direct the bug to the right component if I can see a test log for it.
Comment 47•10 years ago
|
||
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff
Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------
lgtm :) if it works on ash, shweet.
But I am not sure if it is all working. My main concern is that this still uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but scripts/mulet_unittest.py uses 'options' now.
::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values
remind me, what's the plan here? are you going to rip out duplicate items from b2g configs and call this after a buildbot-config patch?
I wonder if we should be creating this file in tree and adding it to self.tree_config rather than within mozharn repo. Apologies if you have explained this already.
::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> - self.clone_gaia(dest, repo,
> - use_gaia_json=self.buildbot_config is not None)
> -
> - super(GaiaTest, self).pull(**kwargs)
what pull was this super calling? looks like we are no longer doing that after your patch.
::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
> {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
> {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
> ]
>
> + def __init__(self, config=[], require_config_file=False):
I think because of implentation of python, we have to assign config with [] outside of of the method signature or spurious things can happen: http://stackoverflow.com/questions/9887180/the-value-of-an-empty-list-in-function-parameter-example-here
also, I'm guessing this is supposed to be 'options' now after your change in mulet_unittest.py
::: scripts/mulet_unittest.py
@@ +55,5 @@
> + # of download-and-extract()
> + if self.config.get("binary_path"):
> + self.binary_path = self.config["binary_path"]
> +
> + if not self.binary_path:
should we do this instead incase we have self.binary_path already set?
if not self.binary_path:
if self.config.get("binary_path"):
self.binary_path = self.config["binary_path"]
else:
self.fatal("Use --binary-path as it is needed for _query_abs_dir().")
it might not matter. the above is the idiom we normally do with self attrs and self.config. doesn't block
Attachment #8495506 -
Flags: review?(jlund) → review+
Assignee | ||
Updated•10 years ago
|
Attachment #8494000 -
Flags: review?(jlund)
Assignee | ||
Comment 48•10 years ago
|
||
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff
Review of attachment 8495506 [details] [diff] [review]:
-----------------------------------------------------------------
> But I am not sure if it is all working. My main concern is that this still
> uses a 'config' arg in init of scripts/b2g_desktop_unittest.py but
> scripts/mulet_unittest.py uses 'options' now.
>
You're right. This was fixed on a posterior push to Ash. I have kept on working on a proper refactoring for this.
::: configs/b2g/generic_config.py
@@ +1,1 @@
> +# This is a config with generic releng related values
The plain is experiment with the idea that *all* if not most releng values could be contained on a generic config. Eventually, once we move things into the tree, I would like to make all scripts to point to such universal generic config and then specialize for the specific platform. This is just one step towards it.
I don't see this config in the tree since a lot of these values might be needed before the download-and-extract step happens.
I don't currently see the value of having this in the tree as of now, however, I'm happy to discuss it further now or later down the road.
::: mozharness/mozilla/testing/gaia_test.py
@@ -154,5 @@
> -
> - self.clone_gaia(dest, repo,
> - use_gaia_json=self.buildbot_config is not None)
> -
> - super(GaiaTest, self).pull(**kwargs)
For some reason I recall seeing the tools repo being checked out, however, I can't see "repos" being specified for marionete, b2g desktop or gaia unit.
I will review it again if I do need this.
::: scripts/b2g_desktop_unittest.py
@@ +70,5 @@
> {'substr': 'Timeout waiting for marionette on port', 'level': ERROR},
> {'regex': re.compile(r'''(Timeout|NoSuchAttribute|Javascript|NoSuchElement|XPathLookup|NoSuchWindow|StaleElement|ScriptTimeout|ElementNotVisible|NoSuchFrame|InvalidElementState|NoAlertPresent|InvalidCookieDomain|UnableToSetCookie|InvalidSelector|MoveTargetOutOfBounds)Exception'''), 'level': ERROR},
> ]
>
> + def __init__(self, config=[], require_config_file=False):
That is insane!! That is a rather absurd behaviour of python.
I was thinking of fixing, however, after some thought, we never call an __init__ method of a script more than once.
Do you think I should still fix it?
::: scripts/mulet_unittest.py
@@ +55,5 @@
> + # of download-and-extract()
> + if self.config.get("binary_path"):
> + self.binary_path = self.config["binary_path"]
> +
> + if not self.binary_path:
What do you think of this instead? A bit cleaner; I think.
def preflight_run_tests(self):
# This is if we don't run all actions since we set this inside
# of download-and-extract()
if not self.binary_path:
if self.config.get("binary_path"):
self.binary_path = self.config["binary_path"]
else:
self.fatal("Use --binary-path as it is needed for _query_abs_dir().")
Comment 49•10 years ago
|
||
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Assignee | ||
Updated•10 years ago
|
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Updated•10 years ago
|
Attachment #8494000 -
Flags: review?(jlund) → review+
Assignee | ||
Updated•10 years ago
|
Attachment #8494000 -
Attachment description: mulet_in_tree_configs.diff → [checked-in] mulet_in_tree_configs.diff
Assignee | ||
Comment 50•10 years ago
|
||
Attachment #8497039 -
Flags: review?(jlund)
Assignee | ||
Comment 51•10 years ago
|
||
Comment on attachment 8495506 [details] [diff] [review]
[checked-in] mozharness_mulet.diff
https://hg.mozilla.org/build/mozharness/rev/b59a1a06dd1e
Attachment #8495506 -
Attachment description: mozharness_mulet.diff → [checked-in] mozharness_mulet.diff
Assignee | ||
Comment 52•10 years ago
|
||
You can now run mulet through mozharness.
You can try it out with:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --blob-upload-branch ash --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ash-linux64-mulet/1412005020/firefox-35.0a1.en-US.linux-x86_64.tests.zip --cfg developer_config.py --gaia-repo ~/repos/branches/gaia-central
(In reply to Jonathan Griffin (:jgriffin) from comment #46)
> Understood. In that case, can you remove those bits Armen, and file a bug
> for whichever request it is that causes the test to crash? I can help
> direct the bug to the right component if I can see a test log for it.
I have seen this happen locally [1]
While on Ash I see this instead [2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0
[1]
http://hg.mozilla.org/mozilla-central/file/6eb7121059fb/netwerk/base/src/nsSocketTransport2.cpp#l1184
16:38:06 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizom.org (63.245.215.73) was made.
16:38:06 INFO - You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
16:38:06 INFO - ###!!! [Child][MessageChannel] Error: Channel closing: too late to send/recv, messages will be lost
16:38:06 INFO - [Child 8101] WARNING: pipe error (3): Connection reset by peer: file /builds/slave/ash-l64-mulet-0000000000000000/build/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 451
16:38:06 INFO - ###!!! [Child][MessageChannel] Error: Channel error: cannot send/recv
16:38:06 WARNING - PROCESS-CRASH | b2g_desktop.pyc | application crashed [unknown top frame]
16:38:06 INFO - Crash dump filename: /tmp/tmpc07zn1/minidumps/3b46a9ed-d7b0-b362-782e876c-53ad2a11.dmp
16:38:06 INFO - No symbols path given, can't process dump.
16:38:06 INFO - Traceback (most recent call last):
16:38:06 INFO - File "runreftestb2g.py", line 611, in <module>
16:38:06 INFO - sys.exit(main())
16:38:06 INFO - File "runreftestb2g.py", line 606, in main
16:38:06 INFO - return run_desktop_reftests(parser, options, args)
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
16:38:06 INFO - sys.exit(reftest.run_tests(args[0], options))
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
16:38:06 INFO - self.run_marionette_script()
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/tests/reftest/b2g_desktop.py", line 44, in run_marionette_script
16:38:06 INFO - self.marionette.start_session()
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 818, in start_session
16:38:06 INFO - self.session = self._send_message('newSession', 'value')
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
16:38:06 INFO - return func(*args, **kwargs)
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 615, in _send_message
16:38:06 INFO - response = self.client.send(message)
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 85, in send
16:38:06 INFO - self.connect()
16:38:06 INFO - File "/home/armenzg/repos/mozharness/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 67, in connect
16:38:06 INFO - self.sock.connect((self.addr, self.port))
16:38:06 INFO - File "/usr/lib/python2.7/socket.py", line 224, in meth
16:38:06 INFO - return getattr(self._sock,name)(*args)
16:38:06 INFO - socket.error: [Errno 111] Connection refused
[2]
https://tbpl.mozilla.org/php/getParsedLog.php?id=49121431&tree=Ash&full=1#error0
09:44:19 INFO - Traceback (most recent call last):
09:44:19 INFO - File "runreftestb2g.py", line 611, in <module>
09:44:19 INFO - sys.exit(main())
09:44:19 INFO - File "runreftestb2g.py", line 606, in main
09:44:19 INFO - return run_desktop_reftests(parser, options, args)
09:44:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 187, in run_desktop_reftests
09:44:19 INFO - sys.exit(reftest.run_tests(args[0], options))
09:44:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 95, in run_tests
09:44:19 INFO - self.run_marionette_script()
09:44:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 45, in run_marionette_script
09:44:19 INFO - self._unlockScreen()
09:44:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 39, in _unlockScreen
09:44:19 INFO - self.marionette.execute_async_script('GaiaLockScreen.unlock()')
09:44:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1269, in execute_async_script
09:44:19 INFO - filename=os.path.basename(frame[0]))
09:44:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 35, in _
09:44:19 INFO - return func(*args, **kwargs)
09:44:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
09:44:19 INFO - self._handle_error(response)
09:44:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
09:44:19 ERROR - raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
09:44:19 ERROR - marionette.errors.JavascriptException: JavascriptException: TypeError: this.states.instance is null
09:44:19 INFO - stacktrace:
09:44:19 INFO - @app://system.gaiamobile.org/js/lockscreen_window_manager.js, line 245
09:44:19 ERROR - Return code: 1
Flags: needinfo?(jgriffin)
Assignee | ||
Comment 53•10 years ago
|
||
The code used for the previous comment is this:
https://hg.mozilla.org/projects/ash/rev/6a0c55df4061
Which is the same as attachment 8490106 [details] [diff] [review] but by removing this:
::: layout/tools/reftest/b2g_desktop.py
@@ +61,5 @@
> env = self.buildBrowserEnv(options, self.profile.profile)
> +
> + # FIXME
> + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
Assignee | ||
Comment 54•10 years ago
|
||
jlund, I can't seem to find a way to mark a test job to not be run by default on try.
I can see "try_by_default" for build jobs but not for build jobs.
I could also hide mulet reftests on try until they're fixed.
Flags: needinfo?(jlund)
Reporter | ||
Comment 55•10 years ago
|
||
I'll file a bug for #1. #2 looks like a timing issue...the slave is probably slower than your machine, and possibly the lock screen hasn't appeared by the time we are trying to unlock it. Does the problem on ash disappear if you sleep(15) before unlocking the lockscreen? If so, we probably need to add some code to wait for the lock screen to appear before we attempt to unlock it.
Flags: needinfo?(jgriffin)
Assignee | ||
Comment 56•10 years ago
|
||
With this patch we can now hit my local issue:
https://tbpl.mozilla.org/php/getParsedLog.php?id=49211733&tree=Ash&full=1#error0
I guess we just have to wait now for bug 1053163.
Meanwhile let's get the try support so developers can fix it on try (attachment 8497039 [details] [diff] [review]).
Comment 57•10 years ago
|
||
Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff
Review of attachment 8497039 [details] [diff] [review]:
-----------------------------------------------------------------
lgtm but can we make this a trychooser option? looking at this patch, I assume this will run by default if we -all on try. Which might upset sheriffs as, correct me if I am wrond, we shouldn't have any tests that are perma fail running by default on try.
Assignee | ||
Comment 58•10 years ago
|
||
I'm aiming at hiding it since I don't know of a way to not trigger it by default.
Comment 59•10 years ago
|
||
Merged to production, and deployed.
Comment 60•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #58)
> I'm aiming at hiding it since I don't know of a way to not trigger it by
> default.
if you can not find a way to trigger it only via trychooser, I can only see an example for builds[1], hiding wfm. Sorry to stall you on this. Just trying to avoid adding unneeded jobs
[1] https://bug691177.bugzilla.mozilla.org/attachment.cgi?id=695029
Flags: needinfo?(jlund)
Comment 61•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #54)
> jlund, I can't seem to find a way to mark a test job to not be run by
> default on try.
> I can see "try_by_default" for build jobs but not for build jobs.
>
> I could also hide mulet reftests on try until they're fixed.
well that's embarrassing. just seeing this comment now. looks like I've been one step behind all along.
Updated•10 years ago
|
Attachment #8497039 -
Flags: review?(jlund) → review+
Assignee | ||
Comment 62•10 years ago
|
||
Comment on attachment 8497039 [details] [diff] [review]
[checked-in] bconfigs_enable_mulet_on_try.diff
I did not find a way to disable a specific builder.
I only found a way to disable a whole platform by adding "try_by_default": False (e.g. u'linux64-mulet': ['Ubuntu VM 12.04 x64 Mulet try-nondefault'],).
I'm landing as-is and I hope to hide it on time once it shows up on tbpl.
https://hg.mozilla.org/build/buildbot-configs/rev/3bfd982c2211
Attachment #8497039 -
Attachment description: bconfigs_enable_mulet_on_try.diff → [checked-in] bconfigs_enable_mulet_on_try.diff
Comment 63•10 years ago
|
||
Merged to production, and deployed.
Assignee | ||
Comment 64•10 years ago
|
||
Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff
I think we should land this while it does not get bitrotten.
Attachment #8497650 -
Flags: review?(jgriffin)
Assignee | ||
Comment 65•10 years ago
|
||
This will allow developers to know the current state of mulet reftests without first having to push anything to try.
Attachment #8500612 -
Flags: review?(jlund)
Reporter | ||
Comment 66•10 years ago
|
||
Comment on attachment 8497650 [details] [diff] [review]
mulet_reftests.diff
Review of attachment 8497650 [details] [diff] [review]:
-----------------------------------------------------------------
::: layout/tools/reftest/b2g_desktop.py
@@ +117,5 @@
> # Turn off the locale picker screen
> prefs["browser.firstrun.show.localepicker"] = False
> + # FIXME: these cause Gaia not to launch
> + #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> + #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"
Have we done a try run to make sure this doesn't break reftest on b2gdesktop? I'm afraid the above lines might do that.
Assignee | ||
Updated•10 years ago
|
Attachment #8497650 -
Flags: review?(jgriffin)
Assignee | ||
Comment 67•10 years ago
|
||
(In reply to Jonathan Griffin (:jgriffin) from comment #66)
> Comment on attachment 8497650 [details] [diff] [review]
> mulet_reftests.diff
>
> Review of attachment 8497650 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +117,5 @@
> > # Turn off the locale picker screen
> > prefs["browser.firstrun.show.localepicker"] = False
> > + # FIXME: these cause Gaia not to launch
> > + #prefs["b2g.system_startup_url"] = "app://test-container.gaiamobile.org/index.html"
> > + #prefs["b2g.system_manifest_url"] = "app://test-container.gaiamobile.org/manifest.webapp"
>
> Have we done a try run to make sure this doesn't break reftest on
> b2gdesktop? I'm afraid the above lines might do that.
My bad. We will know tomorrow:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2
Assignee | ||
Comment 68•10 years ago
|
||
It does affect the b2g desktop jobs.
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=3e3714577db2
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2253625&repo=try
15:52:03 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to inapp-pay-test.paas.allizmozcrash INFO | Downloading symbols from: https://ftp-ssl.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-3e3714577db2/try-linux64_gecko/b2g-35.0a1.en-US.linux-x86_64.crashreporter-symbols.zip
jgriffin, what do you suggest I try next? Or leave this until bug 1053163 is fixed?
I've considered adding a --mulet parameter to the harness and use it for mulet jobs, hence not affecting the b2g desktop jobs.
Flags: needinfo?(jgriffin)
Reporter | ||
Comment 69•10 years ago
|
||
Since this is just on cedar for now (or try) I think we should allow network connections for the purpose of greening up the tests themselves. After bug 1053163 is addressed, we can look at disabling network connections again.
Flags: needinfo?(jgriffin)
Comment 70•10 years ago
|
||
Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar
Review of attachment 8500612 [details] [diff] [review]:
-----------------------------------------------------------------
::: mozilla-tests/b2g_config.py
@@ +2242,5 @@
> exclude_suites(('linux32_gecko', 'ubuntu32_vm-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
> exclude_suites(('macosx64_gecko', 'mountainlion-b2gdt'), 'cedar', ('gaia-ui-test',), True, True)
> exclude_suites(('emulator', 'ubuntu64_vm-b2g-lg-emulator'), 'cedar', ('gaia-ui-test',), True, True)
>
> # Enable mulet reftests on Ash and Try
# and cedar
Attachment #8500612 -
Flags: review?(jlund) → review+
Comment 71•10 years ago
|
||
Committed code from this bug has been rolled out to production.
Assignee | ||
Comment 72•10 years ago
|
||
I want to see if I can create a patch that will not turn b2g desktop reftest jobs red for Cedar (like attachment 8497650 [details] [diff] [review] does):
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=dd38ffbef696
I added back allowing external connections.
Assignee | ||
Comment 73•10 years ago
|
||
I'm back to this. I'm adding a --mulet option to runreftestb2g.py/b2g_desktop.py to allow us to have the necessary code to run mulet reftests without affecting the b2g desktop jobs.
Pushing this change to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=24fee2a32d3c
Also pushed without permitting external connections to see where we are:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=109394c6b860
STR:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py \
--cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list \
--blob-upload-branch ash --download-symbols ondemand \
--installer-url TBD \
--test-url TBD \
--cfg developer_config.py --gaia-repo /path/to/gaia/centra/checkout
NOTE: Check-out gaia-central if you don't want the script to be checking out every time.
Assignee | ||
Comment 74•10 years ago
|
||
Updated patch.
Latest run on try looks good:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=07296ac81420 (pending desktop jobs)
There's some clean up of imports that are not used.
Attachment #8497650 -
Attachment is obsolete: true
Attachment #8501770 -
Attachment is obsolete: true
Attachment #8511304 -
Attachment is obsolete: true
Attachment #8512212 -
Flags: review?(jgriffin)
Comment 75•10 years ago
|
||
Comment on attachment 8512212 [details] [diff] [review]
[m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep
Review of attachment 8512212 [details] [diff] [review]:
-----------------------------------------------------------------
::: layout/tools/reftest/b2g_desktop.py
@@ +67,5 @@
> +
> + if self.build_type == "mulet":
> + # FIXME
> + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
Is this still needed now that bug 1053163's PR landed?
If it is needed, could we make it mulet+debug? (Presuming it's green on opt like other B2G)
Assignee | ||
Comment 76•10 years ago
|
||
(In reply to Ed Morley [:edmorley] from comment #75)
> Comment on attachment 8512212 [details] [diff] [review]
> [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external
> connections, 3) do not add gaia prefs and 4) add sleep
>
> Review of attachment 8512212 [details] [diff] [review]:
> -----------------------------------------------------------------
>
> ::: layout/tools/reftest/b2g_desktop.py
> @@ +67,5 @@
> > +
> > + if self.build_type == "mulet":
> > + # FIXME
> > + if "MOZ_DISABLE_NONLOCAL_CONNECTIONS" in env:
> > + del env["MOZ_DISABLE_NONLOCAL_CONNECTIONS"]
>
> Is this still needed now that bug 1053163's PR landed?
> If it is needed, could we make it mulet+debug? (Presuming it's green on opt
> like other B2G)
I have pushed separately to determine that:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5fe267a972e5
Assignee | ||
Comment 77•10 years ago
|
||
All jobs are green except for the mulet reftest ones:
* Log: http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-5fe267a972e5/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm117-tests1-linux64-build40.txt.gz
* Changes: https://hg.mozilla.org/try/rev/07296ac81420
I will run it locally and see what is going on.
16:59:19 INFO - [3086] WARNING: Cannot parse integer value '800x1000' for --screen: 'glib warning', file /builds/slave/try-l64-mulet-0000000000000000/build/toolkit/xre/nsSigHandlers.cpp, line 129
16:59:19 INFO - (firefox:3086): Gtk-WARNING **: Cannot parse integer value '800x1000' for --screen
16:59:19 INFO - Xlib: extension "RANDR" missing on display ":0".
16:59:19 INFO - 1414454351593 addons.xpi WARN Error loading bootstrap.js for reftest@mozilla.org: Error opening input stream (invalid filename?)
16:59:19 INFO - 1414454351594 addons.xpi WARN Add-on reftest@mozilla.org is missing bootstrap method install
16:59:19 INFO - 1414454351624 addons.xpi WARN Disabling foreign installed add-on ubufox@ubuntu.com in app-system-share
16:59:19 INFO - 1414454351631 addons.xpi WARN Add-on reftest@mozilla.org is missing bootstrap method startup
...
16:59:19 INFO - 1414454359735 Marionette INFO sendToClient: {"from":"0","error":{"message":"TypeError: doc is undefined","status":17,"stacktrace":"execute_script @b2g_desktop.py, line 57\ninline javascript, line 280\nsrc: \"undefined\""}}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}, {ae7ae623-0f8f-4b41-85a5-df8df50a4da7}
16:59:19 INFO - Traceback (most recent call last):
16:59:19 INFO - File "runreftestb2g.py", line 633, in <module>
16:59:19 INFO - sys.exit(main())
16:59:19 INFO - File "runreftestb2g.py", line 628, in main
16:59:19 INFO - return run_desktop_reftests(parser, options, args)
16:59:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 208, in run_desktop_reftests
16:59:19 INFO - sys.exit(reftest.run_tests(args[0], options))
16:59:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 105, in run_tests
16:59:19 INFO - self.run_marionette_script()
16:59:19 INFO - File "/builds/slave/test/build/tests/reftest/b2g_desktop.py", line 57, in run_marionette_script
16:59:19 INFO - self.marionette.execute_script(self.test_script)
16:59:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 1221, in execute_script
16:59:19 INFO - filename=os.path.basename(frame[0]))
16:59:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/decorators.py", line 36, in _
16:59:19 INFO - return func(*args, **kwargs)
16:59:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 638, in _send_message
16:59:19 INFO - self._handle_error(response)
16:59:19 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 686, in _handle_error
16:59:19 ERROR - raise errors.JavascriptException(message=message, status=status, stacktrace=stacktrace)
16:59:19 ERROR - marionette.errors.JavascriptException: JavascriptException: TypeError: doc is undefined
16:59:19 INFO - stacktrace:
16:59:19 INFO - execute_script @b2g_desktop.py, line 57
16:59:19 INFO - inline javascript, line 280
16:59:19 INFO - src: "undefined"
Assignee | ||
Comment 78•10 years ago
|
||
I spotted the issue. I will put up a new patch.
Assignee | ||
Comment 79•10 years ago
|
||
I had forgotten to adjust the in-tree mozharness config to switch from --desktop to --mulet.
The new patch being tested:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c
Attachment #8512212 -
Attachment is obsolete: true
Attachment #8512212 -
Flags: review?(jgriffin)
Assignee | ||
Comment 80•10 years ago
|
||
I attached the wrong patch. This is the right one.
We now can run lots of reftests, however, we time out after 2 hours.
I think we have these options:
- We probably have to break the reftests into various chunks
- Determine why we run so slowly
- Run on faster VMs (not sure if we can)
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=caf13ef7ff7c&showall=1
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2757778&repo=try
Attachment #8512740 -
Attachment is obsolete: true
Attachment #8512949 -
Flags: review?(jgriffin)
Reporter | ||
Comment 81•10 years ago
|
||
Comment on attachment 8512949 [details] [diff] [review]
m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) add sleep
Review of attachment 8512949 [details] [diff] [review]:
-----------------------------------------------------------------
::: layout/tools/reftest/b2g_desktop.py
@@ +45,5 @@
> self.marionette.start_session()
> + if self.build_type == "mulet":
> + # XXX: bug 1043699 - unlocking takes too long on EC2
> + import time
> + time.sleep(15)
We already know how to get around this problem, see http://dxr.mozilla.org/mozilla-central/source/testing/web-platform/harness/wptrunner/browsers/b2g.py#236
Attachment #8512949 -
Flags: review?(jgriffin) → review+
Assignee | ||
Comment 82•10 years ago
|
||
Pushed to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=787ab3d5bb8e
I will append a patch for buildbot-configs to chunkify.
I added this to the patch to test chunking:
--- a/testing/config/mozharness/linux_mulet_config.py
+++ b/testing/config/mozharness/linux_mulet_config.py
@@ -1,13 +1,11 @@
# This is used by mozharness' mulet_unittest.py
config = {
# testsuite options
"reftest_options": [
"--mulet", "--profile=%(gaia_profile)s",
- "--appname=%(application)s", "%(test_manifest)s",
- "--this-chunk", "1",
- "--total-chunk", "6"
+ "--appname=%(application)s", "%(test_manifest)s"
],
"run_file_names": {
"reftest": "runreftestb2g.py",
},
}
Attachment #8512949 -
Attachment is obsolete: true
Assignee | ||
Comment 83•10 years ago
|
||
Attachment #8514470 -
Flags: review?(jlund)
Assignee | ||
Comment 84•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
I want to land this once I get a review for the recent changes I made.
For 6 chunks we get a 33 minute run for the push I made.
I will update my b-c configs to have 6 chunks instead of 4.
Chunk 1/6:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try
Attachment #8514467 -
Flags: review?(jgriffin)
Assignee | ||
Comment 85•10 years ago
|
||
Attachment #8514959 -
Flags: review?(jlund)
Reporter | ||
Comment 86•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
Review of attachment 8514467 [details] [diff] [review]:
-----------------------------------------------------------------
Lgtm, thanks!
Attachment #8514467 -
Flags: review?(jgriffin) → review+
Comment 87•10 years ago
|
||
Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests
Review of attachment 8514959 [details] [diff] [review]:
-----------------------------------------------------------------
lgtm :)
I'm assuming we must wait for https://bug1043699.bugzilla.mozilla.org/attachment.cgi?id=8514467 to land?
do we need a tbpl patch to recognize the chunking?
Attachment #8514959 -
Flags: review?(jlund) → review+
Comment 88•10 years ago
|
||
Comment on attachment 8514470 [details] [diff] [review]
[b-c] add 4 chunks for mulet reftests
Review of attachment 8514470 [details] [diff] [review]:
-----------------------------------------------------------------
I'm assuming this is obsolete now based of newer patch.
If you going to iterate by landing 4 chunks and then 6 chunks later and you need a review, feel free to r+ this.
Attachment #8514470 -
Flags: review?(jlund)
Assignee | ||
Updated•10 years ago
|
Attachment #8514959 -
Attachment description: [b-c] add 4 chunks for mulet reftests → [b-c] add 6 chunks for mulet reftests
Assignee | ||
Updated•10 years ago
|
Attachment #8514470 -
Attachment is obsolete: true
Assignee | ||
Comment 89•10 years ago
|
||
Comment on attachment 8500612 [details] [diff] [review]
[checked-in] [buildbot-configs] enable mulet reftests on Cedar
Checked-in a while ago.
Attachment #8500612 -
Attachment description: [buildbot-configs] enable mulet reftests on Cedar → [checked-in] [buildbot-configs] enable mulet reftests on Cedar
Assignee | ||
Comment 90•10 years ago
|
||
Comment on attachment 8514959 [details] [diff] [review]
[checked-in] [b-c] add 6 chunks for mulet reftests
Waiting for a reconfig.
http://hg.mozilla.org/build/buildbot-configs/rev/f6d6c9302c55
Attachment #8514959 -
Attachment description: [b-c] add 6 chunks for mulet reftests → [checked-in] [b-c] add 6 chunks for mulet reftests
Assignee | ||
Comment 91•10 years ago
|
||
We don't need for the mulet change to land first to run the reconfig.
However, I will be landing before EOD assuming I see no new surprises in my last try test:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=5258a11ecc3c
Assignee | ||
Comment 92•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
https://hg.mozilla.org/integration/mozilla-inbound/rev/6921bd616ff1
Attachment #8514467 -
Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen
Comment 93•10 years ago
|
||
Assignee | ||
Comment 94•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
I landed the wrong patch.
I will land the right one tomorrow after I get the right thing pushed to try.
Attachment #8514467 -
Attachment description: [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen
Comment 95•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
(This patch no longer allows external connections, so removing the mention of that)
Attachment #8514467 -
Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag 2) allow external connections, 3) do not add gaia prefs and 4) wait for screen → [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
Updated•10 years ago
|
Keywords: leave-open
Whiteboard: [leave open]
Assignee | ||
Comment 96•10 years ago
|
||
Thanks for updating the comment for the patch.
Pushing the right patch:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=e55985ad7e90
Assignee | ||
Comment 98•10 years ago
|
||
Comment on attachment 8514467 [details] [diff] [review]
[checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
https://hg.mozilla.org/integration/mozilla-inbound/rev/dfd2eb97d840
Attachment #8514467 -
Attachment description: [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen → [checked-in] [m-c] patch to run mulet reftest: 1) add --mulet flag, 2) do not add gaia prefs and 3) wait for screen
Assignee | ||
Comment 100•10 years ago
|
||
Unfortunately something has changed since comment 84 (2014-10-30) [1][2]
Who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]
I can reproduce the issue locally. The Mulet starts up, however, it aborts soon after. [4]
Allowing non-local connections makes it work again (even though this is not what we want to do).
I've pushed it to try so we can know where we stand once we fix the checking for updates.
https://tbpl.mozilla.org/?tree=Try&rev=be3fb02d83bd&show_all=1
[1]
Results from comment 84:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=2833982&repo=try
13:55:45 INFO - TinderboxPrint: reftest<br/>1733/<em class="testfail">120</em>/78
Log:
http://ftp.mozilla.org/pub/mozilla.org/b2g/try-builds/armenzg@mozilla.com-787ab3d5bb8e/try-linux64-mulet/try_ubuntu64_vm-mulet_test-reftest-bm113-tests1-linux64-build55.txt.gz
[2] Chunkified results:
https://tbpl.mozilla.org/?tree=Try&rev=a89990bc0b83&show_all=1
[3]
09:18:10 INFO - UpdatePrompt: Forcing update check
09:18:10 INFO - *** AUS:SVC Checker: checkForUpdates, force: true
09:18:10 INFO - *** AUS:SVC getLocale - getting locale from file: resource://gre/update.locale, locale: en-US
09:18:10 INFO - *** AUS:SVC Checker:getUpdateURL - update URL: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10 INFO - *** AUS:SVC gCanCheckForUpdates - able to check for updates
09:18:10 INFO - *** AUS:SVC recordInHealthReport - updateCheckStart - 0
09:18:10 INFO - *** AUS:SVC Checker:checkForUpdates - sending request to: https://aus4.mozilla.org/update/3/Firefox/36.0a1/20141104131517/%PRODUCT_DEVICE%/en-US/default/Linux%203.13.0-39-generic%20(GTK%202.24.23)/default/default/update.xml?force=1
09:18:10 INFO - FATAL ERROR: Non-local network connections are disabled and a connection attempt to aus4.mozilla.org (63.245.217.219) was made.
09:18:10 INFO - You should only access hostnames available via the test networking proxy (if running mochitests) or from a test-specific httpd.js server (if running xpcshell tests). Browser services should be disabled or redirected to a local server.
[4]
NOTE: If you already have a gaia repo you can point to it like I do
* Installers from 2014-11-05:
python scripts/mulet_unittest.py --cfg b2g/generic_config.py --cfg b2g/mulet_config.py --test-suite reftest --test-manifest tests/layout/reftests/reftest.list --blob-upload-branch try --download-symbols ondemand --installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tar.bz2 --test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-a89990bc0b83/try-linux64-mulet/firefox-36.0a1.en-US.linux-x86_64.tests.zip --gaia-repo ~/repos/branches/gaia-central --cfg developer_config.py --total-chunks 6 --this-chunk 1
Assignee | ||
Comment 101•10 years ago
|
||
ochameau: who knows how we can disable the checking for updates? It is breaking the rule of hitting the network. [3]
Side note, we need to watch for this change that *could* be affecting mulet reftests (probably not related):
https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19
Flags: needinfo?(poirot.alex)
Reporter | ||
Comment 102•10 years ago
|
||
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #101)
> ochameau: who knows how we can disable the checking for updates? It is
> breaking the rule of hitting the network. [3]
>
> Side note, we need to watch for this change that *could* be affecting mulet
> reftests (probably not related):
> https://bugzilla.mozilla.org/show_bug.cgi?id=1093691#c19
Can you help identify someone to help with this, gwagner?
Flags: needinfo?(anygregor)
Assignee | ||
Comment 103•10 years ago
|
||
I have filed two bugs (bug 1043699 and bug 1094369) which (IIUC) need dev attention.
I've also pushed a try push with 25 chunks instead of 6 since we're timing out after two hours.
I wonder what changed since in comment 84 we were having 30 minutes run.
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1
I think we have to wait for the dep bugs to be fixed first before doing the estimate but it might be good to have a data point for reference.
In one sense I think this bug is done wrt to making the harness work, however, I will wait to hear from jgriffing/ochameau as to if more help is needed from me.
Comment 104•10 years ago
|
||
Alex, can you help out here?
Flags: needinfo?(anygregor) → needinfo?(lissyx+mozillians)
Assignee | ||
Comment 105•10 years ago
|
||
I have landed a change on Cedar to allow seeing the jobs run by disabling the network block [1]
Once we have a fix for bug 1094362 we can back the change out on Cedar.
Once we fix the reftests locally (bug 1094369) please let me know so we can determine how many chunks we need (as of _now_ I think we will need 15 or more) [2].
[1] http://hg.mozilla.org/projects/cedar/rev/bfe35dbdef31
https://tbpl.mozilla.org/?tree=Cedar&jobname=Mulet&show_all=1
[2]
https://tbpl.mozilla.org/?tree=Try&rev=1c38b6313fa3&show_all=1
Comment 106•10 years ago
|
||
Armen, is it possible you are being hit by the same thing that made Mnw perma fail ?
Patch for bug 1092941 should help if it's the case.
Or I misread why I'm being asked for help here. If that's the case, please forgive and tell me what's wrong and how I can try to help.
Flags: needinfo?(lissyx+mozillians) → needinfo?(armenzg)
Assignee | ||
Comment 107•10 years ago
|
||
(In reply to Alexandre LISSY :gerard-majax from comment #106)
> Armen, is it possible you are being hit by the same thing that made Mnw
> perma fail ?
> Patch for bug 1092941 should help if it's the case.
>
> Or I misread why I'm being asked for help here. If that's the case, please
> forgive and tell me what's wrong and how I can try to help.
After chatting on IRC, it seems that for Mulet mochitest we disabled the network block [1] (bug 1066044).
We should undo that.
We first need to disable the updates, we can then look at which failures we have.
[1] https://hg.mozilla.org/mozilla-central/rev/3538768a23f4#l7.1
Flags: needinfo?(armenzg)
Comment 108•10 years ago
|
||
It's really unclear why mulet ends up having updates.
We try really hard to disable it in multiple places, and especially, here, for reftest:
http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-preferences.js#17
Aren't we using these prefs when running reftests on mulet??
Flags: needinfo?(poirot.alex)
Assignee | ||
Comment 109•10 years ago
|
||
The prefs are being ignored since we force the update.
Let's move the check for update issue to bug 1094362.
Assignee | ||
Comment 110•10 years ago
|
||
The harness work is done in here.
To make the tests go green please follow up in bug 1094362 and bug 1094369.
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Comment 111•7 years ago
|
||
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in
before you can comment on or make changes to this bug.
Description
•