Filed by: archaeopteryx [at] coole-files.de https://treeherder.mozilla.org/logviewer.html#?job_id=6536135&repo=autoland https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-linux-debug/1478919620/autoland_ubuntu32_vm-debug_test-mochitest-devtools-chrome-5-bm06-tests1-linux32-build52.txt.gz

this is now the #3 intermittent on orangefactor, :gbrown, can you run your magic statistics on this file to see what patterns we have?

might be related to bug 1273879 (devtools/client/framework/test/browser_toolbox_toggle.js), both spikes around Dec 30th and both are in the top 10 intermittents for timeouts. In fact, looking a few more bugs down the list, I see bug 1285594 (devtools/client/webconsole/test/browser_webconsole_split_persist.js) following a similar pattern. maybe 3 bugs for the price of 1

I should mention these are linux32 debug failures, it doesn't seem to be any other platform.

possibly a regression from bug 1245921?

Test durations for devtools/client/framework/test/browser_toolbox_hosts.js on mozilla-central,mozilla-inbound,autoland between Dec 29 and Jan 4: linux32/debug: 41.43 s (29.64 s - 52.64 s over 32 runs) linux32/opt-e10s: 14.49 s (12.77 s - 16.34 s over 22 runs) linux32/opt: 12.84 s (9.33 s - 15.44 s over 19 runs) linux32/pgo-e10s: 11.90 s (10.10 s - 13.31 s over 16 runs) linux32/pgo: 10.88 s (9.15 s - 12.59 s over 12 runs) linux64/asan-chunked: 25.99 s (19.11 s - 29.81 s over 63 runs) linux64/asan-e10s: 26.63 s (23.37 s - 29.81 s over 35 runs) linux64/debug-chunked: 23.12 s (19.66 s - 27.44 s over 33 runs) linux64/debug-e10s: 25.89 s (25.89 s - 25.89 s over 1 runs) linux64/opt-chunked: 6.18 s (4.82 s - 8.52 s over 53 runs) linux64/opt-e10s: 8.78 s (4.82 s - 15.73 s over 42 runs) linux64/opt: 12.13 s (10.73 s - 13.48 s over 13 runs) linux64/pgo-chunked: 5.02 s (4.29 s - 6.76 s over 41 runs) linux64/pgo-e10s: 7.89 s (4.49 s - 13.88 s over 39 runs) linux64/pgo: 9.90 s (8.30 s - 11.44 s over 15 runs) macosx64/debug-e10s: 22.00 s (17.16 s - 25.48 s over 20 runs) macosx64/debug: 23.47 s (18.55 s - 27.93 s over 25 runs) macosx64/opt-e10s: 5.13 s (4.35 s - 6.36 s over 23 runs) macosx64/opt: 5.26 s (4.44 s - 5.98 s over 23 runs) win32/debug-e10s: 29.44 s (24.02 s - 33.28 s over 23 runs) win32/debug: 32.09 s (28.08 s - 36.52 s over 24 runs) win32/opt-e10s: 4.79 s (4.26 s - 5.35 s over 19 runs) win32/opt: 4.87 s (4.40 s - 5.42 s over 23 runs) win32/pgo-e10s: 3.84 s (3.32 s - 4.21 s over 18 runs) win32/pgo: 3.93 s (3.50 s - 4.28 s over 16 runs) win64/debug-e10s: 28.49 s (23.60 s - 32.67 s over 18 runs) win64/debug: 33.01 s (28.25 s - 36.47 s over 28 runs) win64/opt-e10s: 4.31 s (3.87 s - 4.79 s over 24 runs) win64/opt: 4.53 s (4.06 s - 4.99 s over 22 runs) win64/pgo-e10s: 3.57 s (3.21 s - 4.01 s over 15 runs) win64/pgo: 3.88 s (3.42 s - 4.14 s over 16 runs)

odd that linux32 debug is such an outlier, 8 seconds longer than windows debug and 15 seconds longer than linux64 debug. I cannot see anything in the logs which indicates a longer runtime. this runs in dt2 on linux32-debug and dt7 on linux64-debug.

IME for dt tests, linux32-debug is by far the slowest platform and sometimes runs into timeouts even just opening the toolbox depending on the tool. Entire subfolders are skipped in this environment for that reason (debugger, perf). I've considered skipping all dt tests since I've only ever seen timeout-related issues but didn't want to since we then wouldn't have any linux debug coverage. Maybe now that Bug 1242986 has landed and we are running in linux 64 debug we could actually do that.

I guess that would be an interesting question if we should not run devtools on linux32 (or specifically linux32-debug). Possibly using telemetry to determine if our users are running on 32 bit linux might help? On linux64 we run opt/debug/pgo devtools in e10s and non-e10s, that is not the case on linux32, we only run on non-e10s for linux32 debug.

If it helps, here's where I tracked the spike in dt2 failures to: https://treeherder.mozilla.org/#/jobs?repo=autoland&bugfiler&noautoclassify&group_state=expanded&fromchange=9060d5b0d39a05c586f5a5818cdbe1ca63c8b65d&filter-searchStr=4bed075d3e8c33a4780da81fc2508e53e6b932e0&tochange=318b157a1c07e9fa9a35e5a9714dcbbed8d26681&selectedJob=65487253

(In reply to Joel Maher ( :jmaher) from comment #13) > I guess that would be an interesting question if we should not run devtools > on linux32 (or specifically linux32-debug). Possibly using telemetry to > determine if our users are running on 32 bit linux might help? On linux64 > we run opt/debug/pgo devtools in e10s and non-e10s, that is not the case on > linux32, we only run on non-e10s for linux32 debug. This has been discussed on the list, and sounds like we will be disabling linux32-debug tests for dt based on our usage numbers https://groups.google.com/forum/#!topic/mozilla.dev.developer-tools/xxsjFdsjpbo. I guess in the meantime we could disable this test for linux32 debug if we want to get this intermittent off the radar.

(In reply to Wes Kocher (:KWierso) from comment #14) > If it helps, here's where I tracked the spike in dt2 failures to: Thanks Wes. It looks like bug 1245921 had a significant effect on performance for some tests such as this. There is discussion in that bug about its performance impact (though not on these tests). For browser_toolbox_hosts.js, the average duration for runs in the week before bug 1245921 landed was about 37 seconds on linux/debug; afterwards, average duration was 43 seconds. There were increases in duration on all platforms, but linux/debug was the most significant. I see a very similar effect on browser_toolbox_toggle.js (bug 1273879).

(In reply to Brian Grinstead [:bgrins] from comment #16) > This has been discussed on the list, and sounds like we will be disabling > linux32-debug tests for dt based on our usage numbers > https://groups.google.com/forum/#!topic/mozilla.dev.developer-tools/ > xxsjFdsjpbo. I guess in the meantime we could disable this test for linux32 > debug if we want to get this intermittent off the radar. Disabling dt tests on linux32-debug seems like a good idea - could that be done right away?

(In reply to Geoff Brown [:gbrown] from comment #18) > (In reply to Brian Grinstead [:bgrins] from comment #16) > > This has been discussed on the list, and sounds like we will be disabling > > linux32-debug tests for dt based on our usage numbers > > https://groups.google.com/forum/#!topic/mozilla.dev.developer-tools/ > > xxsjFdsjpbo. I guess in the meantime we could disable this test for linux32 > > debug if we want to get this intermittent off the radar. > > Disabling dt tests on linux32-debug seems like a good idea - could that be > done right away? I believe jryans was going to file a bug for it

I am working on a patch for that in the land of buildbot-configs- I will link to a bug from jryans or file my own when I get the patch finshed/tested.

(In reply to Joel Maher ( :jmaher) from comment #20) > I am working on a patch for that in the land of buildbot-configs- I will > link to a bug from jryans or file my own when I get the patch finshed/tested. Thanks!

Joel filed Bug 1328915