with runbydir enabled for all of mochitest-chrome, I would like to split oth up. a11y will be fast, but chrome takes a while and a bit longer with run-by-dir. I propose 3 chunks for mochitest-chrome.

this will need: * buildbot-configs * trychooser syntax * treeherder changes I assume this will ride the trains starting in 44 on trunk and carry on.

I've filed bug 1211905 for the Treeherder parts (it's easier to track changes if they are in our component). Un CCing for now to reduce bugmail - once you know the buildernames drop a message in that bug :-)

these are buildbot configs, we still need mozharness changes for this to work, let alone other dependencies.

Comment on attachment 8670349 [details] [diff] [review] buildbot configs (1.0) Review of attachment 8670349 [details] [diff] [review]: ----------------------------------------------------------------- This looks good to me

armen, this is needed prior to landing the buildbot changes, do let me know if I am overlooking something!

landed code to split up m-oth: https://hg.mozilla.org/build/buildbot-configs/rev/47df7227fa0f when the next reconfig happens this will be live.

backed out https://hg.mozilla.org/build/buildbot-configs/rev/de29b5545a08, builder limits: tst-linux64-spot-544 has 4131 builders; limit is 4084; 101 percent of max :catlee, do I just need to find other tst-linux64-spot-544 jobs to disable?

as seen in this log file: https://travis-ci.org/mozilla/build-buildbot-configs/jobs/85146131

found some builders and landed: http://hg.mozilla.org/build/buildbot-configs/rev/1e5812a86db3 tests passed successfully- this will get picked up in the next reconfig, just missed the one from today.

These new jobs are all failing with "Required config file not set! (use --config-file option)" at the start of the desktop_unittest.py step.

this is the command that is run when this was live: /tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols ondemand and the error: Required config file not set! (use --config-file option) for a dt5 run, we see this type of command executed: /tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite mochitest-devtools-chrome-chunked --total-chunks 8 --this-chunk 5 --blob-upload-branch mozilla-inbound --download-symbols ondemand what appears to be missing is " --cfg unittests/linux_unittest.py ". Most likely we would need to add stubs similar to this for mochitest-a11y and mochitest-chrome-chunked: https://dxr.mozilla.org/build_buildbot-configs/source/mozilla-tests/config.py#876 and for osx and windows as well. :armenzg, can you confirm that, I would be happy to give this another go.

Yes, this is correct. I should have caught this in review. It would be nice if test-master.sh would catch issues like this.

same bbconfig differences, just updated to have --cfg for each job type!

landed this again: https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd will see if a reconfig yields success.

In production: https://hg.mozilla.org/build/buildbot-configs/rev/5d2bd6c3ccdd

Backed out: https://hg.mozilla.org/build/buildbot-configs/rev/a25e920f6d1d I will file a bug that will preven this happening in the future. http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1445613808/mozilla-inbound_ubuntu64_vm-debug_test-mochitest-a11y-bm51-tests1-linux64-build1.txt.gz /tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --mochitest-suite a11y --blob-upload-branch mozilla-inbound --download-symbols true in dir /builds/slave/test/. (timeout 1800 secs) (maxTime 1800 secs) watching logfiles {} argv: ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--mochitest-suite', 'a11y', '--blob-upload-branch', 'mozilla-inbound', '--download-symbols', 'true'] environment: CCACHE_DIR=/builds/ccache CCACHE_UMASK=002 DISPLAY=:0 HOME=/home/cltbld LANG=en_US.UTF-8 LOGNAME=cltbld MAIL=/var/mail/cltbld MOZ_HIDE_RESULTS_TABLE=1 MOZ_NODE_PATH=/usr/bin/node MOZ_NO_REMOTE=1 NODE_PATH=/usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript NO_FAIL_ON_TEST_ERRORS=1 PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games PROPERTIES_FILE=/builds/slave/test/buildprops.json PWD=/builds/slave/test SHELL=/bin/bash SHLVL=1 TERM=linux TMOUT=86400 USER=cltbld XDG_SESSION_COOKIE=9ca12473fbb1d023794ffd180000023c-1445617269.523098-45509168 _=/tools/buildbot/bin/python using PTY: False Required config file not set! (use --config-file option) program finished with exit code 255

Armen, I am unclear on how this happened since we had this same error the first time and fixed it in the new patch/review. some how we are missing the --cfg.

Before I backed out, I could not determine why it would failed (henced backed out). This time I've noticed that this was only added for the 'linux' block and not 'linux64'.

ok, this should do the trick!

Comment on attachment 8680242 [details] [diff] [review] added configs for all platforms (3.0) Review of attachment 8680242 [details] [diff] [review]: ----------------------------------------------------------------- Let's hope this time it works :)

ok, round 3: https://hg.mozilla.org/build/buildbot-configs/rev/8af4ed62fb0f

backed out! https://hg.mozilla.org/build/buildbot-configs/rev/cb2919a9a732 this time we needed builders to open aurora by adding more chunks.

the actual patch that was supposed to be backed out: http://hg.mozilla.org/build/buildbot-configs/rev/3171c91cf580

this previous patch is for all platforms, on linux32 debug we are hitting timeouts more frequently- for taskcluster (linux64 debug) we are already doing this split- for now lets adjust linux32 debug only.

and the differences file

Comment on attachment 8711690 [details] [diff] [review] oth -> a11y/c1/c2/c3 for linux32 debug only Could you also please paste the list_builders_differences output for posterity? Thanks!

landed: https://hg.mozilla.org/build/buildbot-configs/rev/d9e027f12297

http://hg.mozilla.org/build/buildbot-configs/rev/c06c1be8884f and in production.

Do we want to make this change for linux64 as well?

we do, I couldn't land this last year because I hit the builder limits on linux64.

Removing leave-open keyword from resolved bugs, per :sylvestre.