this works for me locally on win10, but fails on try- I think a few small tweaks and it will be good.

Currently stylebench and motionmark run on talos, these are ideal benchmarks to run on raptor

Comment on attachment 8986816 [details] Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood Robert Wood [:rwood] has approved the revision. https://phabricator.services.mozilla.com/D1752

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/683a682e52b2 Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Backed out changeset 683a682e52b2 (bug 1466853) for flake fail on checkouts/gecko/testing/raptor/raptor/benchmark.py on a CLOSED TREE Backout link: https://hg.mozilla.org/integration/autoland/rev/71705fb3d3cfe07453fb90b3f0c44bfe33551610 Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=683a682e52b2ffdc19f7972e45f2f58fedbb80c6 Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=184258421&repo=autoland&lineNumber=273 Log snippet: [vcs 2018-06-21T20:44:43.043Z] 244421 files updated, 0 files merged, 0 files removed, 0 files unresolved [vcs 2018-06-21T20:44:43.296Z] updated to 683a682e52b2ffdc19f7972e45f2f58fedbb80c6 [vcs 2018-06-21T20:44:43.300Z] PERFHERDER_DATA: {"framework": {"name": "vcs"}, "suites": [{"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "clone", "shouldAlert": false, "subtests": [], "value": 133.05697107315063}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "pull", "shouldAlert": false, "subtests": [], "value": 9.588178157806396}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "update", "shouldAlert": false, "subtests": [], "value": 92.2714729309082}, {"extraOptions": ["m3.xlarge"], "lowerIsBetter": true, "name": "overall", "shouldAlert": false, "subtests": [], "value": 236.68608498573303}]} [vcs 2018-06-21T20:44:43.703Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/autoland/rev/683a682e52b2ffdc19f7972e45f2f58fedbb80c6 title='Built from autoland revision 683a682e52b2ffdc19f7972e45f2f58fedbb80c6'>683a682e52b2ffdc19f7972e45f2f58fedbb80c6</a> [task 2018-06-21T20:44:43.703Z] executing ['bash', '-cx', 'cd /builds/worker/checkouts/gecko && ./mach lint -l flake8 -f treeherder'] [task 2018-06-21T20:44:43.706Z] + cd /builds/worker/checkouts/gecko [task 2018-06-21T20:44:43.706Z] + ./mach lint -l flake8 -f treeherder [task 2018-06-21T20:44:44.336Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7 [task 2018-06-21T20:44:44.336Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python [task 2018-06-21T20:44:45.984Z] Installing setuptools, pip, wheel...done. [task 2018-06-21T20:44:47.121Z] running build_ext [task 2018-06-21T20:44:47.121Z] building 'psutil._psutil_linux' extension [task 2018-06-21T20:44:47.121Z] creating build [task 2018-06-21T20:44:47.121Z] creating build/temp.linux-x86_64-2.7 [task 2018-06-21T20:44:47.121Z] creating build/temp.linux-x86_64-2.7/psutil [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o [task 2018-06-21T20:44:47.121Z] creating build/lib.linux-x86_64-2.7 [task 2018-06-21T20:44:47.121Z] creating build/lib.linux-x86_64-2.7/psutil [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so [task 2018-06-21T20:44:47.121Z] building 'psutil._psutil_posix' extension [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o [task 2018-06-21T20:44:47.121Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so [task 2018-06-21T20:44:47.122Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil [task 2018-06-21T20:44:47.122Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil [task 2018-06-21T20:44:47.122Z] [task 2018-06-21T20:44:47.122Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt) [task 2018-06-21T20:45:28.811Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/raptor/raptor/benchmark.py:59:29 | continuation line over-indented for visual indent (E127)

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/96496ebab2d9 Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

Backed out changeset 96496ebab2d9 (bug 1466853) for StyleBench failures on a CLOSED TREE Backout link: https://hg.mozilla.org/integration/autoland/rev/952031252a25a9846f6577fe6ab01dc0fca52dc8 Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=96496ebab2d9d35e8b855a475195e9a4df877cbc Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=184291896&repo=autoland&lineNumber=1830 Log snippet: 16:38:07 INFO - TEST-INFO | 24253: exit 0 16:38:07 INFO - TEST-OK | stylebench | took 77738ms 16:38:07 INFO - SUITE-END | took 304s 16:38:07 INFO - Completed test suite (00:05:05) 16:38:07 ERROR - Traceback (most recent call last): 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 508, in <module> 16:38:07 INFO - main() 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 504, in main 16:38:07 INFO - sys.exit(run_tests(config, browser_config)) 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/run_tests.py", line 327, in run_tests 16:38:07 INFO - talos_results.output(results_urls) 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/results.py", line 43, in output 16:38:07 INFO - results = _output() 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 129, in __call__ 16:38:07 INFO - vals, testname=test.name()) 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 297, in construct_results 16:38:07 INFO - return self.stylebench_score(vals) 16:38:07 INFO - File "/home/cltbld/workspace/build/tests/talos/talos/output.py", line 277, in stylebench_score 16:38:07 INFO - raise Exception("StyleBench has 52 subtests, found: %s instead" % len(results)) 16:38:07 INFO - Exception: StyleBench has 52 subtests, found: 53 instead 16:38:07 ERROR - Return code: 1 16:38:07 WARNING - setting return code to 1 16:38:07 ERROR - # TBPL WARNING #

stylebench and motionmark run in talos, we want these running in raptor

Comment on attachment 8987598 [details] Bug 1466853 - Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood Robert Wood [:rwood] has approved the revision. https://phabricator.services.mozilla.com/D1814

Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/b51ba2011412 Port stylebench and motionmark benchmarks test from talos to raptor. r=rwood

FYI, this change updated our baselines for stylebench: == Change summary for alert #14014 (as of Mon, 25 Jun 2018 18:23:48 GMT) == Regressions: 47% stylebench osx-10-10 opt e10s stylo 34.61 -> 18.41 44% stylebench windows10-64 opt e10s stylo 63.85 -> 35.50 44% stylebench windows7-32 pgo e10s stylo 67.09 -> 37.44 44% stylebench windows7-32 opt e10s stylo 51.65 -> 28.83 43% stylebench linux64 pgo e10s stylo 72.28 -> 40.92 43% stylebench linux64 opt e10s stylo 65.57 -> 37.14 43% stylebench windows10-64 pgo e10s stylo 72.54 -> 41.61 41% stylebench linux64-qr opt e10s stylo 66.25 -> 39.22 For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14014