If you look at "tsvgr_opacity opt" for osx-10-10 at https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&originalRevision=8e5c888d0d89&newProject=try&newRevision=103193c1cd13&hideMinorChanges=1 you will see a supposed 83% regression (at the time of writing) with "high" confidence. > Base Geomean Base StdDev New Geomean New StdDev Delta Delta % Confidence > 398.11 12 +/-334.55 (84.03%) 729.46 6 +/-55.06 (7.55%) 331.35 83.23% 3.34 (high) I'm pretty sure something is wrong here since when I look at the toolkit for the Base Geomean I get: > Runs: < 69.66 71.94 75.62 76.65 87.9 93.99 655.1 666.89 717.1 738.03 755.75 768.7 > 6 runs are ~75, the other 6 runs are ~700, an order of magnitude different. A similar situation applies to other/all tests on 10.10 e.g. "ts_paint opt", probably any that have 12 runs instead of 6. This may end up being a Talos problem but I'm not sure where the extra data is coming from.

a great problem u uncovered, and I believe I can decipher some of this. 1) the values are correct for try 2) the values for m-c include the 's' jobs and the '?' jobs which are 's-e10s' also :) (click the magical square box to show hidden jobs in treeherder at the revision level to see the e10s related jobs) so the low values for m-c is e10s. Now why is this getting mixed up- that is what we need to figure out. I know we do a good job of keeping e10s results away from non-e10s results in perfherder- but osx 10.10 seems to be the exception. In fact, I think the real issue here is that the jobs don't have proper codes and something quirky is going on in treeherder ingestion (bug 1168360) For reference here is a link to a raw log: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/07/2015-07-22-03-02-05-mozilla-central/mozilla-central_yosemite_test-svgr-osx-e10s-bm106-tests1-macosx-build126.txt.gz all the talosdata info looks good there. Another odd thing is clicking on the subtest results seems to not load subtests, on any platform. :wlach, can you take a look at why the subtests are not loading and maybe weigh in on why we are mixing the e10s data up when the job is a ?

Ok, the classification of the e10s property occurs here, based on job symbol: https://github.com/mozilla/treeherder/blob/master/treeherder/etl/perf_data_adapters.py#L247 Since the job symbol was '?', we didn't know to add the e10s property. This is fixed by jmaher's commit to treeherder in bug 1168360. Unfortunately in the interim some bad data was submitting to treeherder/perfherder and it'll be difficult to take it out. To prevent this from happening in the future, I filed bug 1186685.

(In reply to Joel Maher (:jmaher) from comment #1) > Another odd thing is clicking on the subtest results seems to not load > subtests, on any platform. :wlach, can you take a look at why the subtests > are not loading and maybe weigh in on why we are mixing the e10s data up > when the job is a ? The subtests not loading was due to a dumb bug, unrelated. Pretty trivial and obvious fix, which I just went ahead and committed: https://github.com/mozilla/treeherder/commit/d9938bb715e18a873c861845209803f91fb94741