AWSY exhibit a bi-modal behaviour on Try [1], which is not reflected on mozilla-central. This causes perfherder to report regressions which can freeze developers from landing their changes. https://treeherder.mozilla.org/perf.html#/graphs?series=try,1706923,1,4&series=mozilla-central,1707015,1,4&highlightAlerts=0

It looks like there was a brief spike around 10/2 and then continued bimodality around 10/8 which correlates pretty well with bug 1409739 which added support for running with dmd enabled. I'd expect the dmd numbers to have their own graph, maybe this is an issue with perfherder not splitting that out. nbp, does this reproduce if you just focus on your changes? ie do a baseline push w/ retriggers, then a push w/ your changes w/ retriggers and then just compare those two in perfherder? That's how I would normally recommend

(In reply to Eric Rahm [:erahm] from comment #1) > It looks like there was a brief spike around 10/2 and then continued > bimodality around 10/8 which correlates pretty well with bug 1409739 which > added support for running with dmd enabled. I'd expect the dmd numbers to > have their own graph, maybe this is an issue with perfherder not splitting > that out. I've confirmed that the higher values have DMD artifacts and DMD overhead in the memory report whereas the lower values do not.

Okay, it looks like we need to manually annotate the perf data [1] similar to how we did with stylo. Should be pretty straightforward. [1] https://searchfox.org/mozilla-central/rev/c56977420df7a1b692ce0f7e499ddb364d9fd7b2/testing/awsy/awsy/process_perf_data.py#85-88

This will split out AWSY results with DMD enabled from standard results, thus avoiding confusion due to a large discrepency in values due to DMD's overhead.

Based on comment 3, I guess it would have. If you still need me to double check re-needinfo me. Thanks for looking at this issue.

Comment on attachment 9018341 [details] [diff] [review] Annotate DMD perfherder results Review of attachment 9018341 [details] [diff] [review]: ----------------------------------------------------------------- ::: testing/awsy/awsy/process_perf_data.py @@ +89,2 @@ > if 'STYLO_THREADS' in os.environ and os.environ['STYLO_THREADS'] == '1': > + extra_opts = ["stylo-sequential"] Is extra_opts = ["stylo", "stylo-sequential"] a possibility? This change is consistent with the previous behavior, but I wonder are STYLO_FORCE_ENABLED and STYLO_THREADS mutually exclusive? If not, shouldn't we append here instead of setting extra_opts to a fixed list? If they are mutually exclusive do you think it would help readability if we used if...elif here?

Comment on attachment 9018341 [details] [diff] [review] Annotate DMD perfherder results Review of attachment 9018341 [details] [diff] [review]: ----------------------------------------------------------------- Either way, this works for me. r+

(In reply to Bob Clary [:bc:] from comment #7) > Comment on attachment 9018341 [details] [diff] [review] > Annotate DMD perfherder results > > Review of attachment 9018341 [details] [diff] [review]: > ----------------------------------------------------------------- > > Either way, this works for me. r+ Thanks, I'll go ahead and update to an elif.

(In reply to Eric Rahm [:erahm] from comment #8) > (In reply to Bob Clary [:bc:] from comment #7) > > Comment on attachment 9018341 [details] [diff] [review] > > Annotate DMD perfherder results > > > > Review of attachment 9018341 [details] [diff] [review]: > > ----------------------------------------------------------------- > > > > Either way, this works for me. r+ > > Thanks, I'll go ahead and update to an elif. Oh right, I'm going to keep it as-is. Both can be specified, but need to be reported differently. STYLO_THREADS=1 overrides the default stylo behavior. STYLO_FORCE_ENABLED isn't really needed anymore, but removing it would mess up our perf data bucketing. I'll file a follow-up to clean all this up.

https://hg.mozilla.org/integration/mozilla-inbound/rev/ce6fcc65a415d3bee52757e3d5a97216ad27daa7 Bug 1500176 - Annotate DMD perfherder results. r=bc