Closed Bug 1466525 Opened 6 years ago Closed 2 years ago

Can't compare try pushes using `mozharness: --geckoProfile`

Categories

(Testing :: Talos, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ochameau, Unassigned)

References

Details

(Whiteboard: [PI:June])

When pushing to try with `mozharness: --geckoProfile`, DAMP no longer uploads `perfherder-data.json` artifact, nor generates PERFHERDER_DATA in its output. Thus we can no longer compare builds that are profiled. This isn't handy as it doesn't allow to know if a regression that we are tracking without the profiler, also reproduces with the profiler. Here is two example try pushes: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b5b8629a272b5b763f5a9c8b7cafacc5bf4c841&selectedJob=181693848 https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8e543e21997254f9cedf6f86145a403c39839dc
this is by design- a profiled build will produce misleading data and can skew our results for alerts and trends. Why are you trying to compare results with --geckoProfile?
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1) > this is by design- a profiled build will produce misleading data and can > skew our results for alerts and trends. Why are you trying to compare > results with --geckoProfile? I totally agree that the data would be misleading, especially when comparing with a non-profiled one. But it could also help ensuring that a particular regression also reproduces with the profiler turned on. Honza originaly reported that to me when looking at a hard to debug regression on DAMP, on a "settle" subtest regression that involved spurious asynchronous code overlapping between two subtests.
maybe for try runs we could allow it to upload data with profile on
Whiteboard: [PI:June]
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #3) > maybe for try runs we could allow it to upload data with profile on Yes, this would help. The perf/damp issue I am debugging is only reproducible on Try and I wanted to also get profile data to analyze it using perf.html Honza

Here's the logic that prevents the Perfherder data from being logged when --geckoProfile is specified: https://searchfox.org/mozilla-central/rev/15be167a5b436b57fef944b84eef061d24c1af8c/testing/talos/talos/output.py#210-214

We would likely need to move that logic into the mozharness code to allow us to output the results but prevent us from submitting the results to Treeherder.

Priority: -- → P3

:ochameau, this should be resolved now: https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#221-229

Feel free to re-open if this is still an issue.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.