I need to debug a PGO-only Win64 issue, see bug 1180948 and bug 1167883. The former is keeping Aurora closed, so this is pretty urgent. Unfortunately all PGO builds I do are timing out on Try: Try push from yesterday: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebd81d3a3722 Try push from last month: https://treeherder.mozilla.org/#/jobs?repo=try&revision=38a8b8a65b32

I've been looking at this today - no leads just yet. The timeouts for try look to be the same as for mozilla-central, so something is causing this to take a lot longer to build than nightlies on trunk.

Disabling sccache seems to have fixed this for me: https://hg.mozilla.org/try/rev/fa7efed1756d#l2.17 glandium, any ideas why?

As I said on irc, this might be due to the difference in how debug symbols are treated with sccache enabled. That's not a problem on non-try because MOZ_PGO is set before mozconfig.cache is included, and mozconfig.cache disables sccache when MOZ_PGO is set.

Why would having sccache enabled make PGO builds so much slower? In particular, the 2nd link of xul.dll is taking more than 2x as long.

See comment 3.

Anyways, the reasonable thing to do would be to ensure mozconfig.cache is included last (or at least after mozconfig.common.override) instead of wherever it's included now.

(In reply to Mike Hommey [:glandium] from comment #6) > Anyways, the reasonable thing to do would be to ensure mozconfig.cache is > included last (or at least after mozconfig.common.override) instead of > wherever it's included now. great - can you get a patch up for that?

It wasn't enough. sccache doesn't get disabled because mk_add_options doesn't have an effect on local variables in mozconfig, so the tests in mozconfig.cache don't work.

It is useful to be able, during mozconfig execution, to do tests depending on what was previously added with mk_add_options. Specifically, there is a need to do this for MOZ_PGO because developers pushing to try may add it to mozconfig.common.override. While, ideally, it would be nice if we just defined the variable itself in the mozconfig execution environment, that is a tedious task, having to jump through hoops with eval, and handle all cases of variable assigment properly. The hacky alternative is to just treat MOZ_PGO specially, but meh. So instead, we set a ${var}_IS_SET variable to 1, indicating that a mk_add_options defined ${var} to some value.

Comment on attachment 8644180 [details] [diff] [review] Set ${var}_IS_SET variables for mk_add_options-defined variables Review of attachment 8644180 [details] [diff] [review]: ----------------------------------------------------------------- I'm not a huge fan of adding more complexity to mozconfig evaluation. But I don't have a better idea that is just as simple as this.

https://hg.mozilla.org/integration/mozilla-inbound/rev/06de69c7ea12 https://hg.mozilla.org/integration/mozilla-inbound/rev/c24075996e1d

https://hg.mozilla.org/mozilla-central/rev/06de69c7ea12 https://hg.mozilla.org/mozilla-central/rev/c24075996e1d