We need to quickly start investigating the different paths to overcome the PGO problem on Windows. I'll use this bug to track the progress to gather more information and to finally make a decision on what needs to be done.

From what I gather from the PGO thread, it sounds like although we know we're going to get hit with an OOM condition for PGO builds, we are caught off-guard when the build actually runs out of virtual memory. While not a mitigation strategy in of itself, it would be good to institute an early warning system (ok, a notification), when we exceed a threshold of virtual memory usage during a build. This notification should provide a cue to releng and engineering that we're about to hit a PGO failure in the build and that we have to start working on the different mitigation strategies that Ehsan in investigating.

(In reply to comment #1) > From what I gather from the PGO thread, it sounds like although we know we're > going to get hit with an OOM condition for PGO builds, we are caught off-guard > when the build actually runs out of virtual memory. While not a mitigation > strategy in of itself, it would be good to institute an early warning system > (ok, a notification), when we exceed a threshold of virtual memory usage during > a build. This notification should provide a cue to releng and engineering that > we're about to hit a PGO failure in the build and that we have to start working > on the different mitigation strategies that Ehsan in investigating. Absolutely. That's covered by bug 710840.

Here's a mem size number from a pgo build using vs 2012 update 1 on my local system: my system: 3,839,270,912 Comparing that to the last pgo on mc: linker max vsize: 3,452,080,128 Doesn't look like vs 2012 gives us anything. I've also posted some info on getting a builder set up and running in bug 833887.

(In reply to Jim Mathies [:jimm] from comment #3) > Here's a mem size number from a pgo build using vs 2012 update 1 on my local > system: > > my system: 3,839,270,912 > > Comparing that to the last pgo on mc: > > linker max vsize: 3,452,080,128 > > Doesn't look like vs 2012 gives us anything. > > I've also posted some info on getting a builder set up and running in bug > 833887. Thanks for the investigation, Jim.

On the slave I was lent, an opt pgo build with vs 2012 update 1 - linker-vsize: 3823702016 which is about what I was seeing locally.

(In reply to Jim Mathies [:jimm] from comment #5) > On the slave I was lent, an opt pgo build with vs 2012 update 1 - > > linker-vsize: 3823702016 > > which is about what I was seeing locally. That's a very good data point, thanks Jim! Looks like upgrading Visual C++ is not going to be a viable solution.

As a slightly longer term play, we should try to contact Microsoft and figure out if they can provide 64-bit compilers targeting x86 for this reason. They provide x86, x64, and x86-cross-to-x64 already. Do we have any contacts on the VS team? Another option would be for me or someone with MSDN to burn a support incident on this and see what they say.

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #7) > As a slightly longer term play, we should try to contact Microsoft and > figure out if they can provide 64-bit compilers targeting x86 for this > reason. They provide x86, x64, and x86-cross-to-x64 already. Do we have > any contacts on the VS team? Another option would be for me or someone with > MSDN to burn a support incident on this and see what they say. rbryce has the credentials for our MSDN subscription. I don't know of any vs contacts, and I wouldn't consider burning a support request on something like this to be a waste. Also for reference, metro bits should get over to mc and turned on in the next couple weeks. After an mc to elm merge elm vsize is currently 3464966144, mc is 3228155904.

(In reply to comment #7) > As a slightly longer term play, we should try to contact Microsoft and figure > out if they can provide 64-bit compilers targeting x86 for this reason. They > provide x86, x64, and x86-cross-to-x64 already. Do we have any contacts on the > VS team? Another option would be for me or someone with MSDN to burn a support > incident on this and see what they say. I've talked to a few people in the past (not in the recent past, of course) and none of them indicated that Microsoft has any plans to ship a 64-bit PGO compiler that targets x86. But I'm not opposed to asking again. Do you know who I should ping to get a contact on the VS team?

I reached out to my MS contact and the recommended fastest next step is to go with the support request as per comment 7. So +1 for Vlad to go for it.

If rbryce has MSDN support stuff set up, might be easier for him to do it -- mine is a new MSDN sub, and I apparently have to call them first to set up the support incidents.

(In reply to Vladimir Vukicevic [:vlad] [:vladv] from comment #11) > If rbryce has MSDN support stuff set up, might be easier for him to do it -- > mine is a new MSDN sub, and I apparently have to call them first to set up > the support incidents. Sorry for just now responding. I have access to the MSDN accounts only because Im a sysadmin and have access to all the passwords. That being said, I am glad to help any way I can. However its probably better that Ann Ignacio over in desktop handle MSDN support requests, extra accounts, etc. Between Ann and I, we can get you squared away.

(In reply to Rick Bryce [:rbryce] from comment #12) > Sorry for just now responding. I have access to the MSDN accounts only > because Im a sysadmin and have access to all the passwords. That being > said, I am glad to help any way I can. However its probably better that Ann > Ignacio over in desktop handle MSDN support requests, extra accounts, etc. > Between Ann and I, we can get you squared away. Over the years getting support requests filed and the ms support person connected to the appropriate mozilla dev has been a confusing and troublesome process. Since we seem to be working toward geting it nailed down now, I'd just like to request that we document this on the wiki such that whoever has "the power" knows what to do in the future, and can point devs to a wiki page so they know what to do. We really should be using this tool more often, we just didn't know how within the organization.

(Kind of unrelated to this bug, we should maybe start an email thread? :) The problem is that a handful of people have individual MSDN subs, and they each come with some (small -- like 2-4) number of support incidents per year. We don't have a pool of already-paid incidents to draw from, so we sort of need to figure out who has some available and use them. I agree that it's troublesome.. just given how MSDN is structured, I don't really know of a better way to do it :(

Posted an email summarizing our findings here: <https://groups.google.com/d/topic/mozilla.dev.platform/ckO-hox3BL8>. Time for discussion and a decision. :-)

The graph at http://graphs.mozilla.org/graph.html#tests=[[205,63,8]]&sel=none&displayrange=90&datatype=running seems to imply that we have 4-6 weeks left :-s

(In reply to comment #16) > The graph at > http://graphs.mozilla.org/graph.html#tests=[[205,63,8]]&sel=none&displayrange=90&datatype=running > seems to imply that we have 4-6 weeks left :-s I'll post a concrete plan to dev.platform later today hopefully.

I guess we can call this fixed now.

We still have an active support request open which I guess we basically burned. Are there any lingering questions we might ask before I close it out?

(In reply to comment #19) > We still have an active support request open which I guess we basically burned. > Are there any lingering questions we might ask before I close it out? How long can you keep it open for?