Creating library xul.lib and object xul.exp Generating code c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\qcms\transform_util.c(134) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\qcms\transform_util.c(170) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\qcms\transform_util.c(134) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\cairo\cairo\src\cairo.c(2811) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\cairo\cairo\src\cairo.c(2812) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\cairo\cairo\src\cairo.c(2813) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\gfx\cairo\cairo\src\cairo.c(2814) : warning C4756: overflow in constant arithmetic c:\builds\moz2_slave\b2g-in-w64-pgo-000000000000000\build\js\src\jscntxt.cpp(857) : warning C4700: uninitialized local variable 'dummy' used Finished generating code Unified_cpp_accessible_xul0.obj : fatal error LNK1318: Unexpected PDB error; OK (0) '??' c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/config/rules.mk:812: recipe for target 'xul.dll' failed mozmake.exe[6]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/obj-firefox/toolkit/library' mozmake.exe[6]: *** [xul.dll] Error 1318 c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/config/recurse.mk:74: recipe for target 'toolkit/library/target' failed mozmake.exe[5]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/obj-firefox' mozmake.exe[5]: *** [toolkit/library/target] Error 2 c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/config/recurse.mk:36: recipe for target 'compile' failed mozmake.exe[4]: *** [compile] Error 2 mozmake.exe[4]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/obj-firefox' c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/config/rules.mk:541: recipe for target 'default' failed mozmake.exe[3]: *** [default] Error 2 mozmake.exe[3]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/obj-firefox' c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/client.mk:398: recipe for target 'realbuild' failed mozmake.exe[2]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build' mozmake.exe[2]: *** [realbuild] Error 2 c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build/client.mk:233: recipe for target 'profiledbuild' failed mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/b2g-in-w64-pgo-000000000000000/build' mozmake.exe[1]: *** [profiledbuild] Error 2 client.mk:171: recipe for target 'build' failed mozmake.exe: *** [build] Error 2

The list of slaves here sure looks suspiciously like the other problem slaves we're seeing in other bugs.

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #16) > The list of slaves here sure looks suspiciously like the other problem > slaves we're seeing in other bugs. filed 1122975 to investigate individual slaves as a culprit. note: these are all win64 builds for this bug specically we should concentrate on: * not enough memory * mspdbsrc.exe * cache, debug symbol server all the above are parsed from http://stackoverflow.com/questions/4256524/lnk1318-unexpected-pdb-error-ok-0 mshal: buildduty and relops can poke the slaves (1122975). would you or someone from build team help investigate these failures that may be specific to this bug?

I could try, though you might want someone with more Windows/VS knowledge.

I wonder if this is the same root cause as bug 1093664. These machines are 4gb ram + 4gb swap, right? Can we try doubling the pagefile?

(In reply to David Major [:dmajor] (UTC+13) from comment #20) > I wonder if this is the same root cause as bug 1093664. These machines are > 4gb ram + 4gb swap, right? Can we try doubling the pagefile? I think so, see latest findings in https://bugzilla.mozilla.org/show_bug.cgi?id=1122975#c5

Do we also need to update https://developer.mozilla.org/en-US/docs/Simple_Firefox_build to indicate that the Hardware Requirements are no longer: Recommended: 4GB of RAM (having only 2GB RAM and 2GB swap may give memory errors during compile) but instead: Recommended: 8GB of RAM (having only 4GB RAM and 4GB swap may give memory errors during compile) ?

(In reply to Blake Winton (:bwinton) from comment #23) > Do we also need to update > https://developer.mozilla.org/en-US/docs/Simple_Firefox_build to indicate > that the Hardware Requirements are no longer: > Recommended: 4GB of RAM (having only 2GB RAM and 2GB swap may give memory > errors during compile) > but instead: > Recommended: 8GB of RAM (having only 4GB RAM and 4GB swap may give memory > errors during compile) > ? Yes and done.

Non-pgo builds can probably get by with 4 gigs. Oh well.

Could we add instructions on how to do a non-pgo build to that page, too, so that the new contributor I'm mentoring will be able to build Firefox without having to upgrade his computer? ;)

Builds are non-pgo by default.

Hmm. The person I'm mentoring is running into this problem with 4 gigs, so perhaps we can't get by with that little anymore. :( Thanks, though!

Blake, can I ask you to dig into this a little more? I'd like to know what machine and OS your contributor is hitting this on, the jump from a 4GB minimum to an 8GB minimum is going to be a problem for a lot of our contributors, so I'd really like to know what's going on here.

notice we haven't hit this in production since Jan 21st which is around the time we discovered we had some win machines in our pool with 4gb and then upgraded the rest to 8 https://bugzilla.mozilla.org/show_bug.cgi?id=1122975 https://bugzil.la/1125887

Our build infra should be the best hardware money can buy, no question, but we can't ask our community for that.

Hi,my machine has a 64-bit processor (Intel i5-2430M @ 2.40 GHz) and it's running on 32-bit Windows 8.1. If I check my System Information,it says my machine has 4GB of RAM,with 2.92 of it usable.

I've been trying tweaks into the mozconfig file,but the problem persists.

On a 32-bit Windows, the compiler is limited to 2GB memory. That is probably too restrictive. (Yes, I am aware that there is an OS setting to increase the limit to 3GB, but I don't think we should be advocating it as part of our build instructions. And, it still may not be enough.) Just to make sure it's not an isolated machine issue, it would be great if someone else could confirm that the build doesn't work on a 32-bit Windows. If that's the case then we'll need to specify 64-bit on the requirements.

I'm really uncomfortable with that; for a lot - probably most - of our contributors, "upgrade your hardware/OS in order to keep participating" is a huge ask. I'm not saying we can't or shouldn't, but I'm definitely saying that we should do so with a lot more caution and deliberation than updating the wiki and calling it a day. Ankit, I understand that this is frustrating, but figuring out how to address this on your machine is really important for a lot of contributors, so I hope you can help us work this out. So: https://msdn.microsoft.com/en-us/library/windows/hardware/bb613473%28v=vs.85%29.aspx Can you take a look at that document, and this one? https://msdn.microsoft.com/en-us/library/ff542202.aspx They suggest that if you run this command: bcdedit /set increaseuserva 3072 as administrator and reboot, that this will increase the amount of addressable space available to user processes on 32-bit Windows systems. Please check my work there first, but if that looks correct to you per the docs I'd like you to try to build after you've run that command. If it succeeds, that's useful information and we can update the documentation. If it doesn't, you'l need to run: bcdedit /deletevalue increaseuserva as administrator to revert that configuration and then restart again. Thank you.

Unfortunately,the build doesn't go through and halts at the same point even after having run the command and rebooting. Can I be sure it would work on 64-bit Windows?

(In reply to ANKIT JENA from comment #36) > Unfortunately,the build doesn't go through and halts at the same point even > after having run the command > and rebooting. Can I be sure it would work on 64-bit Windows? Did you try to run mach build -j1?

ANKIT - I'm somewhat heartened by the research I've been doing here, which suggests we can solve this without reinstalling your OS. It looks like this is because mspdbsrc.exe is holding on to a pile of memory - possibly from a previous build - and then crashing. Before we try this scorched-earth approach, can you try: - clearing the build cache, - in VS, going into Project properties -> C/C++ -> Code Generation -> Enable Function Level Linking -> and setting that to Yes ... and building again? If that fails - and to all reports it should not - try upgrading to VS 2013 Community edition? http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx#d-community

(In reply to Mike Hommey [:glandium] from comment #37) > (In reply to ANKIT JENA from comment #36) > > Unfortunately,the build doesn't go through and halts at the same point even > > after having run the command > > and rebooting. Can I be sure it would work on 64-bit Windows? > > Did you try to run mach build -j1? Yes,I did.

(In reply to Mike Hoye [:mhoye] from comment #38) > ANKIT - I'm somewhat heartened by the research I've been doing here, which > suggests we can solve this without reinstalling your OS. It looks like this > is because mspdbsrc.exe is holding on to a pile of memory - possibly from a > previous build - and then crashing. > > Before we try this scorched-earth approach, can you try: > > - clearing the build cache, > - in VS, going into Project properties -> C/C++ -> Code Generation -> Enable > Function Level Linking -> and setting that to Yes > > ... and building again? > > If that fails - and to all reports it should not - try upgrading to VS 2013 > Community edition? > > http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx#d- > community MIKE,if you could please tell me what is the actual project file that I need to open and change the property of?Because I think that change in property can only be made to a specific Project and not in general. Thanks :)

I don't know - that was one of the proposed fixes, but I haven't been able to reproduce the problem. Can I ask you to try installing VS2013 Community, per the link?

Wait! Before you do that, can you put as much information about your _current setup_, before you install VS2013CE , into this bug? I'd like to know which versions of what you've got installed, in case we run into this problem in the future. Thanks so much for your work on this, I know how tedious this stuff is, but if we can get this right you'll have helped a lot of contributors who are coming after you avoid these pitfalls.

Changing the generated project files won't do anything: we build through the Firefox build system, not Visual Studio's. The settings in the VS project files only really buy us proper IntelliSense.

(In reply to Mike Hoye [:mhoye] from comment #42) > Wait! Before you do that, can you put as much information about your > _current setup_, before you install VS2013CE , into this bug? I'd like to > know which versions of what you've got installed, in case we run into this > problem in the future. > > Thanks so much for your work on this, I know how tedious this stuff is, but > if we can get this right you'll have helped a lot of contributors who are > coming after you avoid these pitfalls. MIKE,I've got the Visual Studio Express for Windows Desktop 2013 edition installed on my machine and June 2010 edition of DirectX SDK,the latest mozilla-build package and other prerequisites as specified in Windows Build Prerequisites page.

Greg, do you have any insight here? This kind of smells like a problem with the options being passed to the linker. I'm trying to replicate it now, but getting spun up will take some time.

The VS2013 linker can handle much bigger workloads than its predecessor, so as of v36 we are making the linker work a lot harder than before (bug 922912 and bug 609976 come to mind). I assume that's what has made RAM the limiting factor.

We made changes to the build that make the linker work harder. This will increase RAM utilization and thus minimum requirements to build Firefox. Open Question: Is VS2013 Community Edition more efficient with RAM than earlier versions of Visual Studio *with non-PGO linking*? Is it enough to still be able to build on memory constrained machines? If so, we might want to push VS2013 hard onto people with not much RAM. We may also want to look at making configure apply more reasonable defaults (such as JS_SHARED_LIBRARY) when memory is constrained. We've dabbled into hardware detection impacting build settings before. It's a sensitive topic and one prone to accidents. Quite frankly, I don't want to rekindle bad memories. I'd much rather invest in build modes that pull down a pre-built libxul so non-C++ developers don't have to waste time compiling and linking libxul.

> If so, we might want to push VS2013 hard onto people with not much RAM. VS2013 is now required to build at all, no?

As of mid December, Visual Studio 2012 and later is the minimum required to support the C++11 feature set we use. The documentation points all new contributors to 2013 Community Edition, which is the current best available for free and doesn't put a huge burden on contributors in a hardware or OS upgrade sense.

As of bug 1117900 we only support 2013 update 3 and newer.

OK, I can reliably duplicate this in a VM now, with VS2013sp4 on Win7 Ankit - I don't have an answer for you yet, I'm sorry. GPS: I'd really like to have at least some kind of low-and-slow option for compiling, even if it's just a howto guide, until we can figure out how to ship prebuilt components.

(In reply to Mike Hoye [:mhoye] from comment #51) > OK, I can reliably duplicate this in a VM now, with VS2013sp4 on Win7 > > Ankit - I don't have an answer for you yet, I'm sorry. > > GPS: I'd really like to have at least some kind of low-and-slow option for > compiling, even if it's just a howto guide, until we can figure out how to > ship prebuilt components. That's all right Mike as long as we are heading towards a conclusion.Please let me know in whatever way I can contribute to the problem at hand. :)

OK, the last thing I can think to try is: If you've got "ac_add_options --enable-debug" in your mozconfig, remove it. Add in "mk_add_options GKMEDIAS_SHARED_LIBRARY=1" in its place, and try that? I'm trying that as well.

Well, that failed for me. Now attempting with: mk_add_options GKMEDIAS_SHARED_LIBRARY=1 mk_add_options ENABLE_SHARED_JS=1

After testing on VMs of various sizes, I've filed bug 1137346. As far as I can tell it is no longer possible to build Firefox on Win32 regardless of hardware. Ankit - I think you might need to upgrade, I'm sorry. Let me see if we can make your life a bit easier on that front, at least.

Not a problem.

Kind of a problem.

(In reply to Mike Hoye [:mhoye] from comment #57) > Kind of a problem. Okay Mike,the build successfully went through with 64-bit Windows.Thanks! :)

Fantastic. Glad to see some good news in this bug, even if it's not the news we'd hoped for. I'm going to mark this bug dependent on bug 1137346, and close it wontfix. Thanks for your help and for sticking this one out.