Currently get this error when running debug mochitest on win7: ERROR: Cannot resolve reference Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8. 0.50727.762". bc says: i think if we did the manifest right it would package the dlls we need. but looking at the executables, the manifest was empty. I would prefer not having to install visual studio on all talos slaves.

Ted, is it intentional that we package msvc{p,m,r}80.dll on debug, while on opt we have mozcrt19.dll ? We have --enable-jemalloc in the debug mozconfig. There's no mozcrt19.dll (or mozcpp19.dll) in the debug package at all.

Figured the packaging out. At http://hg.mozilla.org/mozilla-central/file/c753325a40ff/browser/installer/package-manifest.in#l60 we include the Microsoft CRT files if MOZ_MEMORY is not defined, otherwise our own mozcrt19 and cpp dlls. The mozconfig for the debug build is http://hg.mozilla.org/build/buildbot-configs/file/45f88d2ae58e/mozilla2/win32/mozilla-central/debug/mozconfig The --enable-jemalloc sets MOZ_MEMORY to 1, but --enable-tracemalloc disables it. AIUI we use tracemalloc to get bloat numbers, and also to detect leaks when running tests.

It's not going to help you either way, because the debug CRT is named differently. It's MSVCRTD80.dll. I'm not about to include that in the packaging manifest though, because Microsoft doesn't give you the rights to redistribute those files to other people, so I don't think we should even upload them to FTP.

We may be able to just copy the debug CRT DLLs to the SxS install folder on the Talos slaves. They get installed to something like c:\windows\winsxs\x86_microsoft.vc80.debugcrt... You could try locating that directory on a build slave, copying it to the same place on a Talos slave and running a debug build. Hopefully there's not other magic at work (like registry keys).

I tried that and couldn't get it to work. Not to say someone else won't be able to though.

Ok, maybe try doing it in a cheesier fashion, just grab the contents of C:\Program Files\Microsoft Visual Studio 9.0\VC\redist\Debug_NonRedist\x86\Microsoft.VC90.DebugCRT and stick it in c:\windows\system32 on the target machine?

comment #6, no love. I really don't want to have to install the whole thing. :(

Yuck. So, one more thought, that would require a little bit of work, would be to copy all those files (the DLLs + the manifest) into the firefox directory after unzipping the build. We used to ship Firefox that way, so I know it will work.

Something that still needs to be tried. bmoss proposed a fix for the missing visual studio environment involved a windows program that forces dlls to register themselves, could work by dropping the dlls we need into windows\system32 and then running that program (http://ss64.com/nt/regsvr32.html).

I think regsvr32 only works on COM components, not on regular DLLs, so I'm not sure it's what you need here.

Alice, are you actively working on this?

Pushing to joduinn.

Putting *debug* unit tests for now on the side (P4) until I get some more air to breathe.

I won't be able to touch this for a while. Check with me if you think I should pick this work ASAP over the other bugs I am already working on.

No surprise here, but this affects 64-bit Windows too. Missing MSVCR100D.dll in this case.

Not sure if having this installed could help in any way: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=bd512d9e-43c8-4655-81bf-9350143d5867 From bug 569268.

No, that's only the release CRT.

When I try to run a debug build from Tinderbox on my own Windows box, I get exactly the same error as in comment 0: Cannot resolve reference Microsoft.VC80.DebugCRT,processorArchitecture="x86",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="8.0.50727.762" I'm using Windows 7 (64-bit) and I have MSVC 2008 Express Edition installed. What else do I need to do in order to run debug builds from Tinderbox?

Version 8.00.50727.762 is MSVC 2005 SP1, I believe, so you'll need that version of the compiler toolchain installed for it to run.

sayre: fyi, this is whats blocking us from starting unittest-on-debug-builds-on-minis.

If you want to hand me a Talos slave I can try fiddling with this and see if I can figure out a way forward.

(In reply to comment #21) > If you want to hand me a Talos slave I can try fiddling with this and see if I > can figure out a way forward. I've sent Ted the login details for a Talos Win7 box.

Copying the contents of the "Microsoft.VC80.DebugCRT" dir into the app directory makes it possible to run the debug build. I'm going to see if I can find any alternative, but at least that's a fallback position.

Is the Debug.CRT stuff redistributable?

Okay, I've got a solution. Turns out Microsoft has a document describing this exact scenario: http://msdn.microsoft.com/en-us/library/aa985618%28v=VS.80%29.aspx except the simple solution there (msiexec /i whatever.msm) doesn't actually work. But, there's a free msm2msi converter available that produces a msi package that can be installed and does exactly what we need. Here's what I did: 1) Download msm2msi from http://www.ethalone.com/articles/61-using-windows-installer-merge-modules-in-ghost-installer-studio.html 2) Run msm2msi.exe "C:\Program Files\Common Files\Merge Modules\Microsoft_VC80_DebugCRT_x86.msm", which produces Microsoft_VC80_DebugCRT_x86.msi in the same "Merge Modules" directory. 3) Install the resulting msi on the Talos slave (I just used msiexec /i Microsoft_VC80_DebugCRT_x86.msi) I was then able to successfully run a debug Firefox build from the tinderbox-builds/mozilla-central-win32-debug directory on the Talos slave.

(In reply to comment #24) > Is the Debug.CRT stuff redistributable? No, it's explicitly not redistributable, unfortunately.

(In reply to comment #25) > Here's what I did: I forgot to mention, steps 1 and 2 were done on a build slave machine (but could be done on any machine with Visual C++ 2005 installed).

These are real great news. ted thanks a lot for your help over here :) I would have wasted a lot of time until I figured this out.

This worked on staging-opsi.

Filed bug 592806 to track that we can't run unit tests on XP slaves. Putting on the side until it gets resolved.

For reference: http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/Microsoft_VC80_DebugCRT_x86.msi ab9907210c0f95733cb8465b4dfa28d8 Microsoft_VC80_DebugCRT_x86.msi

Can you also post instructions for getting the correct version of the debug crt on developer machines? I need to use Tinderbox builds locally in order to develop code for fuzzing jobs that will eventually run on the test pool.

You should be able to follow the steps in comment 25, assuming you have access to a machine with Visual C++ 2005 installed.

(Can I just install Visual C++ 2005 in addition to the newer version I already have?)

That would also work, yeah. The msi trick described there is much simpler, though, assuming you have access to a machine with VC2005. (Plus once you make the MSI once, you can easily install it on as many machines as you need .)

Armen, can you post your Microsoft_VC80_DebugCRT_x86.msi somewhere where I can get it, perhaps in http://fs/IT/Microsoft/ ?

I have emailed Jesse the redistributable for testing purposes. To note that this has not been deployed to our testing machines and nothing has happened since the beginning of September. Hopefully I will get back to this before the end of November.

I would like to get back to this and get it out of the way this following 2-3 weeks.

I am trying to get this done for the Win7 machines (instead of the XP machines) as this will get us unblocked for running debug unit tests on the minis. XP will come later. To summarize what is going on: * To run debug unit tests on the minis we need to run the debug build * To run the debug build we need to install Microsoft_VC80_DebugCRT_x86.msm as mentioned in comment 25 otherwise we get a "sige-by-side" error [1] * To install the Debug CRT through the command line and/or OPSI we need an msi * To install it through the command line we have to run the following (cltbld has admin privileges): > runas /user:cltbld "msiexec /quiet /i Microsoft_VC80_DebugCRT_x86.msi" * To allow that command to succeed *I* believe we have to disable UAC [2] otherwise we can't succeed: > runas /user:cltbld "C:\Windows\System32\cmd.exe /k %windir%\System32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f" * For the previous to work we need to give full control to the cltbld user [3], otherwise, I get denied to change the registry. I am blocked right now. * I don't know how to install the msi without disabling UAC * I can't disable UAC without giving full control to cltbld and that requires using VNC/RemoteDesktop (as I wanted to avoid) For now, will research if we can grant full control through the command line [3] Worst case scenario I will have to deploy this manually on each windows machine. [1] "The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail" http://www.vistax64.com/vista-performance-maintenance/222177-side-side-configuration-incorrect-every-program.html [2] http://www.recipester.org/Recipe:Enable_or_Disable_UAC_From_the_Command_Line_in_Windows_7_37930745 [3] http://www.raymond.cc/blog/archives/2009/03/05/full-control-permission-to-delete-or-edit-restricted-windows-registry/

I am considering deploying the msi manually through VNC in all win7 slaves. A batch at a time.

This has been deployed to the win7 testing slaves: * cd Desktop; wget http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/Microsoft_VC80_DebugCRT_x86.msi # sha1sum ab9907210c0f95733cb8465b4dfa28d8 Microsoft_VC80_DebugCRT_x86.msi * through VNC: * double click the msi * Accept UAC prompt * rm msi These are the slaves that don't have the CRT deployed: * w7-001 loaned to sdwilsh - bug 609050 * w7-012 needs reboot - bug 614821 * w7-024 out of the pool - bug 602312 * w7-040 host non-existant?? - bug 614821 * w7-ref I can't log in into it Documented: https://wiki.mozilla.org/ReferencePlatforms/Test/Win7#Microsoft_VC80_DebugCRT_x86.msi TODO * check-in file into private repos

(In reply to comment #26) > (In reply to comment #24) > > Is the Debug.CRT stuff redistributable? > > No, it's explicitly not redistributable, unfortunately. That really sucks, Tinderbox Debug builds are practically useless. :(

Sorry, you'll have to complain to Microsoft, there's nothing we can do about it. You can install their free compiler and build your own.

(In reply to comment #41) > TODO > * check-in file into private repos Done.

(In reply to comment #41) > # sha1sum ab9907210c0f95733cb8465b4dfa28d8 Microsoft_VC80_DebugCRT_x86.msi This is an MD5 sum. talos-r3-w7-ref is done per steps in comment #41.

Thanks Nick!

Picking this up again.

I have installed the DebugCRT and tried to run various types of unit tests. All of them fail to start with the following message: "xpcshell.exe - The application failed to initialize properly (0xc0150004)." Any ideas on what is going on? I will look deeper into it early next week. I am also looking at this article but I don't know how does it apply if I installed the DebugCRT: http://blogs.msdn.com/b/heaths/archive/2007/02/05/visual-studio-2005-service-pack-1-rollback-breaks-some-applications.aspx

I need real help in here. I don't know what the attached depend walker output should be telling me to look for. These are the steps that I follow: http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1297891792/firefox-4.0b12pre.en-US.win32.zip unzip -o firefox-4.0b12pre.en-US.win32.zip wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1297891792/firefox-4.0b12pre.en-US.win32.tests.zip unzip -o firefox-4.0b12pre.en-US.win32.tests.zip At this point my problems start. I can substitute running the unit tests to just trying bin/xpcshell.exe (Note that I have to add the firefox/ directory to our PATH otherwise you will get can't find xpcom.dll message). There are 3 problems that I have but I have only been able to figure out two of them: Solution to 1st problem ######################## I placed the Microsoft.VC80.DebugCRT folder (from a build machine) into the bin/ folder that comes from unzipping the tests package to avoid the following message: "This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem" Solution to 2nd problem ####################### Once the DebugCRT directory is placed under the bin/ directory I have to put the manifest file under firefox/ (Microsoft.VC80.DebugCRT.manifest) to avoid the following message: "This application failed to initialize properly (0xc0150002)" 3rd problem ########### The previous does not seem sufficient as I get the following: "xpcshell.exe - Application Error" - The application failed to initialize properly (0xc0000034). Click on OK to terminate the application" FTR: python mochitest/runtests.py --appname=firefox/firefox.exe --utility-path=bin --extra-profile-file=bin/plugins --symbols-path=symbols --certificate-path=certs --autorun --close-when-done --console-level=INFO --total-chunks=5 --this-chunk=1 --chunk-by-dir=4 Created dev.builds thread http://groups.google.com/group/mozilla.dev.builds/browse_thread/thread/37032380988633d1#

Armen, if you can get me a Windows XP slave to poke at, I can take a look at this on Monday.

Thanks ted, I have filed bug 634992 to grant you access to our VPN so you can have access to a machine.

I talked to Armen on IRC, but I'll record it here for posterity. I logged into this slave, and installed the Debug CRT MSI that he had left on the Desktop (by double-clicking). I downloaded a build + test package from http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32-debug/1298468457/ , unzipped them both to c:/temp, and was able to successfully run Mochitest. Armen's going to check some things, I'm not sure if I missed something or if he was testing a different scenario.

I have verified what ted says. I don't why the DebugCRT does not install anything (C:\Windows\WinSxS) on the two staging slaves I worked on. I will look next week into how to deploy this through OPSI. thanks ted for your help!

Comment on attachment 470858 [details] [diff] [review] Add VC2005 Debug CRT for r3 XP slaves The Debug CRT got deployed to: http://hg.mozilla.org/build/opsi-package-sources/rev/9126788ed78a This also got documented in here: https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Microsoft_Visual_C.2B.2B_2005_Non-Redistributable_Debug_CRT_.28x86.29 NOTE: I have deployed to the first 20 XP slaves. If find no major problem I will mark the remaining slaves to pick up the change. FTR I am not doing manual reboots of slaves until I figure out what is going on with the new 3 masters coming up.

(In reply to comment #54) > Comment on attachment 470858 [details] [diff] [review] > Add VC2005 Debug CRT for r3 XP slaves > > The Debug CRT got deployed to: > http://hg.mozilla.org/build/opsi-package-sources/rev/9126788ed78a > > This also got documented in here: > https://wiki.mozilla.org/ReferencePlatforms/Test/WinXP#Microsoft_Visual_C.2B.2B_2005_Non-Redistributable_Debug_CRT_.28x86.29 > > NOTE: I have deployed to the first 20 XP slaves. If find no major problem I > will mark the remaining slaves to pick up the change. FTR I am not doing manual > reboots of slaves until I figure out what is going on with the new 3 masters > coming up. buildbot-master4 is out for hardware repair, buildbot-master5 and 6 are coming online now.

I have marked the remaining XP slaves to receive this package.

I can see this being deployed properly under C:\Windows\WinSxS 03/07/2011 01:11 PM <DIR> x86_Microsoft.VC80.DebugCRT_1fc8b3b9a1e18 e3b_8.0.50727.762_x-ww_5490cd9f

cltbld@production-opsi:~/opsi-binaries$ cvs add -kb vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi cvs add: scheduling file `vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi' for addition cvs add: use 'cvs commit' to add this file permanently cltbld@production-opsi:~/opsi-binaries$ cvs commit -m "Bug 562459 - Install Microsoft_VC80_DebugCRT_x86.msi on Rev3 Windows machines. r=coop" RCS file: /mofo/opsi-binaries/vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi,v done Checking in vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi; /mofo/opsi-binaries/vc2005_debugcrt_x86/Microsoft_VC80_DebugCRT_x86.msi,v <-- Microsoft_VC80_DebugCRT_x86.msi initial revision: 1.1 done talos-r3-xp-ref is also updated. Nothing left to be done.

I will assume that this package have not been installed on the tool staging boxes?

Alice the Debug CRT is on fs under IT/Microsoft.

Armen - does this bug being fixed mean we can file a bug to turn on debug tests for w764? http://hg.mozilla.org/build/buildbot-configs/file/3c3ff08406a5/mozilla-tests/config.py#l274 seems to suggest this would be the case. You know the most about this, can you confirm?

This was not deployed to the Win7x64 machines so we should not enable it yet.

(In reply to comment #62) > This was not deployed to the Win7x64 machines so we should not enable it yet. good to know - is there a bug tracking deploying this to win764 machines that we can update that comment with?

No, there isn't. Feel free to file one but I would not bother until we have the win64 build slaves up which is going to take quite a while. You could also change it to 'debug_unittest_suites':[] and get rid of lines 274 & 275.