Running mach, I suddenly started getting: ... The details of the failure are as follows: ObjdirMismatchException: Objdir mismatch: o:\src\mm\mozilla-hg\mc-socialapi-landing\obj-i686-pc-mingw32 != o:/src/mm/mozilla-hg/mc-socialapi-landing/obj-@CONFIG_GUESS@ ... File "o:\src\mm\mozilla-hg\mc-socialapi-landing\python/mozbuild\mozbuild\base.py", line 163, in from_environment raise ObjdirMismatchException(topobjdir, config['topobjdir']) -- .mozconfig includes the line: mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-@CONFIG_GUESS@ -- output of "mach environment" (from bug 894973) platform: Windows-7-6.1.7601-SP1 python version: 2.7.4 (default, Apr 6 2013, 19:54:46) [MSC v.1500 32 bit (Intel)] python prefix: c:\mozilla-build\python mach cwd: o:\src\mm\mozilla-hg\mc-socialapi-landing os cwd: o:\src\mm\mozilla-hg\mc-socialapi-landing mach directory: o:\src\mm\mozilla-hg\mc-socialapi-landing state directory: o:/home/.mozbuild object directory: o:\src\mm\mozilla-hg\mc-socialapi-landing/obj-i686-pc-mingw32 mozconfig path: o:\src\mm\mozilla-hg\mc-socialapi-landing\.mozconfig mozconfig objdir: o:\src\mm\mozilla-hg\mc-socialapi-landing/obj-@CONFIG_GUESS@ mozconfig configure args: --enable-application=browser --disable-optimize --enable-debug --enable-tests mozconfig extra make args: AUTOCLOBBER=1 mozconfig make flags: - j 1 2 config topsrcdir: o:/src/mm/mozilla-hg/mc-socialapi-landing config topobjdir: o:\src\mm\mozilla-hg\mc-socialapi-landing\obj-i686-pc-mingw32

I guess we need to throw in a normpath somewhere... Until then, you should be able to work around this by using a backslash in your mozconfig or by not defining an objdir in your mozconfig.

(In reply to Gregory Szorc [:gps] from comment #1) > I guess we need to throw in a normpath somewhere... > > Until then, you should be able to work around this by using a backslash in > your mozconfig or by not defining an objdir in your mozconfig. I don't think the problem is the lack of a backslash or a normpath, but instead that we are simply using config['topobjdir'] while it still has the literal '@CONFIG_GUESS@' in it. Simply replacing that with the literal 'i686-pc-mingw32' (and without adding or changing any slashes) works around the problem.

I can reproduce this on OS X as well.

This is really weird, actually, because I'm using an environment variable for my .mozconfig, and am running the command (./mach mochitest-browser foo/bar/baz) from my srcdir, not my objdir.

Adding a print(topobjdir) in base.py before it throws the exception, I'm seeing it invoked once with 'None' and once with the correct normalized path to my objdir. I don't know why it does that.

... but fixing the paths to make them match gets me: gkruitbosch-16516:ux gkruitbosch$ ./mach mochitest-browser browser/components/customizableui/test/ Error running mach: ['mochitest-browser', 'browser/components/customizableui/test/'] The error occurred in the implementation of the invoked mach command. This should never occur and is likely a bug in the implementation of that command. Consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: ImportError: No module named automation File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 280, in run_mochitest_browser return self.run_mochitest(test_file, 'browser', **kwargs) File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 299, in run_mochitest **kwargs) File "/Users/gkruitbosch/dev/ux/testing/mochitest/mach_commands.py", line 85, in run_mochitest_test from automation import Automation (should this be a different bug? I don't know.) :-(

(In reply to :Gijs Kruitbosch from comment #6) > ... but fixing the paths to make them match gets me: > ImportError: No module named automation This was a pebkac, because I mistyped the objdir in my mozconfig, which was therefore empty. The error message could have been better ("You haven't built at this path yet"), but fine. Now that the path is correct, and I can run ./mach mochitest-browser path/to/test, I can no longer run ./mach build path/to/stuff/that/changed, because it errors out with: gkruitbosch-16516:ux gkruitbosch$ ./mach build browser/components/customizableui/ 0:00.19 /usr/bin/make -C rowser/components/customizableui -j8 -s 0:00.20 make: *** rowser/components/customizableui: No such file or directory. Stop. (the 'b' seems to randomly go missing from the path?) Again, not sure if this is related or not.

(In reply to :Gijs Kruitbosch (PTO Aug 8-Aug 18) from comment #7) > (In reply to :Gijs Kruitbosch from comment #6) > > ... but fixing the paths to make them match gets me: > > ImportError: No module named automation > > This was a pebkac, because I mistyped the objdir in my mozconfig, which was > therefore empty. The error message could have been better ("You haven't > built at this path yet"), but fine. > > Now that the path is correct, and I can run ./mach mochitest-browser > path/to/test, I can no longer run ./mach build path/to/stuff/that/changed, > because it errors out with: > > gkruitbosch-16516:ux gkruitbosch$ ./mach build > browser/components/customizableui/ > 0:00.19 /usr/bin/make -C rowser/components/customizableui -j8 -s > 0:00.20 make: *** rowser/components/customizableui: No such file or > directory. Stop. > > (the 'b' seems to randomly go missing from the path?) > > Again, not sure if this is related or not. And this seems unrelated but to do with whether or not the path specified has a trailing slash. Filed bug 902389.

I'm going to look into this today.

This sounded like a trivial fix but the patch bloated up a bit. Essentially the problem was MozbuildObject.from_environment() and MozbuildObject's .topobjdir getter had slightly different code paths for resolving topobjdir from mozconfigs. This patch hopefully fixes that. Before, @CONFIG_GUESS@ just wasn't getting substituted in MozbuildObject.from_environment(). This was causing the topobjdir sanity checking comparison to fail and causing errors for users with @CONFIG_GUESS@ in their mozconfig. We now substitute @CONFIG_GUESS@ in from_environment() by using the same code path as the .topobjdir getter. The included test case failed before the patch and now passes. https://tbpl.mozilla.org/?tree=Try&rev=9f312c918fd9

I had a bad patch in my try push, so https://tbpl.mozilla.org/?tree=Try&rev=a608c8f25785

Comment on attachment 788277 [details] [diff] [review] Unify code paths for load mozconfigs Is this the right patch? You mentioned that the test case passes, but it's failing in the try jobs (as well as when I run it locally).

markh: I was trying to reproduce this on Windows with this mozconfig: mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj2-@CONFIG_GUESS@ However, I don't hit the ObjdirMismatchException (before applying gps' patch). I just ran a './mach build' Can you walk me through the steps to reproduce it?

FWIW, ted pointed me here since I'm hitting the same exception, and I'm not using @CONFIG_GUESS@. I'm simply using: mk_add_options MOZ_OBJDIR=./objdir-droid (This line is given as an example in https://wiki.mozilla.org/Android?title=Mobile/Fennec/Android#Setup_Fennec_mozconfig.) Printing out the absolute paths, I see that: topobjdir is /data/mozilla/central/objdir-droid config['topobjdir'] is /data/mozilla/central/objdir-droid/objdir-droid

BTW, I'm building Fennec. I see this only when trying to run robocop tests (make mochitest-robocop); I do not see this when doing a normal mach build.

Brian: Please paste the contents of your objdir/mozinfo.json file.

(In reply to Gregory Szorc [:gps] from comment #17) > Brian: Please paste the contents of your objdir/mozinfo.json file. {"bin_suffix": "", "tests_enabled": true, "bits": 32, "appname": "fennec", "mozconfig": "/home/brian/Dropbox/mozilla/mozconfig-droid", "topsrcdir": "/data/mozilla/central", "toolkit": "android", "crashreporter": true, "debug": false, "asan": false, "os": "android", "processor": "arm"}

(In reply to Michael Shal [:mshal] from comment #14) > markh: I was trying to reproduce this on Windows with this mozconfig: > > mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj2-@CONFIG_GUESS@ > > However, I don't hit the ObjdirMismatchException (before applying gps' > patch). I just ran a './mach build' 'build' seems to work, but I get it trying to run, eg, './mach mochitest-browser browser/base/content/test/social/'

Brian: I suspect this is a current working directory issue. Can you paste exact steps to reproduce. Start your shell at topsrcdir and include cd's, etc. FWIW, I could probably ask you to file a followup, but I want this bug to be the last time we have to track down the lingering issues with mozconfig loading.

(In reply to Gregory Szorc [:gps] from comment #20) > Brian: I suspect this is a current working directory issue. Can you paste > exact steps to reproduce. Start your shell at topsrcdir and include cd's, > etc. Sure. Using the attached mozconfig: cd /data/mozilla/central ./mach build ./mach package ./mach install export MOZ_HOST_BIN=/data/mozilla/central/objdir-linux/dist/bin/ cd objdir-droid make mochitest-robocop Output: Traceback (most recent call last): File "_tests/testing/mochitest/runtestsremote.py", line 19, in <module> from runtests import Mochitest File "/data/mozilla/central/objdir-droid/_tests/testing/mochitest/runtests.py", line 25, in <module> from mochitest_options import MochitestOptions File "/data/mozilla/central/objdir-droid/_tests/testing/mochitest/mochitest_options.py", line 17, in <module> build_obj = MozbuildObject.from_environment() File "/data/mozilla/central/python/mozbuild/mozbuild/base.py", line 165, in from_environment raise ObjdirMismatchException(topobjdir, config['topobjdir']) mozbuild.base.ObjdirMismatchException: Objdir mismatch: /data/mozilla/central/objdir-droid != ./objdir-droid make: *** [mochitest-robocop] Error 1

Brian: I wrote an explicit test for relative-defined objdirs with cwd inside the objdir and it passes with my patch. Please confirm my patch fixes things for you.

Let's see if this has better results on Try... https://tbpl.mozilla.org/?tree=Try&rev=83293931fd36

(In reply to Gregory Szorc [:gps] from comment #22) > Brian: I wrote an explicit test for relative-defined objdirs with cwd inside > the objdir and it passes with my patch. Please confirm my patch fixes things > for you. I was able to reproduce the ObjdirMismatchException by loosely following bnicholson's instructions. I have: .mozconfig: mk_add_options MOZ_OBJDIR=./objdir-droid Then did: ./mach build ./mach package (not sure if this was necessary) cd objdir-droid python ../build/pymake/make.py mochitest I also confirmed that your latest patch fixes the error (at least, some mochitests start running). Hopefully try is happy this time :)

I got the info I needed to reproduce the error.

Yep, I can also confirm that the patch fixes the issue for me.

Note to self: try @ https://tbpl.mozilla.org/?tree=Try&rev=aaf0eb2895a3

Random thought: if I'm unable to figure out how to get the tests to pass on automation, would reviewers consider skipping the tests for the initial landing? The way I see it, this bug is interfering with the ability of some people to optimally perform work and that takes precedence over test coverage in the short term.

Yes, let's get this landed. Testing these code paths is a PITA.

Let's get this pushed so developers don't have a hard time.

Try was green: https://tbpl.mozilla.org/?tree=Try&rev=3146231953ba

Comment on attachment 790815 [details] [diff] [review] Unify code paths for load mozconfigs Review of attachment 790815 [details] [diff] [review]: ----------------------------------------------------------------- ::: python/mozbuild/mozbuild/base.py @@ +185,5 @@ > + topobjdir = os.path.abspath(os.path.join(topsrcdir, topobjdir)) > + > + if '@CONFIG_GUESS@' in topobjdir: > + topobjdir = topobjdir.replace('@CONFIG_GUESS@', > + MozbuildObject.resolve_config_guess(mozconfig, topsrcdir)) Is doing this test after the abspath test going to screw things up? @CONFIG_GUESS@ at the beginning of the objdir path wouldn't be treated as an abspath right?

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #32) > ::: python/mozbuild/mozbuild/base.py > @@ +185,5 @@ > > + topobjdir = os.path.abspath(os.path.join(topsrcdir, topobjdir)) > > + > > + if '@CONFIG_GUESS@' in topobjdir: > > + topobjdir = topobjdir.replace('@CONFIG_GUESS@', > > + MozbuildObject.resolve_config_guess(mozconfig, topsrcdir)) > > Is doing this test after the abspath test going to screw things up? > @CONFIG_GUESS@ at the beginning of the objdir path wouldn't be treated as an > abspath right? Good catch. I moved the abspath after the .replace() and verified all tests still pass.

https://hg.mozilla.org/integration/fx-team/rev/734f0dc3958e Inbound was closed. And since this issue impacts mochitest running, I figured fx-team was as good of an alternative as there is.

BTW: So for now comm-central users (Thunderbird, SeaMonkey) are left a bit in the rain regarding running tests? I know that some people are working on really integrating comm-central with mozilla-central, but I guess this might still take a while. As an example for what does not work atm: "TEST_PATH=suite/common/tests/browser/ pymake mochitest-browser-chrome [...] mozbuild.base.ObjdirMismatchException: Objdir mismatch: c:\mozilla\tree-hg\objdirs\seamonkey-objdir\mozilla != c:\mozilla\tree-hg\comm-central\objdirs\seamonkey-objdir" when executing that command in the objdir or in objdir/mozilla (does not matter). Hm, I guess I should file a follow-up bug on that... :)

Uh-huh, I just noticed the config_topobjdir path is completely wrong (I built with the patch in this bug here). I'll build a Firefox build for now and report back, I use a hardlink for the objdirs on my HDD. Maybe that matters, too.