User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Firefox/78.0 Waterfox/78.7.0

Steps to reproduce:

Upgraded to the latest ESR tip (cecd979ce5ff9d66d6b61f3534b80164065e7f99) from ESR 78.7 tag.

Actual results:

Build throws error when building immediately after pull with:

 0:00.80 /Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin/_virtualenvs/init_py3/bin/python /Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin/config.status
Traceback (most recent call last):
  File "/Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin/config.status", line 843, in <module>
    from mozbuild.util import patch_main
ModuleNotFoundError: No module named 'mozbuild'
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin/_virtualenvs/init_py3/bin/python', '/Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin/config.status']

  File "/Users/akontos/Workspace/Firefox/python/mozbuild/mozbuild/build_commands.py", line 130, in build
    return driver.build(
  File "/Users/akontos/Workspace/Firefox/python/mozbuild/mozbuild/controller/building.py", line 1103, in build
    self.run_process(args, cwd=self.topobjdir, pass_thru=True)
  File "/Users/akontos/Workspace/Firefox/python/mach/mach/mixin/process.py", line 159, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

After a clobber, the error is:

Configure complete!
Be sure to run |mach build| to pick up any changes
 0:42.71 Clobber not needed.
 0:42.71 Adding make options from /Users/akontos/Workspace/Firefox/.mozconfig
    AUTOCLOBBER=1
    MOZ_MAKE_FLAGS=-j18
    MOZ_OBJDIR=/Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin
    OBJDIR=/Users/akontos/Workspace/Firefox/obj-x86_64-apple-darwin
    FOUND_MOZCONFIG=/Users/akontos/Workspace/Firefox/.mozconfig
    export FOUND_MOZCONFIG
 0:42.72 /usr/bin/make -f client.mk MOZ_PARALLEL_BUILD=18 -s
 0:42.82 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.preprocessor' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.83 make[2]: *** [.cargo/config] Error 1
 0:42.83 make[2]: *** Waiting for unfinished jobs....
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 /Applications/Xcode.app/Contents/Developer/usr/bin/python3: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
 0:42.84 make[3]: *** [install-dist/include] Error 1
 0:42.84 make[3]: *** Waiting for unfinished jobs....
 0:42.84 make[3]: *** [install-dist/xpi-stage] Error 1
 0:42.84 make[3]: *** [install-dist/public] Error 1
 0:42.84 make[3]: *** [install-dist/private] Error 1
 0:42.85 make[3]: *** [install-dist/bin] Error 1
 0:42.85 make[3]: *** [install-_tests] Error 1
 0:42.85 make[2]: *** [pre-export] Error 2
 0:42.85 make[1]: *** [default] Error 2
 0:42.85 make: *** [build] Error 2

Expected results:

Build should have run without issue.

We recently uplifted a virtualenv upgrade to ESR78 that bumped virtualenv from 16.7.8 to 20.0.31. As noted in the virtualenv changelog, they rewrote it from scratch for version 20.

Related to this, Xcode Python on Mac has some special behaviour: if the magic environment variable __PYVENV_LAUNCHER__ is set, then the python executable it is referring to will be used instead of the proposed python.
So, for your failure here, we're supposed to $objdir/_virtualenvs/.../python3, but that gets replaced with __PYVENV_LAUNCHER__'s value of /Applications/Xcode.app/.../python3.

My educated guess here is that virtualenv@16.7.8 would honour __PYVENV_LAUNCHER__, but would still do local site customization (adding .pth links, running sitecustomize.py, etc).
Meanwhile, virtualenv@20.0.31 seems to not want to do that local site customization, which makes sense to me - if our requested python executable is being ignored, then why would its site customization still be run?

Anyways, the solution is to pummel __PYVENV_LAUNCHER__ a little more than we have previously.
We already did so over here, so let's uplift that. I'll make the request a little later today.

Uplift requested over here.
(Also FTR this issue can be worked around by using Python from brew/Macports instead of the Xcode python).

Patch has been uplifted, please re-open this ticket if you run into this again after updating to newest ESR78!