Bisected to https://hg.mozilla.org/mozilla-central/rev/a31a95d6a390d1ba471fe4ab0830edec689ade3d bug 1681560

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

The changeset backs out cleanly and I can get a working build with that, in case anyone else comes into this problem.

Set release status flags based on info from the regressing bug 1681560

Mitch is looking at this along with other MacOS bustages around zstd/Big Sur. Seems the issue is probably unrelated to the SHA pinning.

For the record, I was manually able to install using something like

/Users/nalexander/.mozbuild/_virtualenvs/mach/bin/pip install --requirement /Users/nalexander/Mozilla/gecko/build/zstandard_requirements.txt --require-hashes

I think the issue has to do with setup.py and the clean step; it makes things build from source, which fails, instead of using a cached wheel, which succeeds.

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

For the record, I was manually able to install using something like

/Users/nalexander/.mozbuild/_virtualenvs/mach/bin/pip install --requirement /Users/nalexander/Mozilla/gecko/build/zstandard_requirements.txt --require-hashes

I think the issue has to do with setup.py and the clean step; it makes things build from source, which fails, instead of using a cached wheel, which succeeds.

Correction: I manually installed with /Users/nalexander/.mozbuild/_virtualenvs/mach/bin/pip install zstandard==0.13.0, which doesn't enforce hashes and therefore uses the previously-built cached wheel.

Mitch suggests the workaround of brew uninstall zstd --ignore-dependencies, then running mach create-mach-environment -f (for force), then reinstalling brew install zstd. That works for me.

(In reply to Connor Sheehan [:sheehan] from comment #5)

Mitch is looking at this along with other MacOS bustages around zstd/Big Sur. Seems the issue is probably unrelated to the SHA pinning.

Okay thanks. Note that I'm on 10.15.7, so it's not just a Big Sur issue.

After a good chat with Nick, I think I've got a good handle on this bug - Timothy, you have zstd installed via brew, right?
The fix here is to make sure that python-zstandard prioritizes its own native files before /usr/local/include, I'll dig into that.
I bet that this is caused by bug 1681560 because you have a local zstandard wheel that was being used, but once expected hashes were set, the wheel that was built locally was no longer used.

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #10)

Timothy, you have zstd installed via brew, right?

Yeah, I think so.

Issue reported upstream.

Is the sha256 for the wheel that is installed for big sur missing from zstandard_requirements.txt?

zstandard-0.13.0 only has wheels up to Python 3.8. brew's Python is 3.9.
Devs affected by this have a wheel locally, but it was either built:

• Before they installed zstd via brew.
• Back when zstd with brew matched the version of zstd vendored with the python-zstandard.

The sha256 is triggering this issue because that locally-built wheel doesn't have a hash in our list, so it gets skipped.

I got same zstandard failure but I'm on Ubuntu 20.04, here is the failure.

Collecting zstandard==0.13.0
  Using cached zstandard-0.13.0.tar.gz (659 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/alwu/.mozbuild/_virtualenvs/mach/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jwj03b7k/zstandard_b47bd0d6ea124ebbb583fa4048450561/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jwj03b7k/zstandard_b47bd0d6ea124ebbb583fa4048450561/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-px2orf8z
         cwd: /tmp/pip-install-jwj03b7k/zstandard_b47bd0d6ea124ebbb583fa4048450561/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-jwj03b7k/zstandard_b47bd0d6ea124ebbb583fa4048450561/setup.py", line 74, in <module>
        import make_cffi
      File "/tmp/pip-install-jwj03b7k/zstandard_b47bd0d6ea124ebbb583fa4048450561/make_cffi.py", line 170, in <module>
        ffi = cffi.FFI()
      File "/home/alwu/.local/lib/python3.6/site-packages/cffi/api.py", line 54, in __init__
        raise Exception("Version mismatch: this is the 'cffi' package version %s, located in %r.  When we import the top-level '_cffi_backend' extension module, we get version %s, located in %r.  The two versions should be equal; check your installation." % (
    Exception: Version mismatch: this is the 'cffi' package version 1.14.3, located in '/home/alwu/.local/lib/python3.6/site-packages/cffi/api.py'.  When we import the top-level '_cffi_backend' extension module, we get version 1.14.4, located in '/home/alwu/.mozbuild/_virtualenvs/mach/lib/python3.9/site-packages/_cffi_backend.cpython-39-x86_64-linux-gnu.so'.  The two versions should be equal; check your installation.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Error running mach:

    ['create-mach-environment']

@Alstor, I believe the issue you're running into is different. This ticket is about zstandard failing to compile due to mismatched C headers, while your stack trace shows a strange package collision. Would you mind creating a new ticket for me?

Just dropping by to add a note to the workaround provided in comment 8 with an issue I just ran into. If you see errors like

/usr/local/include/xxhash.h:615:8: error: redefinition of 'XXH32_state_s'

then make sure to also uninstall xxhash via brew, and reinstall after the bootstrap command ran. Since both zstd and xxhash are dependencies of rsync, this is probably not an uncommon scenario. :)

Upstream brew patch to handle this is here, crossing my fingers that it'll be accepted.

Note: a faster workaround than brew uninstall --ignore-dependencies zstd/xxhash is to do brew unlink zstd/xxhash. After the bootstrap, you can brew link zstd/xxhash.

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #18)

Upstream brew patch to handle this is here, crossing my fingers that it'll be accepted.

Note: a faster workaround than brew uninstall --ignore-dependencies zstd/xxhash is to do brew unlink zstd/xxhash. After the bootstrap, you can brew link zstd/xxhash.

In case this is helpful to anyone else: I tried brew unlink zstd on another machine (never tried it on my primary machine where I'm sticking with backing out locally the regression changeset) and it does not seem to fix the problem there. bootstraping results in what looks like the same error.

Actually, brew uninstall zstd and backing out the regressing changeset (and one on top of it that touched the same file) doesn't seem to fix the problem on that machine. Not sure what is going on, but I'm giving up on that machine for the time being. I'll come back to it though.

bootstraping results in what looks like the same error.

Would you mind providing the error log here to confirm? There's been a few related bootstrap failures with distinct causes.

Comparing the two error logs it looks like it might be a different compile failure?

:tnikkel that looks like an unrelated issue:

ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd' for architecture x86_64

Looking at some other reports of this situation, you can try:

• Updating your ld/clang (I'm guessing they're installed via brew, so maybe do a brew update && brew upgrade and try again?)
• Update your CommandLineTools (xcode-select --install, I believe? You might need to uninstall your old CLT first)
• Are you running Catalina? Perhaps the most up-to-date CLT are too new, and you need to downgrade.

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #24)

:tnikkel that looks like an unrelated issue:

ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd' for architecture x86_64

Looking at some other reports of this situation, you can try:

• Updating your ld/clang (I'm guessing they're installed via brew, so maybe do a brew update && brew upgrade and try again?)
• Update your CommandLineTools (xcode-select --install, I believe? You might need to uninstall your old CLT first)
• Are you running Catalina? Perhaps the most up-to-date CLT are too new, and you need to downgrade.

Thanks. It's probably the last one. An earlier mach bootstrap run that day told me I needed to run 'xcode-select --install' to get the command line tools, even though those have been installed on that machine for a long time. So I just did what mach bootstrap told me and it installed something. So something got confused and told me to update to too new command line tools for my OS version? I'll try downgrading CLT.

Yeah, it was that CLT are too new.

Where does your ld come from that CLT is too new for it, considering CLT normally provides ld?

Worked with glandium we determined that it wasn't CLT being too new that was the problem, but that my Xcode was too old. It seems I must have installed it a long time ago via a method other than the app store, so that it wasn't getting updated.

:tnikkel that's good news! We have a minimum and maximum compatible SDK version, but we only check that at build-/configure-time, but not during bootstrap (Perhaps we should? 🤔 That's a problem for another day). Nice work getting it fixed 👍

Related: this bug is resolved because upstream brew has accepted our patch to avoid python module <-> brew module conflicts.