https://tbpl.mozilla.org/?tree=Cedar&rev=2fafa4415c2e One linux build failed at: /bin/sh: make: command not found A Linux 64 build failed with: /builds/slave/ced-l64-0000000000000000000000/build/source/client.mk:316: *** Could not find autoconf 2.13. Stop.

Oh, and the code seems to build locally under mach.

I would not be surprised if the machines were not able to get to tooltool (where a lot of those tools that we download from it). Unfortunately, the logs do not say that it failed. I've triggered a 3rd Linux opt to see if it repeats.

That retrigger seems to have failed in the same way.

I think we might have been running mozharness fx desktop on Cedar and we have now been regressed. jlund: would you mind helping us disable fx desktop mozharness? Thanks!

yes it seems like things have bit-rotted. /me adds this to buildduty queue. I'll resolve by EOD

it turns out this wasn't bit rot. This mozharness patch broke it: http://hg.mozilla.org/build/mozharness/rev/b2069bb5736e Basically I was calling run_command_m() directly instead of using enable_mock() and disable_mock() in between run_command() creating a patch now to fix.

see https://bugzilla.mozilla.org/show_bug.cgi?id=1026468#c6 for the reason of this patch. implementation details: the paradigm for b2g builds usually gets us to enable_mock() and disable_mock() once or twice for a script run and most jobs use mock. for desktop builds, we go back and forth a lot between using and not using mock and many of our platforms don't use it at all. For those reasons, I am creating a _possibly_run_command_with_mock() method so I do not repeat myself and I am explicit about what I am doing.

Comment on attachment 8441857 [details] [diff] [review] 140617_bug_1026468_cedar_linux_builds_stopped_using_mock.patch Review of attachment 8441857 [details] [diff] [review]: ----------------------------------------------------------------- r=me for now, but I think we'll want to merge buildbase.py and buildb2gbase.py (unlanded; part of bug 898554), and that uses a separate config (b2g_build.py passes in a gecko_config) to choose the mock config (including the mock_target). But I think that just means we'll want to look at this again and figure out how to make it all work as cleanly as possible. Oh. Another way that duplicating this logic will cause trouble for the merge -- buildb2gbase.py has a --disable-mock flag that I know you also wanted here, and checks it during enable_mock. That actually wouldn't break with this patch because (1) they're separate inheritance hierarchies atm anyway, so you'd never call both; and (2) _possibly... would just let the call go through to enable_mock, which would do nothing. Anyway, good enough for now. Sorry for the bustage.

this didn't land. we opted to backout original cause of bustage for now and come up with a more robust solution later: https://bugzilla.mozilla.org/show_bug.cgi?id=1027146#c8 backout is live in prod and this should be resolved. Let's re-trigger some linux jobs to confirm before resolving. That is after the tree closure lifts :)