jmaher@jmaher-ThinkPad-X230:~/mozilla/mozilla-inbound$ ./mach build 0:00.16 Running |mach artifact install|. 0:00.16 /home/jmaher/mozilla/mozilla-inbound/mach artifact install 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_url-cache.pickle: IOError(2, 'No such file or directory') 0:03.49 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/artifact_tree-cache.pickle: IOError(2, 'No such file or directory') 0:05.85 Installing from remote pushhead e36505acb6f3e7451d0b5a435fa6bf65ed026384 on mozilla-inbound 0:05.85 Installing from https://queue.taskcluster.net/v1/task/XC9Aiu1HTueC8HV03PeIbQ/artifacts/public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip 0:05.85 Ignoring exception unpickling cache file /home/jmaher/.mozbuild/package-frontend/fetch-cache.pickle: IOError(2, 'No such file or directory') 0:05.85 Downloading to temporary location /home/jmaher/.mozbuild/package-frontend/4608132da4558757-public%2Fbuild%2Ffirefox-47.0a1.en-US.linux-x86_64.common.tests.zip Error running mach: ['artifact', 'install'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp' File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 1488, in artifact_install return artifacts.install_from(source, self.distdir) File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 835, in install_from return self.install_from_hg(source, distdir) File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 819, in install_from_hg if self.install_from_url(url, distdir): File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 800, in install_from_url filename = artifact_cache.fetch(url) File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 418, in wrapper result = method(self, *args, **kwargs) File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/artifacts.py", line 615, in fetch dl.wait() File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 97, in wait self.raise_if_error() File "/home/jmaher/mozilla/mozilla-inbound/objdir-frontend/_virtualenv/lib/python2.7/site-packages/mozregression/download_manager.py", line 162, in _download dir=os.path.dirname(dest)) as temp: File "/usr/lib/python2.7/tempfile.py", line 462, in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags) File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner fd = _os.open(file, flags, 0600) Error running mach: ['build'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: Exception: Process executed with non-0 exit code 1: [u'/home/jmaher/mozilla/mozilla-inbound/mach', u'artifact', u'install'] File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 412, in build self._run_mach_artifact_install() File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 645, in _run_mach_artifact_install pass_thru=True, ensure_exit_code=True) File "/home/jmaher/mozilla/mozilla-inbound/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir return self.run_process(cwd=self.topsrcdir, **args) File "/home/jmaher/mozilla/mozilla-inbound/python/mach/mach/mixin/process.py", line 147, in run_process raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

if this helps shed light on file system issues, it seems as though I have the correct paths: ls -la ~/.mozbuild total 48 drwx------ 10 jmaher jmaher 4096 Feb 5 12:42 . drwxr-xr-x 70 jmaher jmaher 12288 Feb 4 12:25 .. drwxrwxr-x 3 jmaher jmaher 4096 Aug 31 10:27 android-device drwxr-xr-x 10 jmaher jmaher 4096 Mar 19 2013 android-ndk-r8e drwxrwxr-x 10 jmaher jmaher 4096 Aug 26 13:30 android-sdk-linux drwxrwxr-x 5 jmaher jmaher 4096 Jan 18 06:17 hgwatchman drwxrwxr-x 3 jmaher jmaher 4096 Jul 28 2014 mercurial drwxrwxr-x 2 jmaher jmaher 4096 Aug 12 15:31 mozboot drwxrwxr-x 2 jmaher jmaher 4096 Feb 5 12:42 package-frontend drwxrwxr-x 14 jmaher jmaher 4096 Jan 28 04:13 version-control-tools jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$ ls -la ~/.mozbuild/package-frontend/ total 20 drwxrwxr-x 2 jmaher jmaher 4096 Feb 5 12:42 . drwx------ 10 jmaher jmaher 4096 Feb 5 12:42 .. -rw-rw-r-- 1 jmaher jmaher 228 Feb 5 12:42 artifact_tree-cache.pickle -rw-rw-r-- 1 jmaher jmaher 403 Feb 5 12:42 artifact_url-cache.pickle -rw-rw-r-- 1 jmaher jmaher 6 Feb 5 12:42 fetch-cache.pickle jmaher@jmaher-ThinkPad-X230:~/mozilla/buildbot-configs$

So, we use mozregression to download artifacts, but the version is pinned. Code is around https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#632. This almost looks like a transient download error not being surfaced. jmaher, is this reproducible?

:parkouss, does the stack in comment 0 ring any bells from the mozregression side?

I got this working on try #3, iam fine closing this as resolved/worksforme. :parkouss there is a needinfo info in comment 3 for you!

Oh! Looks like there is some mozregression code used by mozbuild: https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#72-77 Wow. I wasn't aware of that. Hm, that seems a bit dangerous because mozregression is not in tree (at least as far as I know ?)! Maybe I should take the code of download_manager, and put it in a third party package on pypi, but anyway if it is used in tree it should live in tree no (e.g. a copy in tree) ? To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/ does not exists or is not a directory, reading the error OSError: [Errno 2] No such file or directory: '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp'

(In reply to Julien Pagès (:parkouss) from comment #5) > Oh! Looks like there is some mozregression code used by mozbuild: > > https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/ > artifacts.py#72-77 > > Wow. I wasn't aware of that. Hm, that seems a bit dangerous because > mozregression is not in tree (at least as far as I know ?)! Maybe I should > take the code of download_manager, and put it in a third party package on > pypi, but anyway if it is used in tree it should live in tree no (e.g. a > copy in tree) ? Artifacts pull a published version of mozregression: https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mach_commands.py#1461. > To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/ > does not exists or is not a directory, reading the error > > OSError: [Errno 2] No such file or directory: > '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp' Hmm, it's possible we're not careful about arranging the path to download into. In fact, it looks like we're not! That's a bug. I'll try to look further shortly.

(In reply to Nick Alexander :nalexander from comment #6) > (In reply to Julien Pagès (:parkouss) from comment #5) > > Oh! Looks like there is some mozregression code used by mozbuild: > > > > https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/ > > artifacts.py#72-77 > > > > Wow. I wasn't aware of that. Hm, that seems a bit dangerous because > > mozregression is not in tree (at least as far as I know ?)! Maybe I should > > take the code of download_manager, and put it in a third party package on > > pypi, but anyway if it is used in tree it should live in tree no (e.g. a > > copy in tree) ? > > Artifacts pull a published version of mozregression: > https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/ > mach_commands.py#1461. I just saw! Sorry for the noise :) > > To answer the question, I would say /home/jmaher/.mozbuild/package-frontend/ > > does not exists or is not a directory, reading the error > > > > OSError: [Errno 2] No such file or directory: > > '/home/jmaher/.mozbuild/package-frontend/tmpzCERCP.tmp' > > Hmm, it's possible we're not careful about arranging the path to download > into. In fact, it looks like we're not! That's a bug. I'll try to look > further shortly. Maybe the folder in which the downloads should persist have been removed at some point ? The DownloadManager should create the dirsetory if required (it does on master - I *think* it does in 2.0.1 too): https://github.com/mozilla/mozregression/blob/master/mozregression/download_manager.py#L233).

Oh, I see that you are using 1.0.2 and not 2.0.1 - there is probably not that code, so maybe this is the cause of the issue (download dir not created).

I can reproduce this reliably by removing the destination folder, should be an easy fix.

(In reply to Chris Manchester [:chmanchester] from comment #9) > I can reproduce this reliably by removing the destination folder, should be > an easy fix. Agreed. Just need to `ensureParentDirs` in the right place. Let's throw this one into the mentored bug pond.

Comment on attachment 8718504 [details] [diff] [review] Ensure cache directory exists for artifacts installation Review of attachment 8718504 [details] [diff] [review]: ----------------------------------------------------------------- Thank you for the patch. This is a simple bug, but fixing it is very valuable. Let me know if you have any question about the comment below. ::: python/mozbuild/mozbuild/mach_commands.py @@ +1467,5 @@ > > state_dir = self._mach_context.state_dir > cache_dir = os.path.join(state_dir, 'package-frontend') > > + # Ensure the cache directory exists. Not sure this comment is necessary. @@ +1469,5 @@ > cache_dir = os.path.join(state_dir, 'package-frontend') > > + # Ensure the cache directory exists. > + if not os.path.exists(cache_dir): > + os.makedirs(cache_dir) The preferred way to do this is to attempt creation, catch any OSError, and re-raise for any error condition other than the directory already existing. This is most robust, because it avoids the "check then act" concurrency error. I believe there are some examples of this in the tree.

Thank you for the explanation. I have made the necessary changes.

Comment on attachment 8718544 [details] [diff] [review] Ensure cache directory exists for artifacts installation Review of attachment 8718544 [details] [diff] [review]: ----------------------------------------------------------------- Looks good. Usually things can be landed by setting the "checkin-needed" keyword on the bug, but I have something else to land this evening, and the tree appears open, so I'll go ahead and get this in.

(In reply to Chris Manchester [:chmanchester] from comment #14) > Comment on attachment 8718544 [details] [diff] [review] > Ensure cache directory exists for artifacts installation > > Review of attachment 8718544 [details] [diff] [review]: > ----------------------------------------------------------------- > > Looks good. Usually things can be landed by setting the "checkin-needed" > keyword on the bug, but I have something else to land this evening, and the > tree appears open, so I'll go ahead and get this in. Thanks for the review! I will remember to add the "checkin-needed" keyword to bugs from now onward.