When I update the emulator to the latest version, 29.0.11, mkdir intermittently fails, causing task retries.

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=6d78db647d54490642042b45d0d720bd4cfd9804

https://taskcluster-artifacts.net/eL5zCYX1RXyfIXn3i2qODA/0/public/logs/live_backing.log

[task 2019-06-04T14:09:43.664Z] 14:09:43     INFO -  Traceback (most recent call last):
[task 2019-06-04T14:09:43.664Z] 14:09:43     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 401, in <module>
[task 2019-06-04T14:09:43.664Z] 14:09:43     INFO -      sys.exit(main())
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 397, in main
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -      return run_test_harness(parser, options)
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 359, in run_test_harness
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -      mochitest = MochiRemote(options)
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 90, in __init__
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -      self.device.mkdir(self.remoteChromeTestDir)
[task 2019-06-04T14:09:43.665Z] 14:09:43     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2149, in mkdir
[task 2019-06-04T14:09:43.665Z] 14:09:43 CRITICAL -      raise ADBError('mkdir %s Failed' % path)
[task 2019-06-04T14:09:43.665Z] 14:09:43 CRITICAL -  mozdevice.adb.ADBError: mkdir /sdcard/tests/chrome Failed

https://taskcluster-artifacts.net/VjbLtgf7QFW8qxsgoH1cnw/0/public/logs/live_backing.log

[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -  Traceback (most recent call last):
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 401, in <module>
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -      sys.exit(main())
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 397, in main
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -      return run_test_harness(parser, options)
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 359, in run_test_harness
[task 2019-06-04T14:04:21.165Z] 14:04:21     INFO -      mochitest = MochiRemote(options)
[task 2019-06-04T14:04:21.166Z] 14:04:21     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 54, in __init__
[task 2019-06-04T14:04:21.166Z] 14:04:21     INFO -      self.device.mkdir(logParent)
[task 2019-06-04T14:04:21.166Z] 14:04:21     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2149, in mkdir
[task 2019-06-04T14:04:21.166Z] 14:04:21 CRITICAL -      raise ADBError('mkdir %s Failed' % path)
[task 2019-06-04T14:04:21.166Z] 14:04:21 CRITICAL -  mozdevice.adb.ADBError: mkdir /sdcard/tests/logs Failed

https://taskcluster-artifacts.net/BbQ4q3O8RNa6H1xGQC64KQ/0/public/logs/live_backing.log

[task 2019-06-04T02:07:32.950Z] 02:07:32     INFO -  Traceback (most recent call last):
[task 2019-06-04T02:07:32.951Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py", line 186, in run
[task 2019-06-04T02:07:32.951Z] 02:07:32     INFO -      self.run_test()
[task 2019-06-04T02:07:32.951Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py", line 795, in run_test
[task 2019-06-04T02:07:32.951Z] 02:07:32     INFO -      if self.checkForCrashes(self.tempDir, self.symbolsPath, test_name=name):
[task 2019-06-04T02:07:32.951Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/remotexpcshelltests.py", line 181, in checkForCrashes
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      self.initDir(self.remoteMinidumpDir)
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/remotexpcshelltests.py", line 43, in initDir
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      self.device.mkdir(path, parents=True, timeout=timeout, root=root)
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 2140, in mkdir
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      timeout=timeout, root=root)
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 1479, in shell_output
[task 2019-06-04T02:07:32.952Z] 02:07:32 CRITICAL -      raise ADBProcessError(adb_process)
[task 2019-06-04T02:07:32.952Z] 02:07:32 CRITICAL -  ADBProcessError: args: adb wait-for-device shell mkdir /sdcard/tests; echo adb_returncode=$?, exitcode: 1, stdout: mkdir: '/sdcard/tests': File exists
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -  Traceback (most recent call last):
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/remotexpcshelltests.py", line 554, in <module>
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      main()
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/remotexpcshelltests.py", line 549, in main
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      mobileArgs=xpcsh.mobileArgs):
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py", line 1439, in runTests
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      mobileArgs, **kwargs)
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -    File "/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py", line 1635, in runTestList
[task 2019-06-04T02:07:32.952Z] 02:07:32     INFO -      raise exceptions[0]
[task 2019-06-04T02:07:32.952Z] 02:07:32 CRITICAL -  mozdevice.adb.ADBProcessError: args: adb wait-for-device shell mkdir /sdcard/tests; echo adb_returncode=$?, exitcode: 1, stdout: mkdir: '/sdcard/tests': File exists

:bc - Thoughts?

I can reproduce this locally, with an x86-64 build, the x86-7.0 avd, and the 29.0.11 emulator: mach xpcshell-test fails this way, eventually, at least on some runs.

I wasn't able to reproduce with an x86_64 build with the x86-7.0 avd and the 29.0.11 emulator on Fedora 30 using xpcshell. I tried multiple times. I'll try again this weekend with xpcshell and mochitest again and report back.

Most of the mkdir failures are in xpcshell tests. I think that is because xpcshell tests call mkdir a lot - a few times for each test. The majority of those xpcshell mkdir calls are from initDir(), which calls mkdir with parents=True. I noticed that parents=True is not necessary in that case, and with parents=False, the failures become more consistent: individual mkdir calls are still intermittent, but frequent enough to turn xpcshell tasks perma-fail:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=4516d1020b0f619568e31b941e644806ccc2a0bb

Notice that the ADBError is raised if is_dir() is false after the mkdir. If we retry the is_dir() up to 10 times over 10 seconds, almost all such failures are avoided:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0f25ed35e00829ee1df76dfd8c72508ea1f654e6

I think that suggests that the mkdir is successful, but is_dir doesn't reliably detect a newly created directory, in the new emulator.

(I tried just retrying once -- there were fewer failures, but still quite a few.)

I think this will work, but I don't want to use this work-around unless
it is necessary, and I am not sure if we will go ahead with the emulator
upgrade...holding off on review for now.

https://hg.mozilla.org/mozilla-central/rev/88737086fe42