lint tasks use a workspace cache to hold the source checkout. On Try (level 1 scopes), this cache is removed and we use local Docker container storage for the checkout. This is both slow (because AUFS is slower than host-native ext3/4 volume mount) and buggy (I see intermittent failures in automation when hg does checkouts). We should use a cache on Try / level 1 scopes so these issues aren't present. My original patches in bug 1247168 had this implemented. Then I removed it to make the review simpler and less contentious. Unfortunately, we are seeing intermittent failures due to AUFS, so this needs to be addressed.

https://public-artifacts.taskcluster.net/Wd041B0aQjOoHqgPer_nng/0/public/logs/live_backing.log is an example log showing checkout failure due to aufs bugs. I haven't seen one of these failures when using a host-mounted cache/volume.

Maybe this just means adding a CLOBBER_OBJDIR env var that build.sh treats as an indication it should `rm -rf` the objdir? Then we can set that variable when config.params['project'] == 'try'

Or we can trust Mercurial will reset the checkout to a clean slate and not clobber on Try. If nothing else, this will make Try behave like all the other repos :)

Maybe I'm confused, but I thought clobbering meant deleting the objdir and any stray files in the repo, while I assume mercurial is only cleaning up the repo. I think we would want the latter on all trees, but i assume on non-try trees that we want to take advantage of the objdir, right?

It depends where the "objdir" is. If the objdir is under the source checkout, `hg robustcheckout --purge` will nuke it because it purges *all* untracked and ignored falls. The lint tasks/images don't really have "objdirs." They may leave state under the source checkout. But that state will get nuked by `hg robustcheckout --purge`. The only thing we should potentially save the objdir for are build tasks. And the "objdir" in this case should be running out of a separate directory or cache from the source checkout. This bug doesn't deal with build tasks - just the ones based on the lint image. So let's not worry about "objdirs" here.

Got it, sorry for my confusion

All the other expressions are anchored via ^ and $. Review commit: https://reviewboard.mozilla.org/r/67996/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/67996/

Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. Review commit: https://reviewboard.mozilla.org/r/67998/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/67998/

Previously, the VCS checkout cache would be optimized away on Try. This meant every task on Try had to create its own checkout from scratch. This meant writing ~150,000 files. By allowing use of the cache, we minimize the amount of work tasks need to perform on Try and therefore make tasks complete faster, on average. Review commit: https://reviewboard.mozilla.org/r/68000/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/68000/

Comment on attachment 8776071 [details] Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/1-2/

Comment on attachment 8776072 [details] Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/1-2/

Comment on attachment 8776071 [details] Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/67998/diff/2-3/

Comment on attachment 8776072 [details] Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/68000/diff/2-3/

Comment on attachment 8776070 [details] Bug 1289643 - Add $ to regular expression in hg-shared cache name; https://reviewboard.mozilla.org/r/67996/#review65374

Comment on attachment 8776071 [details] Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; https://reviewboard.mozilla.org/r/67998/#review65376 ::: taskcluster/docs/index.rst:32 (Diff revision 3) > kinds > transforms > yaml-templates > how-tos > docker-images > + caches I feel like the "how tos" should be the last element, to be honest. But if you have a specific reason to put `docker-images` and `caches` in this order that's fine.

Comment on attachment 8776072 [details] Bug 1289643 - Allow level-{{level}}-checkouts cache to be used on Try; https://reviewboard.mozilla.org/r/68000/#review65380

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/0beb6afa58b7 Add $ to regular expression in hg-shared cache name; r=dustin https://hg.mozilla.org/integration/autoland/rev/50b87c38d18d Change path for checkouts from "workspace" to "checkouts"; r=dustin https://hg.mozilla.org/integration/autoland/rev/6ea38e685679 Allow level-{{level}}-checkouts cache to be used on Try; r=dustin

https://hg.mozilla.org/mozilla-central/rev/0beb6afa58b7 https://hg.mozilla.org/mozilla-central/rev/50b87c38d18d https://hg.mozilla.org/mozilla-central/rev/6ea38e685679