We're going to start executing more mozharness scripts from a source checkout. Rather than add config options to specify the location of a source checkout - something that must be added to every mozharness invocation - we teach BaseScript.__init__ to recognize when we're running from a source checkout and set self.topsrcdir accordingly. This will allow any script or class to check for self.topsrcdir and change behavior accordingly. Review commit: https://reviewboard.mozilla.org/r/79858/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79858/

Now that we can detect when we're running from a source checkout, we can start using things from source checkouts instead of relying on host machine state or grabbing files from another server. We start by using the vendored tooltool.py if available. This avoids non-determinism. It avoids a possible 3rd party hosting dependency on github.com. It avoids a possible MitM attack vector. Wins all around. Review commit: https://reviewboard.mozilla.org/r/79860/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79860/

If mozharness is running from a source checkout, it has access to a modern virtualenv+pip/setuptools vendored as part of the source checkout. This commit changes the virtualenv creation code to use the vendored virtualenv when it is available. A side effect of this change is that a modern version of pip will now be used by mozharness when a source checkout is available. This has a number of consequences. First, modern versions of pip automatically create and cache wheels when building packages. This should make automation faster since it can now reuse cached wheels instead of having to download and rebuild packages all the time. Second, modern versions of pip support pinning package hashes. This opens the door to use having more secure package downloads and more determinism in our test environment. Third, modern versions of pip require connections to package servers be secure by default. Plaintext connections are disallowed by default. A --trusted-host option or environment variable can be used to override this behavior. Since upgrading pip resulted in some jobs failing due to disallowed connections to insecure servers, I've added code to sniff the pip version and add --trusted-host where it is needed/supported. This retains the existing behavior. This is insecure. But fixing that is for another bug. Review commit: https://reviewboard.mozilla.org/r/79862/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79862/

The Python code is now intelligent enough to add this flag on the command line if supported. Eliminate the copy pasta and help prevent cargo culting. Review commit: https://reviewboard.mozilla.org/r/79882/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79882/

Not sure why we support this. The code goes all the way back to the import of mozharness 0.4 into the old mozharness Mercurial repo in bug 651974. Having fewer variations makes it easier to search for usage. So nuke the variation. Review commit: https://reviewboard.mozilla.org/r/79884/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79884/

We don't need a variable to hold the result. Just use return. The "virtualenv_path" option has a default value, so it should always be set. Add code confirming that. And refactor the code to use less indentation. And remove a branch that can never occur since the virtualenv path is guaranteed to be defined. Review commit: https://reviewboard.mozilla.org/r/79886/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79886/

For mozharness scripts running out of a source checkout, we now use the virtualenv vendored in the source checkout. So there's no need to define the path to virtualenv.py in mozharness configs. There are probably more mozharness configs that could have the virtualenv path removed. These are the ones that didn't require me think very hard. Review commit: https://reviewboard.mozilla.org/r/79888/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/79888/

Comment on attachment 8793049 [details] Bug 1304176 - Set BaseScript.topsrcdir if we have a source checkout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79858/diff/1-2/

Comment on attachment 8793050 [details] Bug 1304176 - Use vendored tooltool.py if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79860/diff/1-2/

Comment on attachment 8793051 [details] Bug 1304176 - Use vendored virtualenv if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79862/diff/1-2/

Comment on attachment 8793109 [details] Bug 1304176 - Remove PIP_TRUSTED_HOST and trust-host pip.conf option; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79882/diff/1-2/

Comment on attachment 8793110 [details] Bug 1304176 - Remove --venv-path as an alias to --virtualenv-path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79884/diff/1-2/

Comment on attachment 8793111 [details] Bug 1304176 - Document and refactor query_virtualenv_path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79886/diff/1-2/

Comment on attachment 8793051 [details] Bug 1304176 - Use vendored virtualenv if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79862/diff/2-3/

Comment on attachment 8793109 [details] Bug 1304176 - Remove PIP_TRUSTED_HOST and trust-host pip.conf option; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79882/diff/2-3/

Comment on attachment 8793110 [details] Bug 1304176 - Remove --venv-path as an alias to --virtualenv-path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79884/diff/2-3/

Comment on attachment 8793111 [details] Bug 1304176 - Document and refactor query_virtualenv_path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79886/diff/2-3/

Nearly every job installing things from pip defines a custom find_links config option listing the same 2 URLs to Mozilla's pip mirrors. There are a few problems with this. First, it is excessive typing and config noise. If we always have a config, it should be the default. Second, recent changes to use modern pip and always define --trusted-host when needed cause a regression under certain circumstances. When the private pypi host is not available, pip wastes several seconds attempting to connect and reconnect to it before moving on to the public host. This caused pip operations to slow down. As part of moving the links URLs to the Python, we put the public server first. Review commit: https://reviewboard.mozilla.org/r/80158/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/80158/

Comment on attachment 8793049 [details] Bug 1304176 - Set BaseScript.topsrcdir if we have a source checkout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79858/diff/2-3/

Comment on attachment 8793050 [details] Bug 1304176 - Use vendored tooltool.py if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79860/diff/2-3/

Comment on attachment 8793051 [details] Bug 1304176 - Use vendored virtualenv if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79862/diff/3-4/

Comment on attachment 8793109 [details] Bug 1304176 - Remove PIP_TRUSTED_HOST and trust-host pip.conf option; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79882/diff/3-4/

Comment on attachment 8793110 [details] Bug 1304176 - Remove --venv-path as an alias to --virtualenv-path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79884/diff/3-4/

Comment on attachment 8793111 [details] Bug 1304176 - Document and refactor query_virtualenv_path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79886/diff/3-4/

Automation spends time downloading psutil and mozsystemmonitor from an external host. These packages are vendored in the source repo. This commit changes mozharness to use the vendored copy if available. Review commit: https://reviewboard.mozilla.org/r/80176/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/80176/

Comment on attachment 8793427 [details] Bug 1304176 - Move pip find-links defaults out of mozharness configs; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/80158/diff/1-2/

Comment on attachment 8793049 [details] Bug 1304176 - Set BaseScript.topsrcdir if we have a source checkout; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79858/diff/3-4/

Comment on attachment 8793050 [details] Bug 1304176 - Use vendored tooltool.py if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79860/diff/3-4/

Comment on attachment 8793051 [details] Bug 1304176 - Use vendored virtualenv if available; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79862/diff/4-5/

Comment on attachment 8793109 [details] Bug 1304176 - Remove PIP_TRUSTED_HOST and trust-host pip.conf option; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79882/diff/4-5/

Comment on attachment 8793110 [details] Bug 1304176 - Remove --venv-path as an alias to --virtualenv-path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79884/diff/4-5/

Comment on attachment 8793111 [details] Bug 1304176 - Document and refactor query_virtualenv_path; Review request updated; see interdiff: https://reviewboard.mozilla.org/r/79886/diff/4-5/