This bug is for getting chrome supported in raptor-browsertime along with adding a single cold pageload "smoke" test that will run on integration.

This patch adds chrome support to raptor-browsertime.

First, it adds fetch tasks to handle multiple chrome versions running in production. These fetch tasks provide a tar that the chrome browsertime use to get a chromedriver compatible wiht the chrome version available on the testing machine.

In raptor, to handle the version differences, a get_browser_meta function is added which returns the name and version of the browser being tested. This version is then used in the variable passed in through --browsertime-chromedriver (only when running not locally, or if we find a %s in the string).

Finally, chrome is missing some results in browsertime (relative to firefox) and at least one of the other results is scattered into a different location. The results.py and output.py changes handle this issue.

Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=62963f85e521ef071f444c35520f9ce691184f0a

Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=465f80b93df509c7ab1c4c7978e4e5f69d5eff31

Having some issues dealing with the path formatter on windows. This one should solve the issue.

This is the first part of a patch that adds chrome support to raptor-browsertime.

In this patch, a get_browser_meta function is added which returns the name and version of the browser being tested. This version will then be used (in part 3) in the variable passed in through --browsertime-chromedriver (only when running not locally, or if we find {} in the string).

This is the second part of a patch that adds chrome support to raptor-browsertime.

This part of the patch adds fetch tasks to handle multiple chrome versions running in production. These fetch tasks provide a tar that the chrome browsertime can use to get a chromedriver compatible with the chrome version available on the testing machine.

Depends on D48895

This is the third, and final, part of a patch that adds chrome support to raptor-browsertime.

In this part, all the changes from the previous 2 parts are integrated in raptor and browsertime. The main change is that the browsertime-chromedriver paths created in taskcluster's tests.py are changed to include a {} for inserting the chrome version that is used in production (formatting is done within raptor)

Finally, chrome is missing some results in browsertime (relative to firefox) and at least one of the other results is scattered into a different location. The results.py and output.py changes handle this issue.

Depends on D48897

New try runs after the split. This one has the browsertime tests: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=d520476a0388a0384bac0aabd0dfeb050184d40e

This one has the chromedriver fetch tasks (since they weren't run in the other try run): https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0f7d2ea3f9ca0a1e6750aca50d996fed4b04e3d8

Thanks for blocking!

You can pull in the patches in that bug locally if you don't want to wait (using moz-phab patch or arc patch). Let me know if you need help solving the conflicts.

Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f4fd6bdf922f00fba33e9e147ce3d127171e001b

Try run for the latest patch which includes chromedriver v78: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=c6bfb94249900df6895cac4903cedb32eb965de5

There are also new errors caused by bug 1576244 resulting in permafails on linux, and intermittents on windows.

Try run for the latest patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fec19f55a58400ca8741b445c285fbefb83959c

Newest try run, removed visual-metrics from chrome tasks: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f99f2bb2f04b2fba1fae188c35286354f4b1066c

New try run, the last patch didn't disable it completely for chrome: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8efbdbd3d9dffa30e94ed5c4bfa06af5d30c26ee

Last one broke, this one should be good now: https://treeherder.mozilla.org/#/jobs?repo=try&revision=562d549d250b5142fa381c2669e784103c47cceb

https://hg.mozilla.org/mozilla-central/rev/0fa8451286db
https://hg.mozilla.org/mozilla-central/rev/1e209c909ea7
https://hg.mozilla.org/mozilla-central/rev/96b0cb9268b9

:glandium, here's a more detailed description of what we did here - I hope it provides more clarity on this:

We want to always be able to test and compare firefox against the latest chrome version. So chrome was installed on the hardware machines and set to auto-update itself so we won't have to deal with this and we would always test against the latest chrome version. For one of our performance frameworks (raptor) this works fine, but browsertime (our newer framework) requires chromedriver to run tests on chrome.

Unfortunately, chrome doesn't update itself on the machines at the same time, so we end up testing against multiple versions of chrome when these updates are occurring. For raptor, this is fine and doesn't cause any issues, but for browsertime it does because it requires a chromedriver and chromedriver's are only compatible with a single chrome version. So to prevent losing a lot of data intermittently we have to make sure that we have multiple chromedriver's available for browsertime so that we can still run the tests on the older versions if they haven't updated yet.

To make the multiple chromedriver's available, we had to add multiple fetches to the browsertime tasks. But since the file that is extracted from the chromedriver archives has the same name we couldn't tell which chromedriver file goes with which chrome version. Because of that, we had to either use (i) a fetch task like chromium to change the file names, or (ii) use the add-prefix solution. In this case, after discussing with :tomprince and :ahal, we found that the add-prefix was the ideal solution for this, which is what we implemented.

Why not install the latest version of chrome consistently via a fetch?

Primarily because chrome release doesn't provide a simple enough installation method (like chromium does). See the discussion here (and the dependent bugs) for why we went with this installation method: https://bugzilla.mozilla.org/show_bug.cgi?id=1528731