This bug is to update the google-chrome being used in production to the latest version. We are currently using v76 and there is a newer version available (v77): https://support.google.com/chrome/a/answer/7679408?hl=en

Hi, the Chrome version on linux and mac is at v76 while the newest available is v78. Can we get these updated and/or set to auto-update if possible? On windows the chrome version is at v78 and should update itself in the next week.

Auto-update doesn't work the same across all three OSes. Windows is the best at it, with a service that checks regularly. OSX it's disabled by default, but exists. Linux depends on the distro package manager, so requires the most effort. We discussed auto update in bug 1563708#c2 and opted to keep it off and update manually every ~6 weeks.

Ah right, thanks. So we only need a manual update here for linux and mac :)

EDIT: And it can wait until v79 is out next week.

If there is a need to have the same version across all platforms, I can disable the service(s) on Windows, and then have it uninstall and reinstall the needed version on Windows. Then all future updates will need to have a bug filed when there is a desired upgrade, or we can follow the ~6 weeks update interval as mentioned in comment 2.

:markco the 6 week update interval is fine for us, no need to change this.

Also, I want to mention that the new google chrome version (v79) is available now and the linux/macos machines can be updated.

(In reply to Greg Mierzwinski [:sparky] from comment #5)

:markco the 6 week update interval is fine for us, no need to change this.

I just want to confirm before I make any changes. We want stop Chrome from auto-updating on Windows and move to updating the version every 6 weeks?

If this is the case it maybe a few weeks before i have the time to look at instituting those changes.

We don't need to make any changes, everything is fine as it is - sorry for not being clear about that.

Sounds good and no worries.

Hi, I'm wondering when it might be possible to update chrome on the machines? The new versions are available now.

:sparky, which pools do you want to upgrade? Do you want to plan an update to Window, Linux, and MacOS at the same time? What version, and do you want a specific date/time for the switch or can we roll them out when they're ready and notify you for each pool's status?

For the latest stable chrome, I'm seeing 79.0.3945.88
https://chromereleases.googleblog.com/2019/12/stable-channel-update-for-desktop_17.html

Hi :dhouse, we need all pools to have chrome updated to 79 - windows doesn't need to be updated since it auto-updates, linux and mac don't.

Would it be possible to start by updating chrome on 1 or 2 machines from each of the pools so I can run some tests on them (to make sure they will still work in the newer version and primarily because we will be jumping a few versions here)? If these tests show that everything is good, then we would be able to roll them out as they become ready and update chrome on all platforms at the same time.

By "update to window, linux, and macos at the same time" do you mean updating the OS or to have all of those update chrome at the same time?

(In reply to Greg Mierzwinski [:sparky] from comment #11)

Hi :dhouse, we need all pools to have chrome updated to 79 - windows doesn't need to be updated since it auto-updates, linux and mac don't.

I didn't remember, or know, that windows workers were auto-updating chrome.

Would it be possible to start by updating chrome on 1 or 2 machines from each of the pools so I can run some tests on them (to make sure they will still work in the newer version and primarily because we will be jumping a few versions here)? If these tests show that everything is good, then we would be able to roll them out as they become ready and update chrome on all platforms at the same time.

By "update to window, linux, and macos at the same time" do you mean updating the OS or to have all of those update chrome at the same time?

I was asking if you'd want for us to coordinate changing the binary on the workers to happen at the same time.
I'm not opposed to changing macos+linux to autoupdate to match windows, but I'll have to review the history of how that was decided and set up.

For now, I'll get this version, 79.0.3945.88, set up for linux64(moonshot) and the macos(minis, and the bitbar mbp's) hardware. And I'll update you when that is ready.

Oh ok, yes it would be good if we could update the binaries on mac and linux at the same time. And sounds good, thanks!

testing update on a mac mini:

[dhouse@t-mojave-r7-444.test.releng.mdc1.mozilla.com ~]$ ls -la /var/db/.puppet*chrome*
-rw-r--r--  1 root  wheel  162 Sep 16 16:52 /var/db/.puppet_appdmg_installed_googlechrome_v76.0.3809.132.dmg
[dhouse@t-mojave-r7-444.test.releng.mdc1.mozilla.com ~]$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 76.0.3809.132

According to the source for the pkgdmg module we're using to install the dmg, the filename must be changed to get a new version installed (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pkgdmg.rb#L34).
I tested with a new filename and it was not installed. So I need to test further.

(In reply to Dave House [:dhouse] from comment #14)

testing update on a mac mini:

[dhouse@t-mojave-r7-444.test.releng.mdc1.mozilla.com ~]$ ls -la /var/db/.puppet*chrome*
-rw-r--r--  1 root  wheel  162 Sep 16 16:52 /var/db/.puppet_appdmg_installed_googlechrome_v76.0.3809.132.dmg
[dhouse@t-mojave-r7-444.test.releng.mdc1.mozilla.com ~]$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version
Google Chrome 76.0.3809.132

According to the source for the pkgdmg module we're using to install the dmg, the filename must be changed to get a new version installed (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pkgdmg.rb#L34).
I tested with a new filename and it was not installed. So I need to test further.

It applied correctly: pr'd

tested (and pr'd) the update for linux64 hardware:

[root@t-linux64-ms-447 ~]# /opt/google/chrome/chrome --version
Google Chrome 76.0.3809.132 unknown
[root@t-linux64-ms-447 ~]# puppet agent --test --server=releng-puppet2.srv.releng.mdc1.mozilla.com --environment=dhouse
[...]
Notice: /Stage[main]/Packages::Mozilla::Google_chrome/Package[google-chrome-stable]/ensure: ensure changed '76.0.3809.132-1' to '79.0.3945.117-1'
[...]
[root@t-linux64-ms-447 ~]# /opt/google/chrome/chrome --version
Google Chrome 79.0.3945.117 unknown

(In reply to Greg Mierzwinski [:sparky] from comment #11)

Would it be possible to start by updating chrome on 1 or 2 machines from each of the pools so I can run some tests on them (to make sure they will still work in the newer version and primarily because we will be jumping a few versions here)? If these tests show that everything is good, then we would be able to roll them out as they become ready and update chrome on all platforms at the same time.

I'll set up separate pools for testing only 2 machines each for linux64 and mojave, and one of the mbp's (leaving the mbp without the upgrade during the current queue/tasks).

When the power macbookpro's queue is down to zero, I'll test the chrome79 upgrade on one of the two mbp's. After that, all three (linux64(moonshot), macos mojave (mac minis), and the bitbar power macbookpro's) will be ready for your testing the upgrade.

here are the test pools with the v79 chrome:
macos mojave: gecko-t-osx-1014-beta (2 r7 mac minis: 444, 446)
linux64: gecko-t-linux-talos-beta (2 moonshot cartridges: 444, 446)

I skipped the macos mojave macbookpros (bitbar/power) since we only have 2 and it will be easy to back-out if there is a problem.

Thanks Dave! I've started a test run here: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f4b8fa5f8b60d6499f425a4ebbf00c160891565f

:dhouse, the mac machines are having issues: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f4b8fa5f8b60d6499f425a4ebbf00c160891565f&selectedJob=284896106

It's not related to chrome though, it seems to be because it's missing python3.7 on the machine - is that possible?

(In reply to Greg Mierzwinski [:sparky] from comment #21)

:dhouse, the mac machines are having issues: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=f4b8fa5f8b60d6499f425a4ebbf00c160891565f&selectedJob=284896106

It's not related to chrome though, it seems to be because it's missing python3.7 on the machine - is that possible?

Thanks! I ran into the same problem with another test pool. I'm guessing there is a hardcoded/switch setting for the python3 path based on the workerType -- which is broken by this workerType name. So I've manually symlinked the python3.7 executable into the expected place, /tools/python37/bin/python3.7.

Please try again and we can see if it gets farther. If there are significant other issues, it may be better to just start upgrading a portion of the main pool and test through that.

Ah ok, I've started a new test run here, I'll let you know how it goes: https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=7231888f8a83c38ee23999edeba4e59b318e1939

:dhouse, the tests look good. We can proceed with updating all machines.

(In reply to Greg Mierzwinski [:sparky] from comment #24)

:dhouse, the tests look good. We can proceed with updating all machines.

Thanks! I'll get the PR's approved and merged and update here with progress (and when we have all updated).

Both are merged.
90% of linux64 have picked up the new version. I'll check later this evening to verify all linux64+macos hardware workers are running v79.

the linux64(moonshot) and macos(mini) production-pool workers are all running chrome v79 now

There are ~10 linux64 and ~30 mac minis that are still on v76 because their staging/dev branches are not updated yet.

Tests are looking good and I see that most of them (if not all) are on v79 now. :dhouse feel free to close this bug when you think it's good. Thanks a lot! :)

The bitbar macbookpro's are updated.
the linux64(moonshots) are 100% updated
the production macos mojave (mac minis) machines are also all updated (staging/dev are not updated yet.