At this point we control all the mirrors in bouncer, so sentry adds no value. We should delete it so all products added are instantly available in bouncer without having to wait for health checking.

nthomas, can you please review https://github.com/mozilla-services/go-bouncer/pull/211 when you get a chance?

\o/ We are going to stop using the uptake API soon too!

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2) > We are going to stop using the uptake API soon too! That's for Firefox 59 and higher, AIUI, but we'll also need to think about 58, 52ESR, and Thunderbird. We run a risk of the bouncer PR resulting in no mirror map data in the DB, so the Django uptake API [1] won't be able to return non-zero values, and those releases will be blocked after we 'push to cdns'. oremj, would it take much more effort to modify the uptake endpoint too ? [1] https://github.com/mozilla/tuxedo/blob/1f355359773f25fde519a40ed4aee9fa3ec80be5/apps/mirror/models.py#L143 - maybe!

I can send fake data, but need to know what exactly the code is looking for. Sentry is the only thing that updates the mirror map table, so that will no longer be populated. Can you link the code that uses the uptake endpoint?

Forgot to mention, I'm not deleting the uptake data, so once this is pushed live only new releases will have no uptake data. In addition, I can leave sentry running to fill in that data for some period of time, so the API endpoint will work as it does now.

Completely agree that this would only affect new releases after Sentry is stopped. ESR52 and Thunderbird could still be quite long-lived though and keep Sentry running for several months, so faking the uptake response is appealing. The uptake checks make several requests like: 07:20:41 INFO - Requesting https://bounceradmin.mozilla.com/api/uptake/?product=Firefox-58.0.2&os=win from tuxedo 07:20:42 INFO - Current uptake for Firefox-58.0.2 is 2000100 and it's looking for a value of 10000. If we could just returned that value for any product and they would be happy. FTR here's and example log for Firefox 58.0.2 [1], along with code [2], and config [3]. IIRC ESR52 and Thunderbird do the same kind of thing, but perhaps not in the exact same code. [1] http://mozilla-release-logs.s3.amazonaws.com/mozilla-release/firefox-58.0.2/build1/firefox_mozilla-release_uptake_monitoring-all-d_aAgpTmQXyDFQhSmvcfUQ-0 [2] https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/scripts/release/uptake_monitoring.py [3] https://hg.mozilla.org/releases/mozilla-release/file/default/testing/mozharness/configs/releases/bouncer_firefox_release.py

mbrandt, this bug will result in bouncer always redirecting to the CDNs, rather than performing existence checks before enabling a 302. It might affect your bouncer test suite if it looks for a redirect but doesn't follow it. Does the suite care if it gets a 404 from the CDN instead of bouncer, in error situations and prior to a release being copied into .../releases ? And does it do any uptake checks, similar to the one in commment #6 ? That would require a bouncer login so I suspect not. oremj, I can't quite tell from the headers if the CDN caches 404 responses. Do you know ?

nthomas, can you review https://github.com/mozilla/tuxedo/pull/55/?

Yes, the CDN will cache 404s, but only for 5 minutes.

(In reply to Jeremy Orem [:oremj] from comment #8) > nthomas, can you review https://github.com/mozilla/tuxedo/pull/55/? Done. (In reply to Jeremy Orem [:oremj] from comment #9) > Yes, the CDN will cache 404s, but only for 5 minutes. OK, that doesn't seem like a big deal.

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #7) > Does the suite care if it gets a 404 from the CDN instead of bouncer, in > error situations and prior to a release being copied into .../releases ? And > does it do any uptake checks, similar to the one in commment #6 ? That would > require a bouncer login so I suspect not. The end-to-end tests shouldn't be effected by the changes. The tests follow the redirects to their logical endpoint and verify the expected product is downloaded (based off of the product-details service). You are correct, the current tests aren't doing any uptake checks.

I've deployed the tuxedo and bouncer changes to: admin-bouncer.stage.mozaws.net admin-bouncer-releng.stage.mozaws.net bouncer-bouncer.stage.mozaws.net bouncer-bouncer-releng.stage.mozaws.net Let me know if you see any issues.

I'm not seeing any issues in the go-bouncer e2e test suites. I'm comfortable moving forwards. Do we need sign-off from stephend as well, not sure if this could effect his stub-attribution tests.

Rail, Nick, do either of you have a preference on when this goes out?

(In reply to Matt Brandt [:mbrandt] from comment #13) > I'm not seeing any issues in the go-bouncer e2e test suites. I'm comfortable > moving forwards. Do we need sign-off from stephend as well, not sure if this > could effect his stub-attribution tests. Thanks, Matt. I appreciate you roping me in; I should be good - my Stub Attribution automation tests only use locators.

Testing - I modified the Firefox 58.0.2 uptake check (taskId: d_aAgpTmQXyDFQhSmvcfUQ) to run against staging bouncer, the log is at [1]. Tuxedo reports uptake of 2000000 instead of 2000100, as expected, with no errors. I'm using 58 as a proxy for 52ESR and Thunderbird, but of course 58.0.x is almost finished with. The code looks good to go to prod from here. For timing that's really up to releaseduty which is bhearsum/mtabara/jlund. I'll just note March 13 is when we ship Firefox 59, and we start building that on March 5, see the calendar [2]. Ben, are you happy to deploy this to bouncer prod with all the other changes landing this week ? Shouldn't have any impact on 59.0, just the older branches. [1] https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-noarch/release-mozilla-release-firefox_uptake_monitoring-bm77-build1-build4.txt.gz [2] https://calendar.google.com/calendar/embed?src=mozilla.com_dbq84anr9i8tcnmhabatstv5co@group.calendar.google.com

adding Mihai as well

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #17) > adding Mihai as well I think the sooner we test land and test this (via one of the upcoming betas) the sooner we can rollback in case it affects 59(+).

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #16) > Testing - I modified the Firefox 58.0.2 uptake check (taskId: > d_aAgpTmQXyDFQhSmvcfUQ) to run against staging bouncer, the log is at [1]. > Tuxedo reports uptake of 2000000 instead of 2000100, as expected, with no > errors. I'm using 58 as a proxy for 52ESR and Thunderbird, but of course > 58.0.x is almost finished with. The code looks good to go to prod from here. > > For timing that's really up to releaseduty which is bhearsum/mtabara/jlund. > I'll just note March 13 is when we ship Firefox 59, and we start building > that on March 5, see the calendar [2]. Ben, are you happy to deploy this to > bouncer prod with all the other changes landing this week ? Shouldn't have > any impact on 59.0, just the older branches. > > > [1] > https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release- > noarch/release-mozilla-release-firefox_uptake_monitoring-bm77-build1-build4. > txt.gz > [2] > https://calendar.google.com/calendar/embed?src=mozilla. > com_dbq84anr9i8tcnmhabatstv5co@group.calendar.google.com Let's do it!

Does tomorrow work?

Let's do it!

r+ let's release this tamed beastie to wilds.

This has been released to production. https://github.com/mozilla-services/go-bouncer/releases/tag/v1.8.0 https://github.com/mozilla/tuxedo/releases/tag/v1.1.0