User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 Build Identifier: Permalinks should be made available to download the latest version Eg: http://download.mozilla.org/?product=firefox&version=latest&os=...&lang=... Currently we can only create links for a specific product version. Links are outdated as soon as a new version is released. This cause a lot of links to point to older Firefox versions as you can see in google: http://www.google.com/search?q=link:"http://download.mozilla.org/?product=firefox" Reproducible: Always With permalink, users would always download the latest version.

Oops, meant to do this to the AMO bug he filed, not this one!

Maybe Bouncer should support aliases for products so that a "firefox-latest" product would be an alias to whatever the latest Firefox release is...

(In reply to comment #4) > *** This bug has been confirmed by popular vote. *** Ugh. I upped the confirmation threshold from 1 to 10000, so this shouldn't happen again.

(In reply to comment #3) > Maybe Bouncer should support aliases for products so that a "firefox-latest" > product would be an alias to whatever the latest Firefox release is... > That would be sweet. I like that better than adding an &version to the URL.

Just came up on IRC again. We're getting an increasing amount of local community sites with MCS now, it'd be great if we wouldn't have to hack more logic and could just offer them a link to the latest stable version. Might be good to use those links in general. Not really sure if it's a good idea to have people add download links on their sites and end up with links to insecure old versions.

I'd like to suggest we implement this alias as a double redirect: http://download.mozilla.org/?product=firefox-latest&os=linux&lang=en-US 302s to http://download.mozilla.org/?product=firefox-3.5.7&os=linux&lang=en-US which 302s to ftp://ftp.mozilla.org/pub/firefox/releases/latest/linux-i686/en-US/firefox-3.5.7.tar.bz2 It is an extra round trip for the client, but I feel it is the easiest to implement and maintain since it requires no modifications to the mirror network and it doesn't impact metrics tracking of downloads either.

We could use the product-details library for this, but that means every time a product is released, bouncer needs to be SVN upped in production. Who's going to be in charge of that?

Product-details is here: http://viewvc.svn.mozilla.org/vc/libs/product-details/ It's a library we use on mozilla.com to track release information. It should probably be made into a feed at some point. Since this is responsible for generating a lot of our outgoing links on mozilla.com, it's a logical place for some of this stuff.

CCing Pedro because the product-details library might be useful for some work we've been doing determining what the latest product release and dev release are.

This is blocked on the product-details development. We need to -- at the very least -- figure out how to import the existing PHP data into our Python projects, or we need to write a wrapper/web service that'll provide this data for us in a readable format.

This will be needed for the stub installer... something along the lines of latest-release without a version in the path. How difficult would this be to fix?

Why does this need to rely on product-details? releng already have access to bouncer, so couldn't they update these links as part of the automation when pushing releases live?

(In reply to Mark Banner (:standard8) from comment #15) > Why does this need to rely on product-details? releng already have access to > bouncer, so couldn't they update these links as part of the automation when > pushing releases live? I think metrics uses the Bouncer logs to track downloads per version, so that could be a possible reason. I'm not fully plugged into this though, so I'm not sure.

(In reply to Ben Hearsum [:bhearsum] from comment #16) > (In reply to Mark Banner (:standard8) from comment #15) > > Why does this need to rely on product-details? releng already have access to > > bouncer, so couldn't they update these links as part of the automation when > > pushing releases live? > > I think metrics uses the Bouncer logs to track downloads per version, so > that could be a possible reason. I'm not fully plugged into this though, so > I'm not sure. Well, let's chase that down. Adding Gilbert to connect us with the right people in metrics. Ben/Catlee - if metrics gives us the go ahead, who should we work with on the RelEng side?

If we want this change to bouncer not to disrupt any of our existing systems for calculating downloads, we would need it to be implemented as a redirect rather than a symlink or alias. That way, there would be a request logged for a url containing product=firefox-latest which we could ignore, and a following request logged for a url containing product=firefox-<version> that we could count as we currently do. I am happy to entertain alternatives to being able to count downloads outside of the current log parsing mechanism, that would just require some communication with other teams such as engagement who rely on these numbers.

Is this progressing out of the realm of bug discussion into something we should convene a quick meeting around? (Not meant as a leading question, I'm just not sure)

Who owns this? Need to make progress ASAP as it blocks the stub installer effort.

I was under the impression that Bouncer has been deprecated in favor of the CDN. We were told that no staging server would be forthcoming, for example.

The conversations I had read stated that bouncer would still be used and redirect as appropriate to the CDN. The end result we are looking for is the ability to have a permalink to the latest release build as well as for nightly, aurora, and beta though those are secondary.

OK, I'll resurrect that. There were a number of patches in-queue before we were told to stop work, so we'll be able to ship those, too. (IPv6 support and regional fallback among them).

(In reply to Robert Strong [:rstrong] (do not email) from comment #22) > The conversations I had read stated that bouncer would still be used and > redirect as appropriate to the CDN. The end result we are looking for is the > ability to have a permalink to the latest release build as well as for > nightly, aurora, and beta though those are secondary. hrmm.. this is all news to us, like Laura mentioned we saw a light at the end of the tunnel for Bouncer. so, yeah, we'll need to stand up some dev resources for it. (can IT be included on these discussions so we can plan for this?) Maybe we need to address the requirements that need fulfilled and whether it is worthwhile supporting bouncer going forward. Weighting and geolocation are moot at this point, all it is doing is serving up redirects. What is the timeline for this?

In one of the emails regarding the CDN mrz requested that all requests still go through bouncer. If that isn't the case and there is a way to get a permalink to the latest that would be just as good.

Yes, and today they still do because of metrics. I think the issue is being able to move away from it versus adding new development to it. I'm happy to spin up what we need to for the latter, just wishing there was a bigger picture here that we could have pointed to back when we stopped development and say "yes, we will need it after the switch to CDN" as we've wasted a bit of time since.

If we had the following url (for all latest l10n builds) without the version in the string then it would be possible. http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/win32/en-US/Firefox%20Setup%2015.0.exe catlee, nthomas, bhearsum could you weigh in on if this is something that can be done and how difficult it would be?

Presently all requests do still go through Bouncer. Bouncer provides the translation between: http://download.mozilla.org/?product=whatever&os=whatever&lang=whatever ... into a link to an actual file. This is still useful, because it abstracts having to know the actual filename away from the location providing the link (www.mozilla.org, etc). It also provides a convenient centralized point for metrics to be gathered. If all these links went straight to the CDN name, we wouldn't know how often the button got clicked... so there needs to be at least some logic around this. As I see it those are the only two still-useful functions that Bouncer provides. Both could be done in other ways, but perhaps a centralized app is still the right way. In effect, Bouncer could now theoretically be reduced to a large list of Apache rewrites/redirects. Not that I'm necessarily advocating for this design, but that's what it boils down to. One URL in, another (predictable) URL out.

If the product=whatever only contained the product without the version info to download the latest it would meet the requirements. Currently http://download.mozilla.org/?product=firefox&os=win&lang=en-US offers Firefox Setup 1.0.exe

(In reply to Robert Strong [:rstrong] (do not email) from comment #25) > In one of the emails regarding the CDN mrz requested that all requests still > go through bouncer. If that isn't the case and there is a way to get a > permalink to the latest that would be just as good. Just to clarify what I said or what I recall being discussed - dslater argued that bouncer is unowned code and moving to CDN negated needing to resource bouncer. What bouncer provides in terms of metrics could be replicated in smaller code that always punted to CDN (in fact we talked about doing simple Apache rewrites or within the load balancer itself). If bouncer is a blocking req for stub installer, then we should certainly resource the development side.

If we want to continue to have initial download request metrics per product version, then at a minimum, a replacement to bouncer would need to be able to redirect from a request for "firefox-latest" to "firefox-<version>", and then it would need to redirect to the actual file name in the CDN or wherever.

Just wanting to point out that the developers are available. However, we were not at any point involved in the discussion of Bouncer, just informed that it was no longer needed. It's simple for us to resume maintenance, just need proper dev and staging environments before it can be released.

(In reply to Robert Strong [:rstrong] (do not email) from comment #27) > If we had the following url (for all latest l10n builds) without the version > in the string then it would be possible. > http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/win32/en- > US/Firefox%20Setup%2015.0.exe > > catlee, nthomas, bhearsum could you weigh in on if this is something that > can be done and how difficult it would be? Our quick SWAG is that doing this or having a version-less product in bouncer is about the same amount of effort for us. Neither are particularly difficult, probably needing 1 or maybe 2 weeks of development/testing.

Reassigning to Rik since brandon is out. Also pushing on the staging bug.

To be clear on what is needed here: We want a link like http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US to redirect to http://download.mozilla.org/?product=firefox-15.0&os=win&lang=en-US ? We don't want to remove the os and lang fields, right?

That is correct.

This is the patch to get this into Bouncer. Please note that most of the work will be setting up the servers to include product-details (and set a cron to update product-details)

Lots of miscommunication here, so just met w/rstrong, laura, catlee, gilbert to sort this out. tl;dr: We will do (2a) below to fix this today. We do not need any IT/RelEngAutomation/Webdev Bouncer changes to unblock stub installer work. Summary: 1) We do *not* need changes to bouncer (production), or need to stand up bouncer (staging) in order to fix this bug. We have a few options below, any which can be done today and will unblock stub installer. 2) Possible solutions (we are doing 2a unless anyone has strong objections): 2a) add another entry to bouncer. This new entry would be called "firefox-latest", but redirects to the same URL as the current latest product (FF15.0 as of today). This extra entry in bouncer would need to be manually updated by RelEng as part of every release. This is a few minutes of manual work, and we'll do this manually until we can automate this. ...or... 2b) add another entry to bouncer. This new entry would be called "firefox-latest", but redirects to a symlink on ftp.m.o called "firefox-latest", which points to the current latest product (FF15.0 as of today). This entry on bouncer would not need manual updating. This symlink on ftp would need to be manually updated by RelEng as part of release process. This is a few minutes of manual work, and we'll do this manually until we can automate this. ...or... 2c) setup an apache http redirect so that requests to "firefox-latest" are redirected to FF15.0. This would need to be updated by IT as part of release process. This requires RelEng to coordinate with IT during releases 3) Currently, only beta and release builds use bouncer. Try, nightly and aurora builds do not use bouncer, so we need a slightly different way of handling these. Explicitly, stub installer will not work for try builds. For nightly/aurora builds: * worst case, nightly/aurora users who run stub installer will download latest release build. rstrong is fine with this for the present as this unblocks testing/rollout. * best case, we have nightly stub installer to install the latest nightly builds, and the latest aurora stub installer to install the latest aurora build. ** for details see https://bugzilla.mozilla.org/show_bug.cgi?id=731299 for Aurora Bouncer support 4) metrics can still answer "how many downloads of what version" questions so long as we clearly note the timing of changing "firefox-latest" from pointing at one version to pointing at another version. Still to be done, but not blocking stub installer: * revive projects to support bouncer in production. This includes: ** finalize staging machine, for any further changes to Bouncer (https://bugzilla.mozilla.org/show_bug.cgi?id=750798 ) ** update Bouncer production infrastructure https://bugzilla.mozilla.org/show_bug.cgi?id=786820 * push the code change to Bouncer, to automate this functionality so that Releng don't have to use a manual workaround for every release. Patch is being reviewed, see attachment.

Any option that involves a redirect back to download.mozilla.org with the full product-version string will be sufficient for metrics. We won't need any special notice when the redirect changes because we can just ignore the firefox-latest requests entirely and only count the requests that have a full product-version in them.

Comment on attachment 656607 [details] [diff] [review] Patch Review of attachment 656607 [details] [diff] [review]: ----------------------------------------------------------------- I think this diff is backwards? You're backing out the feature... So *under the assumption* that I am supposed to read every + as a - and vice versa: r+ ::: bouncer/php/index.php @@ -155,5 @@ > $where_lang = 'en-US'; > > - // Special case for bug 398366 > - if ($product_name == 'firefox-latest') { > - $string = file_get_contents(INC . '/product-details/json/firefox_versions.json'); Does it make sense to do json_decode rather than just importing the PHP file? After all, the product details library is a PHP file (then turned into JSON). Though, perhaps if we ever change that, we'll end up using JSON anyway so.... nevermind!

(In reply to John O'Duinn [:joduinn] from comment #38) > Lots of miscommunication here, so just met w/rstrong, laura, catlee, gilbert > to sort this out. > > tl;dr: We will do (2a) below to fix this today. We do not need any > IT/RelEngAutomation/Webdev Bouncer changes to unblock stub installer work. > > > Summary: > 1) We do *not* need changes to bouncer (production), or need to stand up > bouncer (staging) in order to fix this bug. We have a few options below, any > which can be done today and will unblock stub installer. This is wonderful news. Thanks for sorting this out! > 2) Possible solutions (we are doing 2a unless anyone has strong objections): > 2a) add another entry to bouncer. This new entry would be called > "firefox-latest", but redirects to the same URL as the current latest > product (FF15.0 as of today). This extra entry in bouncer would need to be > manually updated by RelEng as part of every release. This is a few minutes > of manual work, and we'll do this manually until we can automate this. Seems like a pretty direct approach. I'm unqualified to comment on the ramifications of this. > ...or... > 2b) add another entry to bouncer. This new entry would be called > "firefox-latest", but redirects to a symlink on ftp.m.o called > "firefox-latest", which points to the current latest product (FF15.0 as of > today). This entry on bouncer would not need manual updating. This symlink > on ftp would need to be manually updated by RelEng as part of release > process. This is a few minutes of manual work, and we'll do this manually > until we can automate this. This is potentially problematic because of caching (our load balancers and the CDNs). It would result in a few days' of inconsistent results, where firefox-latest pointed to the old version on things that have it cached that way, and the new version on things that didn't. We can flush caches, but that takes a couple hours and requires IT involvement... at which point, 2c is just as good... actually better in many ways. > ...or... > 2c) setup an apache http redirect so that requests to "firefox-latest" are > redirected to FF15.0. This would need to be updated by IT as part of release > process. This requires RelEng to coordinate with IT during releases This would be a very simple Apache rewrite on our end. Effectively it would look like the ones we did for funnelcake11/12/13/14, when we were testing CDN product delivery. We can do this, but I definitely understand a preference on not needing more hands in the pot. :) Thinking down the road a bit... this rewrite could be placed into a .htaccess file, giving anyone with commit access to it the ability to update the redirect. On top of that, we could make Bouncer deployed via our Chief system, which allows push-button deployments that don't require IT involvement. I'm not sure if this is ultimately better than the 2a solution, though... perhaps a Bouncer patch like Rik's is better yet. The same basic approach ought to handle Nightly and Aurora properly too, simply by having the relevant .htaccess file be on the FTP cluster, in the right directory.

I've gone ahead and manually created bouncer links in the meanwhile: http://download.mozilla.org/?product=firefox-latest&os=win&lang=en-US http://download.mozilla.org/?product=firefox-latest&os=osx&lang=en-US http://download.mozilla.org/?product=firefox-latest&os=linux&lang=en-US http://download.mozilla.org/?product=firefox-latest&os=linux64&lang=en-US These currently point to Firefox 15.0. We'll need to bump these to 16 manually if the bouncer patch hasn't been deployed by then.

So the patch was pushed to the dev environment. But it's not behaving properly. $ curl -I 'https://download-dev.allizom.org/?product=firefox-latest&os=osx&lang=fr' Location: ?product=firefox-&os=osx&lang=fr

Interesting that download.mozilla.org works http://download.mozilla.org/?product=firefox-latest&os=osx&lang=fr

(In reply to Robert Strong [:rstrong] (do not email) from comment #44) That's hard coded entries from bug 786838.

(In reply to Nick Thomas [:nthomas] from comment #45) > (In reply to Robert Strong [:rstrong] (do not email) from comment #44) > > That's hard coded entries from bug 786838. Right though aren't they all essentially hardcoded with the one's with the version being automated entered and the new ones being manually entered?

To summarise what we said on IRC, firefox-latest on download.m.o is manually configured locations as a workaround, which will go stale when new releases come along. The products like firefox-15.0.1 are submitted automatically by the release automation.

Can the bouncer patch also be made to support beta products?

It would certainly be possible to add the beta products. The way the code is constructed, a special code block handles the special case for "firefox-latest". We could add a second special case codeblock, or make bouncer a bit smarter in terms of handling these two possibilities. Is this something that blocks ship on stub installer, or could it wait until after stub installer is out the door?

Steps to QA: This bug essentially creates an alias from firefox-latest to the very latest version of Firefox that has been released. This can be tested by using cURL against the stage environment: $ curl -IL 'https://download.allizom.org/?product=firefox-latest' What should happen is bouncer should redirect the user back to Bouncer, with the version properly decoded for the current version of Firefox; it should also preserve language and OS arguments. Bouncer will then redirect the user to the appropriate mirror. NOTE: Because the data in stage is slightly older than the data in production, but the firefox-latest obtains it's version information from a different place than the database, the script will attempt to redirect the user to Firefox-15.0.1 (the latest version) and you will get a 404 from Bouncer stage, because it doesn't know about 15.0.1. This is expected behavior and should not be considered a failure.

(In reply to Brandon Savage [:brandon] from comment #49) > It would certainly be possible to add the beta products. The way the code is > constructed, a special code block handles the special case for > "firefox-latest". We could add a second special case codeblock, or make > bouncer a bit smarter in terms of handling these two possibilities. > > Is this something that blocks ship on stub installer, or could it wait until > after stub installer is out the door? I don't think this blocks shipping of stub installer. We have manual workarounds for now.

Environments: ============= * For all of the manual + automation tests below, we tested on: 1. download.allizom.org (staging) 2. Against 'new-prod,' before deploy, for which we tricked-out /etc/HOSTS, locally, and on our Jenkins box: #63.245.217.79 download.mozilla.org 3. download.mozilla.org Manual testing: =============== * firefox-aurora-latest host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-aurora-latest&lang=en-US&os=win' HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-17.0a2.en-US.win32.installer.exe * firefox-beta-latest host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-beta-latest&os=win&lang=en-GB&os=win' HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/16.0b6/win32/en-GB/Firefox%20Setup%2016.0b6.exe * firefox-latest-euballot (must be in a European locale, such as en-GB) host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-latest-euballot&os=win&lang=en-GB'HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/16.0/win32-EUballot/en-GB/Firefox%20Setup%2016.0.exe * firefox-latest host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-latest&lang=en-US&os=win'HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/15.0.1/win32/en-US/Firefox%20Setup%2015.0.1.exe * firefox-beta-stub host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-beta-stub&lang=en-US&os=win' HTTP/1.1 302 Found Location: https://download-installer.cdn.mozilla.net/pub/mozilla.org/firefox/releases/stub/Firefox Beta Stub Installer.exe * firefox-nightly-latest host-7-68:~ sdonner$ curl -ILk 'https://download.mozilla.org/?product=firefox-nightly-latest&lang=en-US&os=win' HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe Automation: =========== * We set up a repo, here: https://github.com/mozilla/bouncer-tests, and are continuing to augment/refactor, and hone in on any missing automation coverage (for the above + other functionality) * Simple script is also at https://gist.github.com/3842439, which we used to both manually test, and, in combination with this bug + real-time IRC updates, test Verified FIXED!

Reopen this bug because firefox-latest seems not get updated, still goes to 16.0.2. HTTP/1.1 302 Found Server: Apache X-Backend-Server: bouncer1.webapp.phx1.mozilla.com Cache-Control: max-age=15 Content-Type: text/html; charset=UTF-8 Date: Wed, 21 Nov 2012 02:42:07 GMT Location: ?product=firefox-16.0.2&os=win&lang=en-US Transfer-Encoding: chunked Connection: Keep-Alive Set-Cookie: dmo=10.8.81.216.1353465727770334; path=/; expires=Thu, 21-Nov-13 02:42:07 GMT X-Cache-Info: caching

This has nothing to do with the code itself. Please file a new bug for webops to look at the cron job that updates the database.

Some of the OSX versions are not found. Working: https://download.mozilla.org/?product=firefox-latest&os=osx&lang=en-US Not working: https://download.mozilla.org/?product=firefox-aurora-latest&os=osx&lang=en-US https://download.mozilla.org/?product=firefox-nightly-latest&os=osx&lang=en-US https://download.mozilla.org/?product=firefox-beta-latest&os=osx&lang=en-US

There's two mechanisms in play here. The working link hits the code from this bug, and gets a redirect to product=firefox-22.0 (at least right now). That has a definition for os=osx so the download works. The other three products don't hit this code; they get updated manually. They also only have a location set up for os=win, which is all we need right now for the stub installer. There is also some en-US hardcoding for nightly and aurora. So this is expected behaviour as of right now. AFAIK we haven't promoted the *-latest links at all.