It would make things like funsize scheduling much easier if we had a way to query the list of blobs and inspect individual blobs without requiring credentials or special network access.

Benson suggested adding an API to the existing public app. This seems like it wouldn't be too difficult, and might be a good opportunity to start using Swagger. Something feels funny to me about allowing anyone in the world to query parts of the database, but I can't find a rational reason not to allow it. We already export a large swath of the database to a public location, and there's nothing secret in it...so it should be fine. To the queries specifically, it sounds like you want two endpoints: 1) Get a list of releases (possibly with some way to filter them on the server?) 2) Get the full blob for a specific release And maybe 3) Get a specific platform+locale from a specific release?

1) yes, maybe with some basic sorting / pagination. like the most recent N releases matching some pattern. 2) yup, definitely need this 3) I think funsize in this case would be interested in an entire blob at once, not necessarily platform/locales specifically. The release or nightly decision task logic would be constructing the set of partials to generate spanning all platforms and locales.

Catlee and I were talking about this a bit more just now, and a couple of things came up: 1) Even if it's not hosted separately, having a different domain for the API would probably be good. It lets us distinguish the API uses from actual updates, move the API to its own thing later if needed, and possibly do some traffic shaping or rate limiting if it becomes necessary. 2) If we build this as a Swagger API, it would be good to build in a way that it can be re-used and extended by a new admin API later (so that the admin API doesn't have to write its own read operations, too).

Updating this summary to expand the scope. Releases are clearly the #1 priority here, but I also think it would be good to have some access to the Rules, particularly so that anyone can see what Releases are live at any given point in time. We should also expose the history tables for Rules & Releases, to allow people to do archaelogy on their own. I don't think that we need to provide public access to Permissions, User Roles, or any of the Scheduled Changes tables.

Allan told me he wants to work on this soon, let's get it assigned :).

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/e04ab19b22fd7fd59913ebf19a2009235efe4672 bug 1325377: restructure web layers and convert public app to swagger (#276). r=bhearsum

(In reply to [github robot] from comment #6) > Commit pushed to master at https://github.com/mozilla/balrog > > https://github.com/mozilla/balrog/commit/ > e04ab19b22fd7fd59913ebf19a2009235efe4672 > bug 1325377: restructure web layers and convert public app to swagger > (#276). r=bhearsum This was part 1 to this bug - Allan did some really great foundational work to make it easier to add the public API. Allan, can we chat about following up with the actual public API when you have time?

We pushed part 1 of this to production yesterday and it uncovered an issue where update URLs with unicode characters in them do not get served updates. It turns out that we get tons of requests from Ubuntu and Mint builds with update URLs like: https://aus5.mozilla.org/update/3/GMP/53.0/20170421105455/Linux_x86_64-gcc3/null/release-cck-%20%C3%A2%C2%80%C2%9Cmint%C3%A2%C2%80%C2%9D/Linux%204.4.0-53-generic%20(GTK%203.18.9,libpulse%208.0.0)/mint/1.0/update.xml. I think this is pretty serious, because it means none of these builds are receiving GMP or SystemAddon updates, which I'm pretty sure we want them to get. Even with older code, I seem to get 500s locally, but the new code caused a massive increase in the number of 500s we saw in production. I'm having trouble explaining that, but I think we should strive to gracefully handful unicode in the URL. We probably want to convert it to a string as best we can.

On my tests I got errors on validation level. This new PR the strings are encoded to utf-8. Do you have another error type in logs?

(In reply to Allan [:ex-dev] from comment #9) > Created attachment 8865167 [details] > PR with fixes the unicode issues in URL. > > On my tests I got errors on validation level. This new PR the strings are > encoded to utf-8. > Do you have another error type in logs? That was the only one that came up.

Comment on attachment 8865167 [details] PR with fixes the unicode issues in URL. Reviewed in github.

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/3afc89072d986e832bcd67afa58e04361ac49c08 bug 1325377 - reorg of web code and swaggerification of existing public endpoints (#314). r=bhearsum

Allan, any idea when you'll be able to have a look at the public API for Rules and Releases?

Next saturday, probably. Do you have a deadline to this bug?

(In reply to Allan [:ex-dev] from comment #14) > Next saturday, probably. Do you have a deadline to this bug? No deadline, but there's some other work we're looking at that is much easier once we have the public API, so we'd just like to plan for it. Do you mean Saturday the 20th or 27th? No problem either way.

(In reply to Ben Hearsum (:bhearsum) from comment #15) > (In reply to Allan [:ex-dev] from comment #14) > > Next saturday, probably. Do you have a deadline to this bug? > > No deadline, but there's some other work we're looking at that is much > easier once we have the public API, so we'd just like to plan for it. Do you > mean Saturday the 20th or 27th? No problem either way. Both, I think. It depends on the demands of my work. haha

(In reply to Allan [:ex-dev] from comment #16) > (In reply to Ben Hearsum (:bhearsum) from comment #15) > > (In reply to Allan [:ex-dev] from comment #14) > > > Next saturday, probably. Do you have a deadline to this bug? > > > > No deadline, but there's some other work we're looking at that is much > > easier once we have the public API, so we'd just like to plan for it. Do you > > mean Saturday the 20th or 27th? No problem either way. > > Both, I think. It depends on the demands of my work. haha No worries, thanks for the quick reply!

Comment on attachment 8877143 [details] Public API - PR Part 2 Reviewed in Github

Commit pushed to master at https://github.com/mozilla/balrog https://github.com/mozilla/balrog/commit/2954a626c6b07d270bc1d703b01ef23136eb491d bug 1325377: Public API for Rules & Releases (#320). r=bhearsum,sfraser,aksareen

This appears to be working well in production, thank you a TON Allan - many people have been looking forward to this. I believe there's a couple of things we wanted to follow up with, but we can do that in separate bugs.

Very Nice, Is the endpoint to API already available?

(In reply to Allan [:ex-dev] from comment #22) > Very Nice, > Is the endpoint to API already available? Yup. The swagger docs are at https://aus5.mozilla.org/ui/, and the API is at https://aus5.mozilla.org/api/v1 (eg: https://aus5.mozilla.org/api/v1/rules)

Newbie question: -Is the database protected against DDOS attack? If, for instance, I get all rules, then all revisions for each rule for a lot times, from different threads. Will it impact the /update endpoints ?

(In reply to Allan [:ex-dev] from comment #24) > Newbie question: -Is the database protected against DDOS attack? > > If, for instance, I get all rules, then all revisions for each rule for a > lot times, from different threads. Will it impact the /update endpoints ? There's no special protection we have against doing that, but aus5.mozilla.org is set-up to handle a very high volume of requests, and automatically scales up if load goes up. The primary bottleneck we may have is the database server. I don't think we need to worry about DDoS for the moment (I think it would be tough to generate enough requests to really bog down the server), but I could certainly see a use case for doing some amount of rate limiting. That might be something we'd want to do at the ELB or ngnix level, though.

Cool, good to know a little more of releng services structure.