Closed Bug 657359 Opened 14 years ago Closed 12 years ago

move pre-buildapi reports off build.mozilla.org

Categories

(Release Engineering :: General, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

(Keywords: buildapi, Whiteboard: [reporting])

Specifically: http://build.mozilla.org/builds/pending.html http://build.mozilla.org/builds/running.html http://build.mozilla.org/builds/pending/ Are any of these still required? Do they only exist for backward-compatibility with some API? Or are they for backward-compatibility with devs' bookmarks? In the latter case, would a 302 redirect work?
pending.html and running.html are what tbpl links to in its Infrastructure menu.
Are they just linked-to, or are they parsed?
Just linked-to, we're not that crazy. We parse http://build.mozilla.org/builds/builds-pending.js and http://build.mozilla.org/builds/builds-running.js. Does that also need to change, or do I just need to change the links?
Depends on: 657404
That will also need to change, but we should be more careful about it. I'm not sure if that's part of buildapi (and thus appropriate for buildapi.pub.build.mozilla.org) or distinct; if the latter, we should probably come up with a new vhost for it. Maybe catlee can weigh in there, when he's back?
Priority: -- → P3
Whiteboard: [buildapi][reporting]
The three locations in comment #0 are all generated on cruncher and pulled by an rsync to build.m.o. The same applies to http://build.mozilla.org/builds/builds-4hr.js.gz http://build.mozilla.org/builds/builds-2011-05-22.js.gz (and many friends) http://build.mozilla.org/builds/last-job-per-slave.txt pending.html is a snapshot of https://build.mozilla.org/buildapi/pending, taken every two minutes (cron:nthomas@cruncher). Similar for running.html, and the json equivalents builds-pending.js and builds-running.js. A redirect might be fine if buildapi can handle the load of tbpl users hitting it. pending/ is my trending graphs for running and pending jobs (cron:nthomas@cruncher). It hits buildapi/pending and running but is otherwise separate. I'd like to keep it going and move it wherever is appropriate. builds-4hr.js.gz and builds-YY-MM-DD.js.gz are generated from statusdb by reporter.py (cron:catlee@cruncher) - it lives at http://hg.mozilla.org/users/catlee_mozilla.com/bb_db/. I'm not sure if anyone is downloading those files, we'd have to get access to and check the apache logs on build.m.o. last-job-per-slave.txt is what it says it is, generated by a python script that queries statusdb (cron:nthomas@cruncher). It could move into buildapi but no idea when I'd get to that.
OK, sounds good. We'll add http://reports.pub.build.mozilla.org to host all of these, with a nice menu page. Bug 657784 updated accordingly.
Did I correctly follow through the chain and see that there will be a *.pub.build.mozilla.org cert, so that the builds-pending.js and builds-running.js which are currently part of the cause of the mixed-content warnings on https://tbpl.mozilla.org/ will work from https://reports.pub.build.mozilla.org/, without the auth requirement that https://build.mozilla.org/builds/builds-pending.js currently has?
Blocks: 686262
Blocks: 686263
I haven't gotten that far, but I'll check with you when we do get there, and see how to make it happen.
Well, s/reports/builddata/, but tbpl would still like there to be a valid cert for https://builddata.pub.build.mozilla.org/.
So from a look at the logs, the only paths on builddata.pub.build.mozilla.org being used are /buildjson/, which is proxied to buildapi01. I'm looking at what's still accessed on build.mozilla.org, and will report back.
Here's the set of paths accessed from http://build.mozilla.org/builds: /builds/buildfaster.csv.gz /builds/builds-xxxx-xx-xx.js.gz /builds/builds-4hr.js.gz /builds/builds-pending.js /builds/builds-running.js /builds/last-job-per-slave.txt /builds/last-job-per-slave.txt /builds/pending.html /builds/running.html /builds/pending/* all of which but the first are succinctly described in comment 5. This entire directory is rsync'd from cruncher, so I can just rsync it to two places and mirror the content effectively at a new URL. I propose mirroring /builds at http://builddata.pub.build.mozilla.org/reports after which time we'd need to clean up a bunch of links and bookmarks and whatnot to point to the new URL. I can add a 301 for the old URLs at any point where it might help. Does this sound OK? philor: A quick tour around tbpl doesn't reveal access to anything on build.mozilla.org (http or https). So, nothing we're changing here affects tbpl, but your request from 6 months ago is a good one and is pretty easy, so it's now bug 780902. Note that this would have the fringe benefit of serving the above data at https://secure.pub.build.mozilla.org/builddata/reports
Oh, and lest I forget, reports have some particular requirements from Apache, which I will reproduce faithfully: <Location /reports> AllowOverride All Options +Indexes AddType text/plain .gz AddEncoding x-gzip .gz ExpiresActive on ExpiresDefault "access plus 1 minute" </Location> <Directory /var/www/html/build/builds> Header set Access-Control-Allow-Origin "*" </Directory>
(this is in place now on the old, still-active cluster)
FYI, this is now ready to roll on the new cluster, too. So, over to releng to update links as noted in comment 11.
No longer blocks: 686263
Keywords: buildapi
Whiteboard: [buildapi][reporting] → [reporting]
I'd like to wrap bug 604688 up this quarter, if possible. To that end, I'd like to 301 http://build.mozilla.org/builds -> http://builddata.pub.build.mozilla.org/reports and similarly (although I don't see it used) https://build.mozilla.org/builds -> http://secure.pub.build.mozilla.org/builddata/reports Any objections?
Assignee: nobody → dustin
Hearing none, so ordered.
Ah, a few minor cleanups first, that I can take care of.
Depends on: 883176
OK, I don't see anything still hitting builds/ here, so I've added the 301.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.