Closed Bug 1160410 Opened 10 years ago Closed 7 years ago

Adapt Treeherder to artifacts in S3 instead of FTP

Categories

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1420983

People

(Reporter: jgriffin, Unassigned)

References

Details

AFAIK, the only place in Treeherder where the current FTP location for artifacts makes any difference is the way we guess the build directory by stripping the filename off of a build artifact. For S3 artifacts, this doesn't work, as artifacts have a url like https://queue.taskcluster.net/v1/task/uEamOq1QQIGjltvLoRnicw/artifacts/public/build/firefox-40.0a1.en-US.linux-x86_64.tests.zip and stripping the filename results in an unbrowsable url: https://queue.taskcluster.net/v1/task/uEamOq1QQIGjltvLoRnicw/artifacts/public/build However, we can grab a JSON feed of artifacts using https://queue.taskcluster.net/v1/task/uEamOq1QQIGjltvLoRnicw/artifacts ...and then populate a list in the UI ourselves (this makes sense given the work in bug 1158901). Or, we could display a link to the artifact browser at e.g., https://tools.taskcluster.net/index/artifacts/#buildbot.revisions.000963271679.try/buildbot.revisions.000963271679.try.linux64. Note that the 'buildbot' index may be changing, see https://bugzilla.mozilla.org/show_bug.cgi?id=1132151#c20. There's also a TaskCluster API we can use for querying this data, see bug 1132151 for discussion.
(In reply to Jonathan Griffin (:jgriffin) from comment #0) > AFAIK, the only place in Treeherder where the current FTP location for > artifacts makes any difference is the way we guess the build directory by > stripping the filename off of a build artifact. That's correct :-) Either of those approaches (displaying in our UI using the json feed or just linking to the artifact browser) WFM. Do we have a timeframe for FTP going away?
(In reply to Ed Morley [:emorley] from comment #1) > > Do we have a timeframe for FTP going away? Not yet; June at the earliest but likely later.
Ok, thank you :-)
Priority: -- → P3
Switching away from using log URLs as the basis for the "build directory" changes a number of things, and likely affects either the approach or the viability of bug 1057314, bug 1158283, bug 1060649 and bug 1175367, so marking this as blocking them for now.
Depends on: 1181710
Depends on: 1192019
There are a whole raft of bugs related to, or in the dependency tree of this. On the releng side: * Bug 1192019 - builds-4hr still links to ftp, making it seem like build logs aren't available, but they can be found at archive.mozilla.org * Bug 1181710 - Devise a way to provide Treeherder with a URL for retrieving build artifacts (to replace the ftp build directory link) * Bug 1100624 - Upload build artifacts to s3 instead of ftp * Bug 1165405 - [tracker] Adjust release automation for ftp.m.o --> S3 migration On the Treeherder side: * <this bug> * Bug 1060649 - Store the build directory as a property on the job, rather than deriving it from the log URL * Bug 1057314 - Guess the build directory for Try jobs, so we can link to them before the jobs have completed, like TBPL * Bug 1175367 - Add a link to the B2G artifacts for a job, which are in a different directory to the rest * Bug 1158283 - Only derive a link to the "build directory" for buildbot jobs I'm still not clear on what the proposed solution is on the releng/taskcluster side for us having a way to figure out either a directory, or individual links to artifacts - we're pretty much blocked on bug 1181710.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.