Closed Bug 1159700 Opened 10 years ago Closed 9 years ago

A uniform way to find tasks for a given revision / branch / platform [/ app_name]

Categories

(Taskcluster :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: parkouss, Assigned: mrrrgn)

References

Details

Attachments

(1 file)

So far I there is way to map hg commits to taskids using the index: https://index.taskcluster.net/v1/task/buildbot.revisions.0500d91113bb.mozilla-central.linux64 So this uses the first 12 characters of the last changeset of a mercurial push. After some discussion with :pmoore on irc, it appears that this can not be used anywhere, because it will only work for something built with buildbot (does not works for b2g for example). Also it seems that when bug 1135206 and bug 1118394 will be landed, this won't work anymore for firefox desktop or fennec. So basically it would be great to have a uniform way to find a task given a revision and a branch in taskcluster, :pmoore talked about something like: taskcluster.revisions.<revision>.<branch>.<platform> Using the index as the route to perform the mapping. Also I have one concern (If I understand well) - the application name. I think that for example b2g and firefox desktop are on mozilla-central, so how do we distinguish those tasks ? On ftp, this was: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2015/01/2015-01-03-03-02-15-mozilla-central/ (firefox) http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2015/01/2015-01-03-16-02-13-mozilla-central/ (b2g)
Discussing with :parkouss in IRC, I think the application name is not needed explicitly in the index for buildbot builds, since the platform name is already unique. Eg: an OSX firefox build is 'macosx64', while an OSX B2G Desktop build is 'macosx64_gecko'. However, we still need to unify the index for buildbot and taskcluster, which should be handled at least in the buildbot side of things in bug 1133074.
I wrote a simple script that basically demonstrate what I need: https://gist.github.com/parkouss/90da45fe39250f817e7f So basically this is an attempt to get inbound builds for fennec/b2g and firefox using a changeset. I'm currently using the route 'buildbot.*', and this does not seems to work for finding b2g builds as stated in comment #1. I suspect that b2g may not be build with buildbot anymore... So what I basically need is a way to find artifacts from a build that occured on a given changeset for a given branch (and os/bits when this make sense). Also I find it not really useful to specify 'buildbot' in the route, in case it is built with something else the code will be broken. Please look at the sample script, this can help to understand it.
There were a couple of issues in the construction of your route, I've fixed these here and it seems to be working. Using my revised script: (taskcluster)vagrant@vagrant-ubuntu-trusty-64:~/90da45fe39250f817e7f$ ./get-inbound-taskcluster.py b2g linux 64 buildbot.revisions.493c86e6d74900dbd66c5c1538d7bf812fc872d7.b2g-inbound.linux64_gecko public/build/fxos-simulator.xpi public/build/target.crashreporter-symbols.zip public/build/target.linux-x86_64.json public/build/target.linux-x86_64.tar.bz2 public/build/target.tests.zip public/logs/live.log public/logs/live_backing.log
Attachment #8626642 - Flags: review?
Assignee: nobody → winter2718
I found the format by looking at the task config in tree at testing/taskcluster/tasks/builds/ Then verified inspecting a task: https://tools.taskcluster.net/task-inspector/#LWHGwpQGSnujlfv4ppB7Cg/0
Oh great, thanks Morgan! So it looks like basically for b2g the fill revision is needed (instead of the only 12 firsts characters) and also I need to add _gecko at the end. So far, so cool - thanks for the investigation. Now, can I use that code or is it going to break soon (because of the 'buildbot.' in the route) ? My main concern is being able to use that kind of route - even if for a some reason buildbot is replaced by something else for the build.
(In reply to Julien Pagès from comment #5) > Oh great, thanks Morgan! > > So it looks like basically for b2g the fill revision is needed (instead of > the only 12 firsts characters) and also I need to add _gecko at the end. > > So far, so cool - thanks for the investigation. > > Now, can I use that code or is it going to break soon (because of the > 'buildbot.' in the route) ? My main concern is being able to use that kind > of route - even if for a some reason buildbot is replaced by something else > for the build. That's a good point, so, I think the more sustainable route would be: index.gecko.v1.{branch}.revision.{platform}.{rev}.{job-type}.{opt|dbg} i.e. index.gecko.v1.mozilla-inbound.revision.linux.ba5f47ced2590d2697bca58596a36addc5d84cd7.b2g-desktop.debug
Setting the bug to resolved, re-open / tag me on another bug if there's more that your missing or this didn't solve your issue. :)
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Hm I don't think it fixes my problem completely, because if I look at a task for fennec: https://tools.taskcluster.net/task-inspector/#H3Wb3P9MQJ-HM699X8Qnnw/ There is no other route than buildbot.*. So it seems likely to break in the future (pretty soon) when fennec won't be built with buildbot anymore (see comment 0). And same thing for firefox.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Julien Pagès from comment #8) > Hm I don't think it fixes my problem completely, because if I look at a task > for fennec: > > https://tools.taskcluster.net/task-inspector/#H3Wb3P9MQJ-HM699X8Qnnw/ > > There is no other route than buildbot.*. So it seems likely to break in the > future (pretty soon) when fennec won't be built with buildbot anymore (see > comment 0). And same thing for firefox. So android builds are not available yet (they are hidden in try); but they do have a route as well: index.gecko.v1.try.revision.linux.cded5316989fd54db08c5787c87b9bdbfa4c00e8.android.opt As we port jobs to TC they will all have routes which follow this format.
Thanks, so yeah this works well! So what can I do for now is use "buildbot.*" routes where "index.gecko.v1.*" does not work (that is, fennec and firefox) I think that other bugs are opened to follow the migration of the route naming, so closing this bugs seems fine now. Thanks Morgan for the help!
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
Hey folks, I have some very recent commits to the tree that consume this indexing API over in Bug 1162191. I want folks to be aware of this consumer, which I expect to grow over time, although I'm not aware of any action needed.
Hello Morgan, I still have some issues to find b2g builds using your revised script. I can only get linux 64 builds this way. I tried some things, and I got this working for some platforms just by using the only 12 first characters of the changeset: buildbot.revisions.7877051b07c4.b2g-inbound.win32_gecko taskId: sPXqDYeWRTWZvWrYfWO6AA buildbot.revisions.2ab4034b2cc1.b2g-inbound.macosx64_gecko taskId: BprJO440T-Oy2jTW63O0Iw This does the trick for mac and win 32. I can not find any build for win 64 and linux 32 though. Do you have an idea of how I can get those ? And is there a bug for tracking b2g route naming uniformisation ?
Flags: needinfo?(winter2718)
(In reply to Julien Pagès from comment #12) > Hello Morgan, > > I still have some issues to find b2g builds using your revised script. I can > only get linux 64 builds this way. > > I tried some things, and I got this working for some platforms just by using > the only 12 first characters of the changeset: > > buildbot.revisions.7877051b07c4.b2g-inbound.win32_gecko > taskId: sPXqDYeWRTWZvWrYfWO6AA > > buildbot.revisions.2ab4034b2cc1.b2g-inbound.macosx64_gecko > taskId: BprJO440T-Oy2jTW63O0Iw > > This does the trick for mac and win 32. > > I can not find any build for win 64 and linux 32 though. > > > Do you have an idea of how I can get those ? And is there a bug for tracking > b2g route naming uniformisation ? Looking at treeherder, I believe the reason you can't find them is because they don't run. There's no task tracking routes as far as I know. We could come up with a style guide for creating jobs: probably a good idea.
Flags: needinfo?(winter2718)
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Version: unspecified → Trunk
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
No longer blocks: 1132151
Attachment #8626642 - Flags: review?
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: