I think we can start working on getting builds from inbounds using taskcluster. We will probably speed up things for hunting inbounds. Pls, this is the future. :) It seems to be working for firefox/fennec and b2g, and those are sufficient for mozregression. The route are different, and some are going to change in the future but that won't impact the code much if it is designed with that in mind. See bug 1159700 as a starting point to see the required code logic.

So, I have a *working* code here: https://github.com/mozilla/mozregression/compare/master...parkouss:taskcluster-inbound There is two things though: 1. I am waiting for taskcluster to be less restrictive on requests version (bug 1178089) 2. b2g inbound builds are not here for win64 and linux 32. It may be normal according to bug 1159700 comment 13. Note that there is an inbound speed download improvement I think. At least the inbound first step is really fast now. Also note that persisting inbound file compatibility is broken: we don't have the same build date, so we can't use the timestamp as the beginning of the name as we did. I take the opportunity to use the 12 first characters of the changeset instead.

Eh! Will, tell me what you think of this, and please give it a try if you find the time (this is an important change for mozregression). Thanks!

Comment on attachment 8629374 [details] Taskcluster inbound The code looks good to me; we may want to figure out where the error mentioned in the PR is coming from before landing.

Well thanks for the review and testing Will! I fixed one thing regarding the error, this is explained in the PR - this works fine now. And I added some unit tests for the new code. Also I tried with mozregression-gui, just to be sure, and it works fine. I am inclined to land this, and to release a new version. I'm sure that we will close every bugs related to inbound hangs/being slow after that. We may (I should say we will!) have to change the taskcluster routes sometimes, when taskcluster routes will change until it become stable. But that's a fair price to pay regarding the improvements for mozregression. Will, your opinion ?

(In reply to Julien Pagès from comment #4) > Well thanks for the review and testing Will! I fixed one thing regarding the > error, this is explained in the PR - this works fine now. And I added some > unit tests for the new code. > > Also I tried with mozregression-gui, just to be sure, and it works fine. > > I am inclined to land this, and to release a new version. I'm sure that we > will close every bugs related to inbound hangs/being slow after that. We may > (I should say we will!) have to change the taskcluster routes sometimes, > when taskcluster routes will change until it become stable. But that's a > fair price to pay regarding the improvements for mozregression. > > Will, your opinion ? I trust your judgement on this. Maybe worth talking to the taskcluster developers and see what they think as well?

17:27 <parkouss> jonasfj: hello. I'm using taskcluster to find inbound builds from a changeset, like here: https://github.com/mozilla/mozregression/commit/d6593d65a18441e5ac996478486d7cc2449617a9#diff-6d21dc0490b0e260736a53c4b8066fc6R195 17:28 <parkouss> jonasfj: sometimes I got a tasknotfound error (it seems rare). do you know why ? 18:30 <&jonasfj> parkouss, do you get tasknotfound from queue or index? we only index successful tasks... Also tasks can expire, but I think we default them to one year... 18:30 <&jonasfj> I assume task not found is a 404 error... 18:31 <parkouss> jonasfj: I check 18:31 <parkouss> from the index 18:32 <parkouss> jonasfj: this fixed the issue: https://github.com/parkouss/mozregression/commit/2bb96c647c2177bd1bb36e3763bd9f59d494ac82 18:32 <parkouss> jonasfj: but I would like to understand the why 18:33 <&jonasfj> parkouss, so probably the task failed to build and wasn't indexed for that revision and platform... is my guess... 18:35 <parkouss> jonasfj: ok - this does not happen often, right ? I mean not more than on ftp 18:35 <&jonasfj> he he, I hope not... 18:36 <parkouss> jonasfj: ok! 18:36 <parkouss> jonasfj: cool, thanks for the explanation 18:37 <parkouss> jonasfj: I'm starting to migrate some code that used ftp to get inbound builds - now it will use task cluster. Do you think it is mature enough ? 18:37 <&jonasfj> parkouss, where do you get the changeset from ? I suspect some of the builds are done through buildbot (and uploaded by mozharness), hence, maybe bb has coalescence... 18:38 <parkouss> I got the changesets from json-pushes 18:38 <&jonasfj> so pushlog? 18:38 <parkouss> well I think so 18:39 <&jonasfj> index + queue is pretty solid... people are still messing a bit around adding routes... but that'll go one forever... 18:38 <parkouss> well I think so 18:39 <&jonasfj> index + queue is pretty solid... people are still messing a bit around adding routes... but that'll go one forever... 18:48 <&jonasfj> parkouss, ahh, yeah that should work unless buildbot coalesces some of them... <-- mshal if entries are missing under "buildbot.revisions.<rev>..." is that because the task failed, or does buildbot have coalescence 18:48 <&jonasfj> (background, parkouss uses pushlog to find things in index, and sometimes things are missing in index) 18:55 <mshal> jonasfj: could be either, I think? I'm not too familiar with how buildbot coalesces, but it does do that as far as I know 18:59 <parkouss> mshal, jonasfj: hm. So I was previously crwling ftp to find builds from mozilla-inbound (looking at the changesets). Can I replace that code by using taskcluster and routes like "buildbot.revsions.<rev>.mozilla-inbound..." safely ? 19:00 <mshal> parkouss: one downside to that is only the buildbot builds (not the taskcluster builds) publish to the buildbot.* routes. Until bug 1133074 is fixed, there isn't an easy way to get a build independent of where it was built 19:01 <mshal> but other than that, I think you should be able to do that (and if not, that's a shortcoming we'd want to address) 19:02 <parkouss> mshal: yeah I know about that. I thought I could use buildbot routes until the bug is fixed 19:02 <parkouss> mshal: It should be easy to change my code after that anyway, only adjusting the route 19:02 <parkouss> mshal: do you think it is ok ? 19:03 <mshal> parkouss: yeah, that should be fine - the buildbot routes won't be going away with bug 1133074, but as things get moved over to taskcluster, they just won't be updated. So if eg: we switch android builds to taskcluster, it won't be publishing to buildbot.revisions.X 19:04 <mshal> it would be good to know anyway what other issues you find, since the unified routing just fixes that one thing :) 19:05 <parkouss> mshal: yep - how can I know if you are moving android to taskcluster ? and should I be able to find the new builds using changesets ? 19:07 <parkouss> mshal: I suppose that if bug 1133074 is fixed before that, it will be easy 19:07 <parkouss> mshal: but if not, I should be informed and have a workaround - well if possible 19:09 <mshal> parkouss: android is bug 1118394. I think you could just use the taskcluster-specific routes then, something like gecko.v1.mozilla-central.revision.linux.0bc555946d99ef74b99cffd0a3aa47064b2a3df1.mulet 19:09 <parkouss> mshal: nice, thanks 19:09 <mshal> parkouss: but yeah, the idea behind 1133074 is that you shouldn't have to care if/when things move from buildbot to taskcluster, it'd just always be the same route :) 19:10 <parkouss> mshal: and that will be awesome! 19:10 <mshal> nod :) Well. :) So from what I understand, this is working now. But routes will change, until bug 1133074 is fixed. So we could wait for this bug to be fixed if you prefer Will. I would land what we have already, but it is possible that something break before bug 1133074 is fixed, so we may have to adapt the routes when they change until then.

Ok, we discussed about this and we decided to land this. Even if this may be a bit unstable, we will do our best to fix it quickly if there are issues. In the mean time, this is the way to go (as ftp will be removed in the future) plus this should improve inbounds performance! Merged this in: https://github.com/mozilla/mozregression/commit/dbd419d7dc5652adb02947867c398a3347b9d6c4 @Elbart Please note that the persist inbounds file names are no more compatible with this change - we had no choice because the timestamp we previously used are not accessible (or not equals) with what can offer the taskcluster API. So now the persist prefixes for inbounds are the changesets. Well, this time this is true, I will release a new mozregression later today or tomorrow!