In the last days, I experienced Decision Task failures/timeouts on try pushes frequently. Today, it only worked on the fourth attempt.

https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=294851879&revision=17bdf54eb556907310e1194393a0de6490a2daa4

https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=294855005&revision=beec17c2c8722532518477fc6e1faf3e1646f030

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a381ecb762e2fd51d9b2f38f0ebbcbe650f2b2a0&selectedJob=294858656

all failed, then finally

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1fc665bd3fd471d955dd7c816b077c9165685970

succeeded (I changed the commit message on the last one, that's why it shows a different revision, but the content was exactly the same).

This has cost me a lot of time, not sure if others are affected as well.

I've asked the Taskcluster team to look into this.
Treeherder mainly displays what happens on Taskcluster.

Hi marco, ahal,
This seems to be an issue with mach try auto.

Is there a way to make it more obvious under which component/repo should issues be filed against?

Fyi ./mach try auto is very experimental atm (we haven't announced it anywhere yet), so expect issues.

I think what's happening here is that for some reason the bugbug service is failing to compute the results for this push, then the taskgraph isn't propagating the error properly. It would also help if ./mach try auto enabled verbose logging in the Decision task to help see what's going on.

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Yikes.. I forgot to increment i in the timeout code:
https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/optimize/bugbug.py#46

So my guess was correct. I'll fix the timeout so that this doesn't wait 30 minutes to fail. Though the underlying cause seems to be that the service just isn't processing this push (it presumably keeps returning 202).

(In reply to Simon Giesecke [:sg] [he/him] from comment #4)

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Have you seen failures with specific patches, or generically? I'm going to add more logging in the bugbug service so I can more easily find out what happens when things go wrong.

Just a suggestion, while this is still experimental, instead of pushing again you could retrigger the decision task.

https://hg.mozilla.org/mozilla-central/rev/cd0c3c759c83

(In reply to Marco Castelluccio [:marco] from comment #8)

(In reply to Simon Giesecke [:sg] [he/him] from comment #4)

Oh, interesting. Sorry I didn't mention that these used mach try auto. Since it didn't fail deterministically, I thought it were an infrastructure issue. (mach try auto is incredibly useful, so it would be really great if it worked reliably)

Have you seen failures with specific patches, or generically? I'm going to add more logging in the bugbug service so I can more easily find out what happens when things go wrong.

I am not completely sure, but I guess the failed attempt were all changing quite basic things in mfbt or xpcom/ds.

(In reply to Marco Castelluccio [:marco] from comment #9)

Just a suggestion, while this is still experimental, instead of pushing again you could retrigger the decision task.

Unfortunately, due to an issue with my account, I can't retrigger any tasks at the moment. Hope this will be resolved soon.

I made quite a few improvements in the bugbug HTTP service, so this should be fixed.