Filed by: philringnalda [at] gmail.com https://treeherder.mozilla.org/logviewer.html#?job_id=82321584&repo=autoland https://queue.taskcluster.net/v1/task/cfOk86KfR8-4Dv1znro3yQ/runs/0/artifacts/public/logs/live_backing.log

Screenshot as in bug 1336541 - related? Also notice that the first run dialog for tab queues is displayed...that seems unexpected.

The job here is Android Debug mochitest-32. This is perma-failing on autoland starting with https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=4dc0edd4681c71fb0e67bc19bbc290616952fdf7&filter-searchStr=mochitest-32...but it was running long (90-100 minutes) in earlier pushes. There was a significant increase in run time a few days earlier, at https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=mochitest-32&tochange=a793136c90bc5d32f2c82aa7dea4bc300c4f1836&fromchange=31c09bb63b697089f9866b7cbfa57b8900df2b1d.

Most of the Android 4.3 Debug mochitest jobs run in 40 to 60 minutes, but mochitest-31 and mochitest-32 are closer to 90 minutes.

This looks to be a regression from bug 1335751: at least some Android tests started running a little slower with that landing: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=android%20mochitest-32&tochange=c10963d3a6872149f753738c265ef56b4394dda0&fromchange=340e7f8a7229786fc8808d4a78983338997f81f3 There was an earlier landing (later backed out) with the same result: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&tochange=3c4a6d36149c38695e0cdccd9e7bbe261d1e17c8&fromchange=8ab76e34613bf4af022e05d14ed5f75529ed951d&filter-searchStr=mochitest-32 I don't think there's anything "wrong" with bug 1335751, but let's just check in there. :jonco - Just wanted to make you aware that some Android mochitests run slower with your patches; if that is surprising or concerning to you, let me know. Probably the best action here is to add more test chunks so we have the capacity to run longer.

At 40 chunks there are still several chunks taking longer than 70 minutes. 48 chunks seems to bring almost all jobs into the 30 - 60 minute range. https://treeherder.mozilla.org/#/jobs?repo=try&revision=97798692b4534acc5f68fc10d0e300790edb4bf5

Comment on attachment 8844988 [details] [diff] [review] increase android debug mochitest chunks to 48 Review of attachment 8844988 [details] [diff] [review]: ----------------------------------------------------------------- insane, but needed.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/ec475e2b1915 Increase Android debug mochitest chunks; r=jmaher

Looks like it worked. No chunks exceeded the max runtime.

(In reply to Geoff Brown [:gbrown] from comment #4) > I don't think there's anything "wrong" with bug 1335751, but let's just > check in there. :jonco - Just wanted to make you aware that some Android > mochitests run slower with your patches; if that is surprising or concerning > to you, let me know. (jonco is on PTO) Yes, it added some slower but more complete gray marking checks.