Filed by: cbook [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=81884803&repo=mozilla-central https://autophone.s3.amazonaws.com/v1/task/LMEMaIfFQbeT8bmvtFeTgw/runs/0/artifacts/public/build/crashtest-webrtc-crashtests-webrtc.ini-1-pixel-05-7ca29a67-90bc-4352-bec2-7cd9b4b1edd3.log as disscued with bc this started today after the merge from m-c - seems its not bug 1308560 and green results where busted on retrigger :(

as discussed on irc

This appear to be more of an infrastructure issue unrelated to the build/merge. Other notes are AWS issues overnight and retriggers with Retry's and Busted results due to crashes. I have shutdown autophone, cleaned the logs and builds cache and restarted each server.

Note that I can see the duplicate requested jobs but not duplicate builds in staging where I was running a test at the same time: https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=966464a68a2cb3ca1125808e34abb5c1d34e3797&filter-searchStr=autophone I'm not sure that a normal Treeherder "Repeat selected Job" on production Treeherder would have appeared on staging. I presume this was initiated by one of the other options found in the pull down menu * Add new jobs * Mark with Bughunter * BuildAPI * Trigger missing jobs * Trigger all Talos jobs * Set as top of range * Set as bottom of range. I vote for one of "Add new jobs", "BuildAPI", "Trigger missing jobs", "Trigger all Talos jobs" with one of the first 3 being most likely I think. I'll run a test later on staging to see if I can reproduce.

happend btw again today like yesterday

The Unit test logs which failed showed a failure when starting where they could not abort fennec but this was due to a different issue. This also did not have anything to do with the AWS/S3 outages. The logcat output for the failing mozilla-central and mozilla-aurora builds https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b7e42143bbbc9dc3e5c05bd1e93b6485ce1d0ad4&filter-searchStr=autophone https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=027664f159bbec461390ee13aef826cd412aa791&filter-searchStr=autophone both showed repeated: 03-06 20:51:28.465 I/Gecko ( 4448): ************************* 03-06 20:51:28.465 I/Gecko ( 4448): A coding exception was thrown and uncaught in a Task. 03-06 20:51:28.465 I/Gecko ( 4448): 03-06 20:51:28.465 I/Gecko ( 4448): Full message: TypeError: osPrefs.getSystemLocale is not a function ... Which is bug 1344978 and bug 1344901 which was introduced in bug 1337551. The build log for the passing build https://public-artifacts.taskcluster.net/XqqvYKTyR1mI6-fT6EwlKg/0/public/logs/live_backing.log shows only /en-US/ while the build log for the failing build https://public-artifacts.taskcluster.net/GKPy6YG_TGGHQ5850zrVnw/0/public/logs/live_backing.log shows a number of locales. It appears the non-nightly with only en-US did not match the |if (Preferences.get(PREF_MATCH_OS_LOCALE, false)) {| in http://searchfox.org/mozilla-central/source/toolkit/modules/Locale.jsm#20 and instead followed the other branch. Thanks Tomcat!

:bc, I assume these 41 failures from yesterday are backfilling? the fix for this landed a few days ago, so I am not sure if we need more work here or if this is just old data.

Not from backfilling but from the original regression which hit two Nightly builds on m-c and one on Aurora before it was fixed. No more work to be done.

Autophone is dead.