Closed Bug 781107 Opened 12 years ago Closed 9 years ago

Intermittent talosError: "Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt]" running Android Talos

Categories

(Testing :: Talos, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: philor, Unassigned)

References

Details

(Keywords: intermittent-failure)

https://tbpl.mozilla.org/php/getParsedLog.php?id=14213890&tree=Mozilla-Inbound Android Tegra 250 mozilla-inbound talos remote-ts on 2012-08-07 21:38:34 PDT for push 9e64393fa8c0 slave: tegra-168 FIRE PROC: 'org.mozilla.fennec -profile /mnt/sdcard/tests/profile http://bm-remote.build.mozilla.org/startup_test/startup_test.html?begin=1344378434627' DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory getting files in '/mnt/sdcard/tests/profile/minidumps/' removing file: /mnt/sdcard/tests/browser_output.txt DeviceManager: error pulling file '/mnt/sdcard/tests/browser_output.txt': No such file or directory NOISE: __startBeforeLaunchTimestamp1344403635134__endBeforeLaunchTimestamp NOISE: __startAfterTerminationTimestamp1344403639043__endAfterTerminationTimestamp getting files in '/mnt/sdcard/tests/profile/minidumps/' Failed ts: Stopped Tue, 07 Aug 2012 22:28:19 Traceback (most recent call last): File "run_tests.py", line 310, in run_tests talos_results.add(mytest.runTest(browser_config, test)) File "/builds/tegra-168/talos-data/talos/ttest.py", line 370, in runTest test_results.add(browser_log_filename, counter_results=counter_results) File "/builds/tegra-168/talos-data/talos/results.py", line 118, in add results = BrowserLogResults(filename=results, counter_results=counter_results, global_counters=self.global_counters).results() File "/builds/tegra-168/talos-data/talos/results.py", line 286, in __init__ self.parse() File "/builds/tegra-168/talos-data/talos/results.py", line 313, in parse self.error("Could not find report in browser output: %s" % self.report_tokens) File "/builds/tegra-168/talos-data/talos/results.py", line 298, in error raise utils.talosError(message) talosError: "Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt]" Traceback (most recent call last): File "run_tests.py", line 358, in <module> main() File "run_tests.py", line 355, in main run_tests(parser.config) File "run_tests.py", line 319, in run_tests FAIL: Busted: ts FAIL: Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt] raise e utils.talosError: "Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt]" program finished with exit code 1
Depends on: 790602
Summary: Intermittent "FAIL: Could not find report in browser output" running Android Talos → Intermittent talosError: "Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt]" running Android Talos
Depends on: 797324
Whiteboard: [orange]
Jeff, please may you take a look at this? :-)
Flags: needinfo?(jhammel)
these all are mobile related failures (panda and tegra), the majority are on remote-ts, and from the looks of it (after examining many of the most recent logs) this is a crash of fennec during startup. We should sit down and figure out a solution for detecting and reporting fennec crashes in automation.
blassey, can you assign somebody to fix this startup crash.
Flags: needinfo?(jhammel) → needinfo?(blassey.bugs)
Depends on: 810471
Summary of recent logs (since Jan 7): Comments 234, 235: Unexplained crash on startup, on mozilla-beta: see https://bugzilla.mozilla.org/show_bug.cgi?id=686245#c1344 Comment 233: Unexplained crash on startup, other.
Assignee: nobody → gbrown
Flags: needinfo?(blassey.bugs)
Review of 20 most recent reports: - 20 Android 4.0 Panda / 0 Tegra - 16 remote-tsvg / 4 remote-tp4m_nochrome / 0 other - 20 tree=Mozilla-Inbound or tree=Firefox / 0 other - 11 shutdown crashes in ssl3_DestroySSL3Info / 9 test completed but no results reported; no other evidence of crash / 0 other The 11 shutdown crashes look very much like the famous bug 761987; the remaining 9 logs could be shutdown crashes for which no stacks were reported, or could be "pandas magically reboot", bug 811444. I'm not finding anything that I can act on in this bug. Un-assigning since I am not actively working on it...but I'll be watching!
Assignee: gbrown → nobody
Most of the recent logcats have something like: 11-05 16:08:58.559 I/dalvikvm( 1370): Wrote stack traces to '/data/anr/traces.txt' 11-05 16:08:58.559 I/Process ( 1020): Sending signal. PID: 1350 SIG: 3 11-05 16:08:58.559 I/dalvikvm( 1350): threadid=3: reacting to signal 3 11-05 16:08:58.569 I/dalvikvm( 1350): Wrote stack traces to '/data/anr/traces.txt' 11-05 16:08:58.569 I/Process ( 1020): Sending signal. PID: 1269 SIG: 3 11-05 16:08:58.569 I/dalvikvm( 1269): threadid=3: reacting to signal 3 11-05 16:08:58.569 I/dalvikvm( 1269): Wrote stack traces to '/data/anr/traces.txt' 11-05 16:08:58.589 E/ActivityManager( 1020): ANR in org.mozilla.fennec 11-05 16:08:58.589 E/ActivityManager( 1020): Reason: Broadcast of Intent { act=org.mozilla.fennec.HEALTHREPORT_UPLOAD_PREF cmp=org.mozilla.fennec/org.mozilla.gecko.background.healthreport.HealthReportBroadcastReceiver (has extras) } 11-05 16:08:58.589 E/ActivityManager( 1020): Load: 0.06 / 0.13 / 0.09 11-05 16:08:58.589 E/ActivityManager( 1020): CPU usage from 10236ms to 9ms ago: 11-05 16:08:58.589 E/ActivityManager( 1020): .mozilla.fennec: 14% = 12% user + 2% kernel / faults: 22988 minor 11-05 16:08:58.589 E/ActivityManager( 1020): SUTAgentAndroid: 0% = 0% user + 0% kernel / faults: 100 minor 11-05 16:08:58.589 E/ActivityManager( 1020): system_server: 0% = 0% user + 0% kernel / faults: 38 minor 1 major 11-05 16:08:58.589 E/ActivityManager( 1020): cpufreq-dvfsd: 0% = 0% user + 0% kernel 11-05 16:08:58.589 E/ActivityManager( 1020): intr_work: 0% = 0% user + 0% kernel 11-05 16:08:58.589 E/ActivityManager( 1020): c.UpdateService: 0% = 0% user + 0% kernel / faults: 21 minor 11-05 16:08:58.589 E/ActivityManager( 1020): mmcqd: 0% = 0% user + 0% kernel 11-05 16:08:58.589 E/ActivityManager( 1020): putmethod.latin: 0% = 0% user + 0% kernel / faults: 29 minor 11-05 16:08:58.589 E/ActivityManager( 1020): ndroid.launcher: 0% = 0% user + 0% kernel / faults: 20 minor 11-05 16:08:58.589 E/ActivityManager( 1020): d.process.media: 0% = 0% user + 0% kernel / faults: 16 minor 11-05 16:08:58.589 E/ActivityManager( 1020): m.android.email: 0% = 0% user + 0% kernel / faults: 27 minor 11-05 16:08:58.589 E/ActivityManager( 1020): .quicksearchbox: 0% = 0% user + 0% kernel / faults: 23 minor 11-05 16:08:58.589 E/ActivityManager( 1020): m.android.music: 0% = 0% user + 0% kernel / faults: 21 minor 11-05 16:08:58.589 E/ActivityManager( 1020): id.defcontainer: 0% = 0% user + 0% kernel / faults: 20 minor 11-05 16:08:58.589 E/ActivityManager( 1020): TOTAL: 23% = 13% user + 3% kernel + 1% iowait + 0% irq + 5% softirq
should we turn of health reporting?
tspaint is not doing well on Panda or Tegra. The Panda logs show: 11:16:16 INFO - mozcrash INFO | Saved dump as /builds/panda-0563/test/build/blobber_upload_dir/6b6a1b03-346d-2e69-71fb4305-08a87153.dmp 11:16:16 INFO - browser_name:Fennec 11:16:16 INFO - browser_version:29.0a1 11:16:16 INFO - buildID:20131217100424 11:16:16 INFO - PROCESS-CRASH | ts_paint | application crashed [Unknown top frame] 11:16:16 INFO - Crash dump filename: /tmp/tmpzcGNJc/6b6a1b03-346d-2e69-71fb4305-08a87153.dmp 11:16:16 INFO - stderr from minidump_stackwalk: 11:16:16 INFO - 2013-12-17 11:16:16: minidump_processor.cc:264: INFO: Processing minidump in file /tmp/tmpzcGNJc/6b6a1b03-346d-2e69-71fb4305-08a87153.dmp 11:16:16 INFO - 2013-12-17 11:16:16: minidump.cc:3815: INFO: Minidump opened minidump /tmp/tmpzcGNJc/6b6a1b03-346d-2e69-71fb4305-08a87153.dmp 11:16:16 INFO - 2013-12-17 11:16:16: minidump.cc:3847: ERROR: Minidump header signature mismatch: (0x0, 0x0) != 0x504d444d 11:16:16 INFO - 2013-12-17 11:16:16: minidump_processor.cc:268: ERROR: Minidump /tmp/tmpzcGNJc/6b6a1b03-346d-2e69-71fb4305-08a87153.dmp could not be read 11:16:16 INFO - 2013-12-17 11:16:16: minidump.cc:3787: INFO: Minidump closing minidump 11:16:16 INFO - 2013-12-17 11:16:16: minidump_stackwalk.cc:529: ERROR: MinidumpProcessor::Process failed 11:16:16 INFO - minidump_stackwalk exited with return code 1 and Panda logcats have: 08:29:20 INFO - 12-17 08:26:29.484 D/GeckoLayerClient( 3649): Screen-size changed to (1280,672) 08:29:20 INFO - 12-17 08:26:29.484 D/GeckoLayerClient( 3649): Window-size changed to (1280,616) 08:29:20 INFO - 12-17 08:26:29.484 W/OrientationEventListener( 3649): Cannot detect sensors. Not enabled 08:29:20 INFO - 12-17 08:26:29.507 E/GeckoConsole( 3649): Adding HealthReport:RequestSnapshot observer. 08:29:20 INFO - 12-17 08:26:29.523 W/GeckoGLController( 3649): GLController::updateCompositor with mCompositorCreated=false 08:29:20 INFO - 12-17 08:26:30.578 D/GeckoAppShell( 3649): Gecko event sync taking too long: 1053ms 08:29:20 INFO - 12-17 08:26:31.507 I/SUTAgentAndroid( 1898): 10.12.135.18 : ps 08:29:20 INFO - 12-17 08:26:31.765 D/GeckoAppShell( 3649): Gecko event sync taking too long: 2238ms 08:29:20 INFO - 12-17 08:26:32.773 D/GeckoAppShell( 3649): Gecko event sync taking too long: 3245ms ... 08:29:20 INFO - 12-17 08:27:57.382 I/Process ( 1401): Sending signal. PID: 1590 SIG: 3 08:29:20 INFO - 12-17 08:27:57.382 I/dalvikvm( 1590): threadid=3: reacting to signal 3 08:29:20 INFO - 12-17 08:27:57.398 I/dalvikvm( 1590): Wrote stack traces to '/data/anr/traces.txt' 08:29:20 INFO - 12-17 08:27:59.257 D/GeckoAppShell( 3649): Gecko event sync taking too long: 89729ms 08:29:20 INFO - 12-17 08:27:59.445 I/Watchdog_N( 1401): dumpKernelStacks 08:29:20 INFO - 12-17 08:27:59.578 I/dalvikvm-heap( 1401): Grow heap (frag case) to 9.120MB for 99800-byte allocation 08:29:20 INFO - 12-17 08:27:59.687 I/dalvikvm-heap( 1401): Grow heap (frag case) to 9.252MB for 199584-byte allocation 08:29:20 INFO - 12-17 08:28:00.562 D/GeckoAppShell( 3649): Gecko event sync taking too long: 91034ms 08:29:20 INFO - 12-17 08:28:02.320 I/Process ( 1401): Sending signal. PID: 1401 SIG: 9 08:29:20 INFO - 12-17 08:28:02.320 D/GeckoAppShell( 3649): Gecko event sync taking too long: 92800ms 08:29:20 INFO - 12-17 08:28:02.320 W/Watchdog( 1401): *** WATCHDOG KILLING SYSTEM PROCESS: null Tegra logcats typically show: 2-16 13:57:21.388 D/GeckoTabs( 4755): handleMessage: DOMTitleChanged 12-16 13:57:21.408 D/GeckoTabs( 4755): handleMessage: DOMContentLoaded 12-16 13:57:21.418 D/GeckoTabs( 4755): handleMessage: Content:PageShow 12-16 13:57:21.448 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:21.448 D/GeckoTabs( 4755): handleMessage: Tab:ViewportMetadata 12-16 13:57:21.448 D/GeckoTabs( 4755): handleMessage: Content:StateChange 12-16 13:57:21.448 E/Profiler( 4755): BPUnw: [6 total] thread_register_for_profiling(me=0x2c54b8, stacktop=0x5d2ffe03) 12-16 13:57:21.458 E/Profiler( 4755): BPUnw: [7 total] thread_register_for_profiling(me=0x2c54f8, stacktop=0x5d3ffdd7) 12-16 13:57:21.528 E/Profiler( 4755): BPUnw: [8 total] thread_register_for_profiling(me=0x2c5538, stacktop=0x5d4ffcef) 12-16 13:57:21.635 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:21.945 D/GeckoThumbnailHelper( 4755): Using new thumbnail size: 58240 (width 160) 12-16 13:57:21.945 D/GeckoThumbnailHelper( 4755): Sending thumbnail event: 160, 91 12-16 13:57:22.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:22.637 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:22.637 W/ActivityManager( 1020): Activity idle timeout for HistoryRecord{48621ae8 org.mozilla.fennec/.App} 12-16 13:57:22.676 I/SUTAgentAndroid( 1487): 10.250.49.155 : ps 12-16 13:57:23.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:23.643 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:24.415 D/GeckoThumbnailHelper( 4755): handleThumbnailData: 58240 12-16 13:57:24.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:24.637 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:25.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:25.639 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:26.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:26.635 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:27.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:27.638 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:27.696 I/SUTAgentAndroid( 1487): 10.250.49.155 : ps 12-16 13:57:28.400 W/ActivityManager( 1020): Timeout of broadcast BroadcastRecord{487717c0 org.mozilla.fennec.HEALTHREPORT_UPLOAD_PREF} - receiver=android.os.BinderProxy@485ae4a0 12-16 13:57:28.400 W/ActivityManager( 1020): Receiver during timeout: ResolveInfo{486867e8 org.mozilla.gecko.background.healthreport.HealthReportBroadcastReceiver p=0 o=0 m=0x108000} 12-16 13:57:28.405 I/Process ( 1020): Sending signal. PID: 4755 SIG: 3 12-16 13:57:28.405 I/dalvikvm( 4755): threadid=3: reacting to signal 3 12-16 13:57:28.415 I/dalvikvm( 4755): Wrote stack traces to '/data/anr/traces.txt' 12-16 13:57:28.415 I/Process ( 1020): Sending signal. PID: 1020 SIG: 3 12-16 13:57:28.415 I/dalvikvm( 1020): threadid=3: reacting to signal 3 12-16 13:57:28.445 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=51, status=0). CPU may be pegged. trying again. 12-16 13:57:28.455 I/dalvikvm( 1020): Wrote stack traces to '/data/anr/traces.txt' 12-16 13:57:28.455 I/Process ( 1020): Sending signal. PID: 1487 SIG: 3 12-16 13:57:28.455 I/dalvikvm( 1487): threadid=3: reacting to signal 3 12-16 13:57:28.465 I/dalvikvm( 1487): Wrote stack traces to '/data/anr/traces.txt' ... 12-16 13:57:28.585 E/ActivityManager( 1020): ANR in org.mozilla.fennec 12-16 13:57:28.585 E/ActivityManager( 1020): Reason: Broadcast of Intent { act=org.mozilla.fennec.HEALTHREPORT_UPLOAD_PREF cmp=org.mozilla.fennec/org.mozilla.gecko.background.healthreport.HealthReportBroadcastReceiver (has extras) } 12-16 13:57:28.585 E/ActivityManager( 1020): Load: 1.4 / 1.18 / 1.11 12-16 13:57:28.585 E/ActivityManager( 1020): CPU usage from 10225ms to 9ms ago: 12-16 13:57:28.585 E/ActivityManager( 1020): zygote: 15% = 13% user + 2% kernel / faults: 22847 minor 12-16 13:57:28.585 E/ActivityManager( 1020): SUTAgentAndroid: 0% = 0% user + 0% kernel / faults: 101 minor 12-16 13:57:28.585 E/ActivityManager( 1020): system_server: 0% = 0% user + 0% kernel / faults: 44 minor 1 major 12-16 13:57:28.585 E/ActivityManager( 1020): cpufreq-dvfsd: 0% = 0% user + 0% kernel 12-16 13:57:28.585 E/ActivityManager( 1020): mmcqd: 0% = 0% user + 0% kernel 12-16 13:57:28.585 E/ActivityManager( 1020): putmethod.latin: 0% = 0% user + 0% kernel / faults: 28 minor 12-16 13:57:28.585 E/ActivityManager( 1020): m.android.phone: 0% = 0% user + 0% kernel / faults: 26 minor 12-16 13:57:28.585 E/ActivityManager( 1020): m.android.email: 0% = 0% user + 0% kernel / faults: 28 minor 12-16 13:57:28.585 E/ActivityManager( 1020): mozilla.watcher: 0% = 0% user + 0% kernel / faults: 17 minor 12-16 13:57:28.585 E/ActivityManager( 1020): .quicksearchbox: 0% = 0% user + 0% kernel / faults: 30 minor 12-16 13:57:28.585 E/ActivityManager( 1020): .cooliris.media: 0% = 0% user + 0% kernel / faults: 26 minor 12-16 13:57:28.585 E/ActivityManager( 1020): com.svox.pico: 0% = 0% user + 0% kernel / faults: 20 minor 12-16 13:57:28.585 E/ActivityManager( 1020): c.UpdateService: 0% = 0% user + 0% kernel / faults: 20 minor 12-16 13:57:28.585 E/ActivityManager( 1020): TOTAL: 32% = 14% user + 3% kernel + 10% iowait + 0% irq + 4% softirq 12-16 13:57:28.635 W/SharedBufferStack( 1020): waitForCondition(ReallocateCondition) timed out (identity=49, status=0). CPU may be pegged. trying again. 12-16 13:57:28.645 I/dalvikvm-heap( 1020): Grow heap (frag case) to 8.100MB for 119356-byte allocation 12-16 13:57:28.795 I/GeckoANRReporter( 4755): processing Gecko ANR
Joel, any ideas about the failures here? They seem to be mainly limited to talos-remote-tspaint.
Flags: needinfo?(jmaher)
yeah, I have no idea why these fail, looking at the last 3 logs in here, I see: 18:42:43 INFO - 01-13 18:41:24.210 W/Watchdog( 1401): *** WATCHDOG KILLING SYSTEM PROCESS: null then all the services shutdown. Is it possible that we are hitting limitations on our panda boards? maybe specific boards? we could turn off health reporting (as I suggested a long time ago on this bug). gbrown, any other ideas?
Flags: needinfo?(jmaher) → needinfo?(gbrown)
The last time we were having "WATCHDOG KILLING SYSTEM PROCESS" problems was with Panda robocop tests, and we "fixed" that by splitting robocop tests into more chunks -- it seemed related to the length of time that the test ran. I notice that ts-paint seems to be one of the longest-running tests we have on Pandas now. Can it be shortened? Run fewer iterations, load fewer pages?
Flags: needinfo?(gbrown)
ah, I didn't look at the time- in fact this is running for 40+ minutes and apparently not loading any pages. In fact this just loads a really simple page 20 times- maybe since we are not running with an addon this is causing problems since the commandline hander for fennec might be not as robust as desktop. edmorley- how often does this happen? once every 20 times, 40 times, 5 times?
Flags: needinfo?(emorley)
(In reply to Joel Maher (:jmaher) from comment #1513) > edmorley- how often does this happen? once every 20 times, 40 times, 5 > times? Looking at the last 60 non-blue jobs here: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=tspaint 4 failed with this failure mode.
Flags: needinfo?(emorley)
some ideas: 1) we are not receiving the mozafterpaint event, therefore timing out 2) we are not starting up properly 3) we cannot handle simple commandline arguments I am voting for #1 and have a patch on try server that will help me figure this out. If #1 is the main cause of this failure, then we have a development issue- in that case we can either hack the harness to work around it (not ideal) or stop using mozafterpaint!
looking at the trends, I would expect to see this within 50 retriggers on try server (panda, not tegra), and I didn't see it. I was trying out a method where I would log if a mozAfterPaint wasn't received after 5 seconds, I never saw that message nor did I see this error. I vote to see what happens over time and if we continue to see a trend of this being in the top 25 oranges over the next couple weeks.
(In reply to Joel Maher (:jmaher) from comment #1526) > looking at the trends, I would expect to see this within 50 retriggers on > try server (panda, not tegra), and I didn't see it. I was trying out a > method where I would log if a mozAfterPaint wasn't received after 5 seconds, > I never saw that message nor did I see this error. > > I vote to see what happens over time and if we continue to see a trend of > this being in the top 25 oranges over the next couple weeks. We just enabled automated GTK3 try runs some days ago.. And this fail seems to be happenning systematically on GTK3 with OMTC basic, that is with this set of prefs : pref("layers.offmainthreadcomposition.enabled", true); pref("layers.offmainthreadcomposition.force-basic", true); See my last recent try push with this configuration : https://tbpl.mozilla.org/?tree=Try&rev=77f8cec08771 cheers,
it sounds like a lot more is broken right now. I suspect when the unittests are green the talos jobs will be as well. We can set those mentioned prefs in talos if we need to :)
Might want to work on slightly more expressive failure messages, inbound's been broken since last night but nobody noticed since it was just another two-year-old intermittent.
Depends on: 1062427
Depends on: 1088019
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.