Marionette has a number of tests with expected failures. Currently, the failures in these tests are logged with TEST-UNEXPECTED-FAIL, which is misleading and gets picked up by TBPL as a real failure.

posted to try: https://tbpl.mozilla.org/?tree=Try&rev=0725231f95b6

Try run shows this patch solves this problem.

Comment on attachment 661372 [details] [diff] [review] Allow tests to provide custom pass/fail strings for is/isnot/ok assertions I don't love this approach but can't think of a better way to handle it. This basically allows a test to define its own status strings for pass/fail.

Try run for 0725231f95b6 is complete. Detailed breakdown of the results available here: https://tbpl.mozilla.org/?tree=Try&rev=0725231f95b6 Results (out of 2 total builds): success: 2 Builds (or logs if builds failed) available at: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jgriffin@mozilla.com-0725231f95b6

Comment on attachment 661372 [details] [diff] [review] Allow tests to provide custom pass/fail strings for is/isnot/ok assertions Review of attachment 661372 [details] [diff] [review]: ----------------------------------------------------------------- Yeah, I'm not 100% happy with this either, but I can't think of a nicer way of doing it. If there isn't any other purpose to passing in your own logging lines than these expected-failures, perhaps we can add a function to toggle logging on and off, and log what we want for these particular tests, but I'm not sure that's any nicer.

(In reply to Malini Das [:mdas] from comment #5) perhaps we can add a function to toggle > logging on and off, and log what we want for these particular tests, but I'm > not sure that's any nicer. Yeah, I'm not sure either. I landed the existing patch as https://hg.mozilla.org/integration/mozilla-inbound/rev/1b9df2fa85a9