A soon to be deployed TBPL parser improvement has found a whole bunch of log spam in green Windows debug mochitest runs. a) Should these be making the test run fail? b) We should either catch these or fix them so we don't spam the annotated failure summary once this TBPL patchset is rolled out. eg: https://tbpl-dev.allizom.org/php/getParsedLog.php?id=25602000&tree=Mozilla-Central#error2 Windows 7 32-bit mozilla-central debug test mochitest-1 on 2013-07-23 02:56:01 PDT for push fb4bf993a58a slave: t-w732-ix-098 { 03:00:16 INFO - 3306 INFO TEST-START | /tests/browser/base/content/test/test_offline_gzip.html 03:00:16 INFO - ++DOMWINDOW == 91 (0D3BA898) [serial = 132] [outer = 0CD9CDA0] 03:00:16 INFO - ++DOCSHELL 055276C8 == 24 [id = 28] 03:00:16 INFO - ++DOMWINDOW == 92 (12FD0190) [serial = 133] [outer = 00000000] 03:00:16 INFO - WARNING: NS_ENSURE_TRUE(NS_SUCCEEDED(rv) && subjPrincipal) failed: file e:/builds/moz2_slave/m-cen-w32-d-000000000000000000/build/docshell/base/nsDocShell.cpp, line 8392 03:00:16 INFO - ++DOMWINDOW == 93 (12FD19A0) [serial = 134] [outer = 12FD0190] 03:00:16 INFO - WARNING: 1 sort operation has occurred for the SQL statement '0xd16d890'. See https://developer.mozilla.org/En/Storage/Warnings details.: file e:/builds/moz2_slave/m-cen-w32-d-000000000000000000/build/storage/src/mozStoragePrivateHelpers.cpp, line 110 03:00:16 INFO - WARNING: 1 sort operation has occurred for the SQL statement '0xd16d590'. See https://developer.mozilla.org/En/Storage/Warnings details.: file e:/builds/moz2_slave/m-cen-w32-d-000000000000000000/build/storage/src/mozStoragePrivateHelpers.cpp, line 110 03:00:16 INFO - WARNING: 1 sort operation has occurred for the SQL statement '0xd16d890'. See https://developer.mozilla.org/En/Storage/Warnings details.: file e:/builds/moz2_slave/m-cen-w32-d-000000000000000000/build/storage/src/mozStoragePrivateHelpers.cpp, line 110 03:00:16 INFO - WARNING: 1 sort operation has occurred for the SQL statement '0xd16d590'. See https://developer.mozilla.org/En/Storage/Warnings details.: file e:/builds/moz2_slave/m-cen-w32-d-000000000000000000/build/storage/src/mozStoragePrivateHelpers.cpp, line 110 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:70 03:00:16 INFO - NS_ERROR_XPC_GS_RETURNED_FAILURE: Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService] 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:768 03:00:16 INFO - ReferenceError: gApp is not defined 03:00:16 INFO - System JS : ERROR j3307 INFO TEST-PASS | /tests/browser/base/content/test/test_offline_gzip.html | Child was successfully cached. }

http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/nsBlocklistService.js#70 69 XPCOMUtils.defineLazyGetter(this, "gApp", function bls_gApp() { 70 return Cc["@mozilla.org/xre/app-info;1"].getService(Ci.nsIXULAppInfo). 71 QueryInterface(Ci.nsIXULRuntime); 72 });

Uh... I have utterly no idea. Maybe Dave has a clue?

I have no idea either, and it's quite worrying as if that were happening it would mean the blocklist service probably didn't work at all. But we know it does... Is it at all possible that these logs are coming from a plugin process I wonder

Note that https://hg.mozilla.org/webtools/tbpl/rev/eb85b76b7b52 has just gone into TBPL production that manually excludes this failure from the annotated summaries/top of log summary (it will still appear when searching in the main body of the log), just to reduce the spam.

Well, I have no idea why this is happening... but I'd agree with bug 900333 comment 0 that the blocklist should probably be disabled for mochitests anyway. So, next dumb question: Where do we do that from? mozprofile/profile.py or somewhere else?

Ok, I assume I've got this right... Try run pending.

Comment on attachment 786068 [details] [diff] [review] Dsable blocklist Review of attachment 786068 [details] [diff] [review]: ----------------------------------------------------------------- Let's just pretend that I'm a peer of this stuff

This is of course assuming a try run doesn't reveal that some of our UI tests rely on the blocklist working...

(In reply to Dave Townsend (:Mossop) from comment #9) > This is of course assuming a try run doesn't reveal that some of our UI > tests rely on the blocklist working... *cough* Yes. Well. Turns out...

FTR: https://tbpl.mozilla.org/?tree=Try&rev=2860ddc61745 Most notably test_bug391728.html and a few browser-chrome tests, which turns out are affected too.

Comment on attachment 786068 [details] [diff] [review] Dsable blocklist Unfortunately the error is still present in the log: https://tbpl.mozilla.org/php/getParsedLog.php?id=26192250&tree=Try (need to search, isn't in summary, since has been temporarily excluded in the TBPL parser)

Current theory: Background thumbnails now use a remote browser, and that seems to be hitting the blocklist service. The blocklist service is getting upset about this.

(In reply to Blair McBride [:Unfocused] from comment #13) > Current theory: Background thumbnails now use a remote browser, and that > seems to be hitting the blocklist service. The blocklist service is getting > upset about this. I thought plugins were disabled there, or does it still do a plugin scan even when it's never going to use any of them?

Thank you for looking at this btw :-)

(In reply to Dave Townsend (:Mossop) from comment #14) > I thought plugins were disabled there, or does it still do a plugin scan > even when it's never going to use any of them? Yea, something like that. Bug 901294 has some details - I'm hopeful the patch in there could fix this. However, for bonus confusion: Same error has been seen in bug 890555, in toolkit/devtools/server/tests/unit/test_sourcemaps-07.js

(In reply to Dave Townsend (:Mossop) from comment #14) > (In reply to Blair McBride [:Unfocused] from comment #13) > > Current theory: Background thumbnails now use a remote browser, and that > > seems to be hitting the blocklist service. The blocklist service is getting > > upset about this. > > I thought plugins were disabled there, or does it still do a plugin scan > even when it's never going to use any of them? Bug 901294 has found that even though the docShell is set to not use plugins, the process of loading the page with a plugin still ends up finding the plugin that *would* be used, which causes them to be enumerated and the blocklist checked - all in the child process. The attachment added in bug 901294 comment 16 prevents this - so maybe some try runs with that in-place could be tried?

Looks like bug 903965 didn't have any effect on this.

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #18) > Looks like bug 903965 didn't have any effect on this.

*sigh* Then I have no clue. Thankfully this isn't affecting anyone, right? Obviously needs fixed... but it's ignored by TBPL, and it's a test-only error for tests unrelated to the blocklist.

(In reply to Blair McBride [:Unfocused] from comment #20) > but it's ignored by TBPL, and it's a test-only error for tests unrelated to the > blocklist. We would like to make these exceptions fatal eventually as we've already seen comments in many other bugs that these were pointing to real bugs.

Yea, I know - there's no way I want to leave this unfixed for too long. Just trying to figure out priorities.

I'm not seeing this in the logs on m-c or Aurora anymore.

So... that good enough to call it fixed? RESOLVED GHOSTINTHEMACHINE.

Sure

I was just getting ready to file a bug and couldn't get the thought out of my head I'd seen this before: https://tbpl.mozilla.org/php/getParsedLog.php?id=34423294&tree=Mozilla-Inbound WINNT 6.2 mozilla-inbound debug test mochitest-1 on 2014-02-10 07:23:41 PST for push 64b3f71d79a8 slave: t-w864-ix-050 07:32:10 INFO - 683 INFO TEST-START | /tests/content/base/test/test_bug338583.html 07:32:10 INFO - ++DOMWINDOW == 75 (1AA14EA8) [pid = 3872] [serial = 725] [outer = 10682FD8] 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004004: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\EventSource.cpp, line 339 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004004: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\EventSource.cpp, line 339 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004004: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\EventSource.cpp, line 339 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80040111: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\nsCrossSiteListenerProxy.cpp, line 569 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x805303F4: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\caps\src\nsScriptSecurityManager.cpp, line 703 07:32:11 INFO - [3872] WARNING: '!fd.IsInitialized()', file c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/base/src/nsSocketTransport2.cpp, line 2601 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(status, status) failed with result 0x804B001E: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\nsCrossSiteListenerProxy.cpp, line 559 07:32:11 INFO - [3872] WARNING: 'NS_FAILED(rv)', file c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/protocol/http/nsHttpConnection.cpp, line 1662 07:32:11 INFO - [3872] WARNING: 'NS_FAILED(rv2)', file c:/builds/moz2_slave/m-in-w32-d-0000000000000000000/build/netwerk/protocol/http/nsHttpConnection.cpp, line 371 07:32:11 INFO - [3872] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80040111: file c:\builds\moz2_slave\m-in-w32-d-0000000000000000000\build\content\base\src\EventSource.cpp, line 346 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined 07:32:11 INFO - System JS : ERROR jar:file:///C:/slave/test/build/application/firefox/omni.ja!/components/nsBlocklistService.js:831 - ReferenceError: gApp is not defined

I can basically reproduce this every time when running mochitest-chrome What I can tell is that it happens when nsPluginTag::GetBlocklistState is examining Flash, and that such call doesn't happen in the browser process since breakpoints I set are not hit. the js stack seems to be coming from httpd.js::_getTypeFromFile that calls into nsExternalHelperAppService::GetTypeFromFile that calls GetTypeFromExtension that calls IsPluginEnabledForExtension... so I guess this ends up trying to use the blocklist service from the httpd.js process.

Taking... I think I have a decent solution for this. Try run https://tbpl.mozilla.org/?tree=Try&rev=dd759f534853

Waiting on try results before requesting review

Fix and a new try run https://tbpl.mozilla.org/?tree=Try&rev=da745be86aec

Comment on attachment 8567547 [details] [diff] [review] patch Try run is looking good https://treeherder.mozilla.org/#/jobs?repo=try&revision=6bf57a7f4dee

Comment on attachment 8567547 [details] [diff] [review] patch Review of attachment 8567547 [details] [diff] [review]: ----------------------------------------------------------------- ::: toolkit/mozapps/extensions/nsBlocklistService.js @@ +550,4 @@ > dsURI = dsURI.replace(/%PRODUCT%/g, gApp.name); > + // Not all applications implement nsIXULAppInfo (e.g. xpcshell doesn't). > + if (gApp.version) > + dsURI = dsURI.replace(/%VERSION%/g, gApp.version); Are there cases where we hit these?

(In reply to Dave Townsend [:mossop] from comment #32) > Comment on attachment 8567547 [details] [diff] [review] > patch > > Review of attachment 8567547 [details] [diff] [review]: > ----------------------------------------------------------------- > > ::: toolkit/mozapps/extensions/nsBlocklistService.js > @@ +550,4 @@ > > dsURI = dsURI.replace(/%PRODUCT%/g, gApp.name); > > + // Not all applications implement nsIXULAppInfo (e.g. xpcshell doesn't). > > + if (gApp.version) > > + dsURI = dsURI.replace(/%VERSION%/g, gApp.version); > > Are there cases where we hit these? Yes, these are hit when running mochitest-chrome tests via httpd.js per comment #27.

Pushed to mozilla-inbound https://hg.mozilla.org/integration/mozilla-inbound/rev/85fcbcc518b9