1199907 - TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js | xpcshell return code: 0

Reporter

Description

•

9 years ago

This has been occurring on all comm trees lately. I first thought it might be fallout from addons signing, but unfortunately it is the dreaded nsIHttpServer not working after a request or two which I have also been getting in bug 1190012. What happens is that the first request succeeds, but all other requests just time out. As a result, the addon download never finishes and the whole test times out. http://mxr.mozilla.org/comm-central/source/mozilla/toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js I've tested this with the latest comm-central clone, where I then went into test_hotfix_cert, commented out all tasks but the first and made an exact copy of the first task. This makes the test timeout as well. I then copied the httpserver initialization into the beginning of tryInstallHotfix, so that on each task a new httpserver is started and stopped. This makes the modified test work again. The bug must therefore be some interaction between nsIHttpServer, comm-central, and xpcshell. I spent a few days trying to debug this in bug 1190012 but I really have no idea how to fix it. Patrick, as you do reviews for Core/Networking, would you be able to help out in trying to figure out the problem here? If you have some ideas on what debugging I can enable or what the problem can be, please let me know. Or maybe you can defer the needinfo to someone else.

Flags: needinfo?(mcmanus)

Comment hidden (Legacy TBPL/Treeherder Robot)

Patrick McManus [:mcmanus]

Updated

•

9 years ago

Flags: needinfo?(mcmanus) → needinfo?(jduell.mcbugs)

aleth [:aleth]

Comment 6

•

9 years ago

Fun fact: if you run this test on an IB build, it fails immediately in the second test (no timeout). Whether that means it's more or less broken there, I don't know.

Philipp Kewisch [:Fallen] [:📆][:🧩]

Reporter

Comment 7

•

9 years ago

For the record: http://hg.mozilla.org/comm-central/pushloghtml?fromchange=dd26276d7402&tochange=bdb1708f8b8c http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a69094e0f2a4&tochange=e537a1ba501b WORKS: c-c: dd26276d7402 m-c: a69094e0f2a4 FAILS: c-c: bdb1708f8b8c m-c: e537a1ba501b The c-c changesets are close together and pretty unrelated, there are a lot of m-c changesets. I don't even know if it is pure coincidence. Maybe someone with a fast machine could build with these changesets, make sure they correctly work/fail, and then bisect the m-c changesets.

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=24295 repository: comm-central start_time: 2015-09-27T04:08:45 who: mkmelin+mozilla[at]iki[dot]fi machine: tst-linux64-spot-1477 buildname: Ubuntu VM 12.04 x64 comm-central opt test xpcshell revision: fb96460ab7b2 TEST-UNEXPECTED-FAIL | mailnews/base/test/unit/test_accountMigration.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | mailnews/base/test/unit/test_accountMigration.js | run_test - [run_test : 94] 0 == 1 TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_channel.js | xpcshell return code: -11 PROCESS-CRASH | netwerk/test/unit/test_packaged_app_channel.js | application crashed [@ mozilla::net::PackagedAppVerifier::VerifyManifest(mozilla::net::PackagedAppVerifier::ResourceCacheInfo const*)] TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_verifier.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_utils.js | xpcshell return code: 0 TypeError: Cc['@mozilla.org/network/packaged-app-utils;1'] is undefined at /builds/slave/test/build/tests/xpcshell/tests/netwerk/test/unit/test_packaged_app_utils.js:83 TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_blocklist.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_blocklist.js | - revocations.txt should be as expected - "# Auto generated contents. Do not edit.\\nMCIxIDAeBgNVBAMTF0Fub3RoZXIgVGVzdCBFbmQtZW50aXR5\\n\\t2ETEb0QP574JkM+35JVwS899PLUmt1rrJy == "# Auto generated contents. Do not edit.\\nMCIxIDAeBgNVBAMTF0Fub3RoZXIgVGVzdCBFbmQtZW50aXR5\\n\\t2ETEb0QP574JkM+35JVwS899PLUmt1rrJy TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js | test_new_version - [test_new_version : 52] false == true TEST-UNEXPECTED-TIMEOUT | xpcshell-unpack.ini:toolkit/mozapps/extensions/test/xpcshell/test_webextension.js | Test timed out TEST-UNEXPECTED-TIMEOUT | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_webextension.js | Test timed out Return code: 1

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=24301 repository: comm-central start_time: 2015-09-27T04:40:47 who: mkmelin+mozilla[at]iki[dot]fi machine: t-xp32-ix-120 buildname: TB Windows XP 32-bit comm-central opt test xpcshell revision: fb96460ab7b2 TEST-UNEXPECTED-FAIL | mailnews/base/test/unit/test_accountMigration.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | mailnews/base/test/unit/test_accountMigration.js | run_test - [run_test : 94] 0 == 1 TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_channel.js | xpcshell return code: 1 PROCESS-CRASH | netwerk/test/unit/test_packaged_app_channel.js | application crashed [@ mozilla::net::PackagedAppVerifier::VerifyManifest(mozilla::net::PackagedAppVerifier::ResourceCacheInfo const *)] TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_verifier.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_packaged_app_utils.js | xpcshell return code: 0 TypeError: Cc['@mozilla.org/network/packaged-app-utils;1'] is undefined at C:/slave/test/build/tests/xpcshell/tests/netwerk/test/unit/test_packaged_app_utils.js:83 TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_blocklist.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_blocklist.js | - revocations.txt should be as expected - "# Auto generated contents. Do not edit.\\nMCIxIDAeBgNVBAMTF0Fub3RoZXIgVGVzdCBFbmQtZW50aXR5\\n\\t2ETEb0QP574JkM+35JVwS899PLUmt1rrJy == "# Auto generated contents. Do not edit.\\nMCIxIDAeBgNVBAMTF0Fub3RoZXIgVGVzdCBFbmQtZW50aXR5\\n\\t2ETEb0QP574JkM+35JVwS899PLUmt1rrJy TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_hotfix_cert.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js | xpcshell return code: 0 TEST-UNEXPECTED-FAIL | all-test-dirs.list:toolkit/mozapps/extensions/test/xpcshell/test_system_reset.js | test_new_version - [test_new_version : 52] false == true TEST-UNEXPECTED-TIMEOUT | xpcshell-unpack.ini:toolkit/mozapps/extensions/test/xpcshell/test_webextension.js | Test timed out TEST-UNEXPECTED-TIMEOUT | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_webextension.js | Test timed out Return code: 1

Magnus Melin [:mkmelin]

Updated

•

9 years ago

Keywords: regression

Comment hidden (Intermittent Failures Robot)

aleth [:aleth]

Comment 32

•

9 years ago

Might this be related to the cache patch in bug 11706464 (now backed out)? Let's see if it works now.

aleth [:aleth]

Comment 33

•

9 years ago

(In reply to aleth [:aleth] from comment #32) > Might this be related to the cache patch in bug 11706464 (now backed out)? > Let's see if it works now. Nope, still failing (at least locally).

Comment hidden (Intermittent Failures Robot)

Magnus Melin [:mkmelin]

Comment 35

•

9 years ago

0:02.58 PROCESS_OUTPUT: Thread-1 (pid:29609) "1447099520991 addons.productaddons WARN Failed downloading XM, status: 0, reason: error" 0:02.58 PROCESS_OUTPUT: Thread-1 (pid:29609) "1447099520995 addons.manager WARN Failed to update system addons: Error: Failed downloading XML from http://%(server)s/dummy-system-addons.xml .... ... So the url isn't expanded to a real server. Do we use http://mxr.mozilla.org/comm-central/source/mozilla/testing/talos/talos/config.py#144

Magnus Melin [:mkmelin]

Comment 36

•

9 years ago

The above might be unrelated, or at least irrelevant. For the failure we have now on trunk, it's a matter if singing. If i build with add-on signing enabled it works, if I build without it the test fails.

Depends on: 1168571

Magnus Melin [:mkmelin]

Comment 37

•

9 years ago

Attached patch bug1199907_hotfix_cert_test.patch (deleted) — Details — Splinter Review

Disable the test for when addon-signing is disabled. This is a bit concerning though. I don't really understand this enough to say if it's just the test that's wrong or if hotfixinig is just not working when addon signing is disabled. If it's not just a test failure, ff dev edition (and thunderbird) can't be hotfixed. xref bug 1210995

Assignee: nobody → mkmelin+mozilla

Status: NEW → ASSIGNED

Attachment #8687713 - Flags: review?(dtownsend)

Dave Townsend [:mossop]

Assignee

Comment 38

•

9 years ago

Comment on attachment 8687713 [details] [diff] [review] bug1199907_hotfix_cert_test.patch Review of attachment 8687713 [details] [diff] [review]: ----------------------------------------------------------------- Hotfixing should still be working. Hotfixes must be signed by something chaining to the AMO root if ADDON_SIGNING is enabled (dev edition) or my any cert that matches the prefs otherwise (Thunderbird). That said I'd rather we fix verifyZipSignedState to always validate the certificate for hotfix add-ons, similar to bug 1199907. Again I can do that if you want, just assign it to me.

Attachment #8687713 - Flags: review?(dtownsend) → review-

Magnus Melin [:mkmelin]

Updated

•

9 years ago

Assignee: mkmelin+mozilla → dtownsend

Component: Testing Infrastructure → Add-ons Manager

Product: Thunderbird → Toolkit

Version: 39 → 45 Branch

Dave Townsend [:mossop]

Assignee

Updated

•

9 years ago

Depends on: 1225629

Dave Townsend [:mossop]

Assignee

Updated

•

9 years ago

Status: ASSIGNED → RESOLVED

Closed: 9 years ago

Resolution: --- → FIXED

Jason Duell

Updated

•

9 years ago

Flags: needinfo?(jduell.mcbugs)