Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=226037795&repo=mozilla-inbound

16:28:56 INFO - TEST-START | security/manager/ssl/tests/unit/test_cert_chains.js
16:28:56 WARNING - TEST-UNEXPECTED-FAIL | security/manager/ssl/tests/unit/test_cert_chains.js | xpcshell return code: 0
16:28:56 INFO - TEST-INFO took 275ms
16:28:56 INFO - >>>>>>>
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
16:28:56 INFO - running event loop
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - Binary util BadCertServer should exist - true == true
16:28:56 INFO - TEST-PASS | security/manager/ssl/tests/unit/test_cert_chains.js | - certificate folder (bad_certs) should exist - true == true
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
16:28:56 INFO - "CONSOLE_MESSAGE: (info) No chrome package registered for chrome://branding/locale/brand.properties"
16:28:56 INFO - PID 7102 | sending 'GET / HTTP/1.0
16:28:56 INFO - PID 7102 | '
16:28:56 INFO - (xpcshell/head.js) | test pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - PID 7102 | HTTP/1.0 200 OK
16:28:56 INFO - PID 7102 | content-type: text/plain
16:28:56 INFO - PID 7102 | connection: close
16:28:56 INFO - PID 7102 | server: httpd.js
16:28:56 INFO - PID 7102 | date: Tue, 05 Feb 2019 00:28:56 GMT
16:28:56 INFO - PID 7102 | content-length: 3
16:28:56 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
16:28:56 INFO - (xpcshell/head.js) | test finished (2)
16:28:56 INFO - security/manager/ssl/tests/unit/test_cert_chains.js | Starting
16:28:56 INFO - (xpcshell/head.js) | test pending (2)

Looks like someone needs to run[1]:

./mach python build/pgo/genpgocert.py

and check in the result but I'm not an expert in this area.

[1] https://searchfox.org/mozilla-central/rev/152993fa346c8fd9296e4cd6622234a664f53341/build/pgo/certs/README#4-5

If it helps, this is what that mach command generated for me on a recent checkout of m-c.

That will certainly help. A number of xpcshell test certificates need to be regenerated as well. I was hoping it would be a quick run-the-utility-and-replace-the-certs kind of thing, but it looks like some of the tests need slightly more involved changes (no overlap in validity periods for some fixed-period certificates and their (non-fixed-period) CAs, etc.). I won't have time to do this today, but I can first thing tomorrow morning.

Scripts:
https://gist.github.com/jcjones/b25e07de3a48c3ed084f0f9e26911693

From the above gist

./jcj-regenerate-certspecs

This is a DER form, not a PEM.

openssl x509 -in security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem -outform der > security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.der
rm security/manager/ssl/tests/unit/test_signed_apps/xpcshellTestRoot.pem

These don't seem to be checked in

rm services/common/tests/unit/test_blocklist_signatures/*.pem

:keller can you please take a look at Comment 6?

Unfortunately, the issue is still present:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&revision=b6ec07118c7058cbd2eaf9b7aa741528b1650047&selectedJob=226092376

:jandem did a Try push: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=28ec774d150497723acb9c4d071c716837bf3deb

Depends on D18661

Depends on D18663

Depends on D18664

Depends on D18665

Depends on D18666

Depends on D18667

https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3
https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968
https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce
https://hg.mozilla.org/mozilla-central/rev/aae349bf4115
https://hg.mozilla.org/mozilla-central/rev/52b73d447c52

I ended up fixing the test failures this time, but can we please automate this process so we don't have to do this again next year?

Anyway, some notes:

• Part 2 (test_signed_apps zip files) was necessary to fix test_signed_apps.js

I generated the zip files by uncommenting the code in security/manager/ssl/tests/unit/test_signed_apps/moz.build and then doing a build and copying the files from the obj dir to the source dir.

• Part 3 (ssl_error_reports.sjs) was necessary to fix browser_ssl_error_reports.js For this I set DEBUG to true in httpd.js and dumped the data we got to figure out the new value for this.

• Part 4 backed out some .pem changes to fix tests that depended on the old files.

• Part 5 (test_x509.js). The serial number and dates can be printed like this:

  openssl x509 -in security/manager/ssl/tests/unit/bad_certs/default-ee.pem -text -noout

Maybe someone actually familiar with the code should double check all this.

https://hg.mozilla.org/releases/mozilla-beta/rev/f4e6efbb6b2e
https://hg.mozilla.org/releases/mozilla-beta/rev/e3f0663bac74
https://hg.mozilla.org/releases/mozilla-beta/rev/da5d5c54fd89
https://hg.mozilla.org/releases/mozilla-beta/rev/87aa24e0ddf8
https://hg.mozilla.org/releases/mozilla-beta/rev/d701840bb03f
https://hg.mozilla.org/releases/mozilla-beta/rev/bf0066dc9314
https://hg.mozilla.org/releases/mozilla-beta/rev/a1087f95eeb8
https://hg.mozilla.org/releases/mozilla-beta/rev/9689c6dc5590
https://hg.mozilla.org/releases/mozilla-beta/rev/1a64a0b6a58e

https://hg.mozilla.org/releases/mozilla-release/rev/52f7ed4a8d7b
https://hg.mozilla.org/releases/mozilla-release/rev/206ee8f5182d
https://hg.mozilla.org/releases/mozilla-release/rev/f6c6ce94b6a5
https://hg.mozilla.org/releases/mozilla-release/rev/012276caccb0
https://hg.mozilla.org/releases/mozilla-release/rev/2a4e40092a1d
https://hg.mozilla.org/releases/mozilla-release/rev/a30e3ea59c4f
https://hg.mozilla.org/releases/mozilla-release/rev/35dafbb1f1e8
https://hg.mozilla.org/releases/mozilla-release/rev/a83ee4ede70d
https://hg.mozilla.org/releases/mozilla-release/rev/7e6db7ca1cf9

J.C., can you backport this to ESR, please?

I have reviewed the changes pushed in comment 5, comment 8, and comment 17 and they look correct in sum.

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #22)

J.C., can you backport this to ESR, please?

Backport patches are mostly done, patterning off m-c (though the patches don't all apply cleanly). I've still got some failures that I haven't had time to work through: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8031b273b2a105dec1d00bf6e3c80e34c110e9e9

Probably going to have to come back to this tomorrow, I'm afraid.

(I abandoned the revisions that landed yesterday just to get them out of my "Needs Review" queue.)

Bug 1525191 part 0 - Regenerate pgo certs

Original commit: https://hg.mozilla.org/mozilla-central/rev/f9b86dec401e

Bug 1525191 part 1 - Regenerate all .pem.certspec files into their .pems

ESR backport of these three commits:

https://hg.mozilla.org/mozilla-central/rev/b6ec07118c70
https://hg.mozilla.org/mozilla-central/rev/66ff28da3e7d
https://hg.mozilla.org/mozilla-central/rev/ad8e75714968

... and additionally these tests, which exist in ESR60 but not in 67:

security/manager/ssl/tests/unit/test_ocsp_fetch_method/
security/manager/ssl/tests/unit/test_getchain/

Bug 1525191 part 2 - Regenerate zip files in security/manager/ssl/tests/unit/test_signed_apps

Original commit:
https://hg.mozilla.org/mozilla-central/rev/df9e185667a3

Removed the .zips that don't exist in ESR.

Bug 1525191 part 3 - Update EXPECTED_CHAIN in ssl_error_reports.sjs.

Original commit: https://hg.mozilla.org/mozilla-central/rev/25e6df21a4ee

Bug 1525191 part 4 - Fix test_x509.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/4c6fca8134ce

Bug 1525191 part 5 - Fix test_content_signing.js for updated certificates.

(Renumbered for ESR)

Original commit: https://hg.mozilla.org/mozilla-central/rev/aae349bf4115

Try run for comment 26:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5a647d95fb1e4faeeb5bbc371447094c9e74b436

https://hg.mozilla.org/releases/mozilla-esr60/rev/2a0b77c6fa9b