Since bug 1473259 has landed, we have win64-asan-reporter builds and updates. However, the NS job keeps throwing an exception. I looked into it and saw this: 2018-07-20 23:57:36,038 - signingscript.utils - INFO - 2018-07-20 23:57:36,038 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: uploading for signing 2018-07-20 23:57:37,417 - signingscript.utils - INFO - 2018-07-20 23:57:37,417 - Starting new HTTPS connection (1): signing4.srv.releng.scl3.mozilla.com:9100 2018-07-20 23:58:35,293 - signingscript.utils - INFO - 2018-07-20 23:58:35,292 - https://signing4.srv.releng.scl3.mozilla.com:9100 "POST /sign/sha2signcode HTTP/1.1" 400 0 2018-07-20 23:58:35,294 - signingscript.utils - INFO - 2018-07-20 23:58:35,294 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: error uploading file for signing: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode 2018-07-20 23:58:35,294 - signingscript.utils - INFO - Traceback (most recent call last): 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 86, in remote_signfile 2018-07-20 23:58:35,295 - signingscript.utils - INFO - r.raise_for_status() 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status 2018-07-20 23:58:35,295 - signingscript.utils - INFO - raise HTTPError(http_error_msg, response=self) 2018-07-20 23:58:35,295 - signingscript.utils - INFO - requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode/4a5eebc5f40da52d86218ee619c829a6fe4bbe58 2018-07-20 23:58:35,295 - signingscript.utils - INFO - 2018-07-20 23:58:35,295 - signingscript.utils - INFO - During handling of the above exception, another exception occurred: 2018-07-20 23:58:35,295 - signingscript.utils - INFO - 2018-07-20 23:58:35,295 - signingscript.utils - INFO - Traceback (most recent call last): 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client.py", line 137, in remote_signfile 2018-07-20 23:58:35,295 - signingscript.utils - INFO - r.raise_for_status() 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line 939, in raise_for_status 2018-07-20 23:58:35,295 - signingscript.utils - INFO - raise HTTPError(http_error_msg, response=self) 2018-07-20 23:58:35,295 - signingscript.utils - INFO - requests.exceptions.HTTPError: 400 Client Error: File too large for url: https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode 2018-07-20 23:58:36,296 - signingscript.utils - INFO - 2018-07-20 23:58:36,295 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: giving up after 5 tries 2018-07-20 23:58:36,313 - signingscript.utils - INFO - 2018-07-20 23:58:36,313 - Failed to sign /builds/scriptworker/work/zip6fgql6os/firefox/xul.dll with sha2signcode 2018-07-20 23:58:36,337 - signingscript.utils - INFO - exitcode 1 (taken from example log here: https://tools.taskcluster.net/groups/IwbRaz3XRumH5OWT-l41mw/tasks/RlpTq2VjRpiAKHtMirTBuw/runs/1/logs/public%2Flogs%2Flive_backing.log) Looks to me as if the size for signing is simply exceeded (and I guess on try, no actual signing is done). ASan builds are larger and my guess is we added an exception for linux64 but not for win64. :catlee, :bhearsum, can any of you help with this?

(In reply to Christian Holler (:decoder) from comment #0) > Since bug 1473259 has landed, we have win64-asan-reporter builds and > updates. However, the NS job keeps throwing an exception. I looked into it > and saw this: > > > 2018-07-20 23:57:36,038 - signingscript.utils - INFO - 2018-07-20 > 23:57:36,038 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: uploading for > signing > 2018-07-20 23:57:37,417 - signingscript.utils - INFO - 2018-07-20 > 23:57:37,417 - Starting new HTTPS connection (1): > signing4.srv.releng.scl3.mozilla.com:9100 > 2018-07-20 23:58:35,293 - signingscript.utils - INFO - 2018-07-20 > 23:58:35,292 - https://signing4.srv.releng.scl3.mozilla.com:9100 "POST > /sign/sha2signcode HTTP/1.1" 400 0 > 2018-07-20 23:58:35,294 - signingscript.utils - INFO - 2018-07-20 > 23:58:35,294 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: error uploading > file for signing: 400 Client Error: File too large for url: > https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode > 2018-07-20 23:58:35,294 - signingscript.utils - INFO - Traceback (most > recent call last): > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File > "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client. > py", line 86, in remote_signfile > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > r.raise_for_status() > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File > "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line > 939, in raise_for_status > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - raise > HTTPError(http_error_msg, response=self) > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > requests.exceptions.HTTPError: 404 Client Error: Not Found for url: > https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode/ > 4a5eebc5f40da52d86218ee619c829a6fe4bbe58 > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - During handling of > the above exception, another exception occurred: > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - Traceback (most > recent call last): > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File > "/builds/scriptworker/lib/python3.6/site-packages/signtool/signing/client. > py", line 137, in remote_signfile > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > r.raise_for_status() > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - File > "/builds/scriptworker/lib/python3.6/site-packages/requests/models.py", line > 939, in raise_for_status > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - raise > HTTPError(http_error_msg, response=self) > 2018-07-20 23:58:35,295 - signingscript.utils - INFO - > requests.exceptions.HTTPError: 400 Client Error: File too large for url: > https://signing4.srv.releng.scl3.mozilla.com:9100/sign/sha2signcode > 2018-07-20 23:58:36,296 - signingscript.utils - INFO - 2018-07-20 > 23:58:36,295 - 4a5eebc5f40da52d86218ee619c829a6fe4bbe58: giving up after 5 > tries > 2018-07-20 23:58:36,313 - signingscript.utils - INFO - 2018-07-20 > 23:58:36,313 - Failed to sign > /builds/scriptworker/work/zip6fgql6os/firefox/xul.dll with sha2signcode > 2018-07-20 23:58:36,337 - signingscript.utils - INFO - exitcode 1 > > (taken from example log here: > https://tools.taskcluster.net/groups/IwbRaz3XRumH5OWT-l41mw/tasks/ > RlpTq2VjRpiAKHtMirTBuw/runs/1/logs/public%2Flogs%2Flive_backing.log) > > > Looks to me as if the size for signing is simply exceeded (and I guess on > try, no actual signing is done). ASan builds are larger and my guess is we > added an exception for linux64 but not for win64. > > > :catlee, :bhearsum, can any of you help with this? It looks like it's failing on xul.dll, which means it alone is already larger than 150MB. When we increased the limit for Linux, that was for the MARs, and we raised it to 200MB (https://bugzilla.mozilla.org/show_bug.cgi?id=1438967). How big should we expect xul.dll to be? And the entire installer? (They both share the same size limit for...reasons.) IIRC, we try to keep the size limits low enough to avoid certain attacks. If the Windows ASan builds are that much larger we may want to consider a separate signing format to avoid raising the limit for other Windows builds. Do you have any thoughts, catlee?

target.installer.exe is currently 162 MB large while xul.dll has 325 MB. The task I took the info from is here: https://tools.taskcluster.net/groups/Dng1X5-kQfqbB_rsdjNc6w/tasks/H7dTd7ooQEWUU7byQxDt4g/runs/0/artifacts

We ended up bumping the max size, should be fixed for the next set of nightlies.

(In reply to Ben Hearsum (:bhearsum) from comment #4) > We ended up bumping the max size, should be fixed for the next set of > nightlies. And that next set of nightlies just made it to the fx-sig-verify check ;) (Installer only at this time.) What value did you end up using?

Looks like 350mb. https://github.com/mozilla-releng/build-puppet/pull/128/files

This was fixed and Nightlies are now produced correctly, thanks!