Closed Bug 509158 Opened 15 years ago Closed 13 years ago

Sign builds as part of the build process

Categories

(Release Engineering :: General, defect, P2)

x86
Windows Server 2003
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

(Whiteboard: [signing][oldbugs])

Attachments

(14 files, 9 obsolete files)

(deleted), patch
Details | Diff | Splinter Review
(deleted), patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
rail
: review+
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
bhearsum
: review+
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
catlee
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
bear
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
bear
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
catlee
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
bear
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
bear
: review+
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(deleted), patch
rail
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
One bottleneck in our current signing process is the amount of time it takes to download all of the builds to the signing machine. We currently download them all from stage before starting any signing. If we could sign the builds as they're finished, our signing process would add very little extra wall clock time to the release process. The streaming process could look like: * Watch for new builds to appear on stage * Download new builds * Make sure we wait until we have en-US before proceeding, it's the canonical version for dlls and exes. ** Do internal and installer signing of windows builds ** Do GPG signing for all platforms ** Verify results & push back to stage * Create MD5SUMS and SHA1SUMS & push back to stage
Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3
Whiteboard: [signing]
Assignee: nobody → jhford
Priority: P3 → P4
Another possible implementation, assuming we have fully automated signing (bug 558464), is to have any builders which produce something that gets signed call out to some process (through some build in buildbot process, a web service, or whatever) to trigger signing of their things.
Depends on: 607389
Looks like bug 558464 has an initial implementation which would resolve this bug. Marking this bug as a duplicate.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → DUPLICATE
I don't see how this is a dupe of 607389. That bug is likely to be one thing that happens as part of signing on demand, but this encompasses a lot more than just that.
Assignee: jhford → nobody
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
Now we have an old bug with no owner.
Whiteboard: [signing] → [signing][oldbugs][triagefollowup]
Reassigning to previous owner.
Assignee: nobody → jhford
Whiteboard: [signing][oldbugs][triagefollowup] → [signing][oldbugs]
(In reply to comment #0) > * Watch for new builds to appear on stage > * Download new builds > * Make sure we wait until we have en-US before proceeding, it's the canonical > version for dlls and exes. > ** Do internal and installer signing of windows builds > ** Do GPG signing for all platforms > ** Verify results & push back to stage > * Create MD5SUMS and SHA1SUMS & push back to stage (In reply to comment #5) > I don't see how this is a dupe of 607389. That bug is likely to be one thing > that happens as part of signing on demand, but this encompasses a lot more than > just that. What is not satisfied by the current signing scripts? From my experience on beta10 and beta11, it seemed like all of the requirements of comment 0 are done as part of the current signing process.
The current signing scripts download builds as they become available, but they do not sign them as they become available.
Whiteboard: [signing][oldbugs] → [signing][oldbugs][triagefollowup]
The current plan we've been discussing is to create a "signing server" that runs on keymaster. This signing server will listen for properly formatted and secured HTTP requests of files to be signed. Once we have this, we can modify the build process so that signing is done as part of the build by having the build machine submit the files to be packaged to the signing server and pulling down the signed bits once they're ready. Updates and snippets can also be generated immediately for builds and repacks.
Assignee: jhford → catlee
Priority: P4 → P2
Depends on: 638896
Blocks: 400296
Whiteboard: [signing][oldbugs][triagefollowup] → [signing][oldbugs]
Summary: Sign builds as they become available → Sign builds as part of the build process
Blocks: 615311
Priority: P2 → P4
Blocks: 663055
cc-ing kev. He's very interested in this for BYOB where they have many GBs of builds that need to make it through signing in one pass currently.
Attached patch Hooks for signing script in build process (obsolete) (deleted) — Splinter Review
The release automation would set a couple of variables so that signing happens during 'make package': MOZ_SIGN_CMD - command to run to do the actual signing MOZ_EXTERNAL_SIGNING_FORMAT - for linux,osx this will be 'gpg' so we can generate the detached gpg signatures. for win32 this will be 'signcode,gpg' so we sign the installers and then the detached gpg signature MOZ_INTERNAL_SIGNING_FORMAT - for linux,osx this will be unset. for win32 this will be 'signcode' so that all the .exe's and .dll's are signed.
Attachment #564855 - Flags: feedback?(ted.mielczarek)
I believe that we can make use of the mono tools to do signing under linux. This requires a little tweaking to the certificates to work: openssl x509 -in MozAuthenticode.spc -inform DER -text > MozAuthenticode.crt makecert MozAuthenticode.crt MozAuthenticode.spc Then signcode works as expected, and accepts a password for the private key on stdin.
Blocks: 481815
Blocks: 607389
No longer depends on: 607389
Priority: P4 → P2
Depends on: 694300
Attached patch Hooks for signing script in build process (obsolete) (deleted) — Splinter Review
Attachment #564855 - Attachment is obsolete: true
Attachment #566851 - Flags: feedback?(ted.mielczarek)
Attachment #564855 - Flags: feedback?(ted.mielczarek)
Comment on attachment 566851 [details] [diff] [review] Hooks for signing script in build process Review of attachment 566851 [details] [diff] [review]: ----------------------------------------------------------------- ::: toolkit/mozapps/installer/signing.mk @@ +39,5 @@ > +_PKG = $(_ABS_DIST)/$(PACKAGE) > +endif > + > +ifeq (WINNT,$(OS_ARCH)) > +MOZ_SIGN_PREPARED_PACKAGE_CMD=$(MOZ_SIGN_CMD) -f $(MOZ_INTERNAL_SIGNING_FORMAT) -i '*.dll' -i '*.exe' -x 'D3DCompiler*.dll' -x 'd3dx9*.dll' $(_PKGDIR) I'd like to see these include/exclude lists factored out into separate variables. Something like: SIGN_INCLUDES := \ '*.dll' \ '*.exe' \ $(NULL) then you can use it like: $foreach(i, $(SIGN_INCLUDES),-i $(i)) It looks like if you did that, you might be able to make MOZ_SIGN_PREPARED_PACKAGE_CMD the same on every platform, and only have the includes/excludes ifdefed per-platform.
Attachment #566851 - Flags: feedback?(ted.mielczarek) → feedback+
Attached patch set signing formats, servers in configs (obsolete) (deleted) — Splinter Review
We'll probably need to move signing_servers to {production,preproduction,staging}_config.py, but this is basically what we need.
Attachment #572491 - Flags: feedback?(catlee)
Attached patch factory/misc.py changes (obsolete) (deleted) — Splinter Review
Attachment #572492 - Flags: review?(catlee)
Depends on: 700347
I tested catlee's latest signing-server tools branch with my buildbotcustom and buildbot-configs patches. Here's the results: * Linux: no problems * Mac: main "make package" worked fine, "make package" with prettynames failed because of missing quotes somewhere * Windows: non-pretty "make package" didn't work at all until I installed the 'ssl' module (see bug 700347). I also had to make some changes to signing.py for Python 2.5 specific things: $ hg diff diff --git a/release/signing/signing.py b/release/signing/signing.py --- a/release/signing/signing.py +++ b/release/signing/signing.py @@ -154,17 +154,17 @@ def copyfile(src, dst, copymode=True): shutil.copyfile(src, dst) if copymode: shutil.copymode(src, dst) shutil.copystat(src, dst) def sha1sum(f): """Return the SHA-1 hash of the contents of file `f`, in hex format""" h = hashlib.sha1() - fp = open(f) + fp = open(f, 'rb') while True: block = fp.read(512*1024) if not block: break h.update(block) return h.hexdigest() def sums_are_equal(base_package, packages): @@ -256,34 +256,34 @@ def packexe(exefile, srcdir): os.unlink(appbundle) def bunzip2(filename): """Uncompress `filename` in place""" log.debug("Uncompressing %s", filename) tmpfile = "%s.tmp" % filename os.rename(filename, tmpfile) b = bz2.BZ2File(tmpfile) - f = open(filename, "w") + f = open(filename, "wb") while True: block = b.read(512*1024) if not block: break f.write(block) f.close() b.close() shutil.copystat(tmpfile, filename) shutil.copymode(tmpfile, filename) os.unlink(tmpfile) def bzip2(filename): """Compress `filename` in place""" log.debug("Compressing %s", filename) tmpfile = "%s.tmp" % filename os.rename(filename, tmpfile) - b = bz2.BZ2File(filename, "w") + b = bz2.BZ2File(filename, "wb") f = open(tmpfile) while True: block = f.read(512*1024) if not block: break b.write(block) f.close() b.close() @@ -553,21 +553,18 @@ def buildValidatingOpener(ca_certs): except ImportError: from httplib import HTTPSConnection from urllib2 import HTTPSHandler class VerifiedHTTPSConnection(HTTPSConnection): def connect(self): # overrides the version in httplib so that we do # certificate verification - sock = socket.create_connection((self.host, self.port), - self.timeout) - if self._tunnel_host: - self.sock = sock - self._tunnel() + sock = socket.socket() + sock.connect((self.host, self.port)) # wrap the socket using verification with the root # certs in trusted_root_certs self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file, cert_reqs=ssl.CERT_REQUIRED, ca_certs=ca_certs, After that, I got a new traceback: WARNING: Found 311 duplicate files taking 968406 bytes Stripped 317565 bytes Deoptimized 0/2047 in ./omni.jar 2011-11-07 13:17:09,092 - ea2613b229ba571130d66d9375331a62fd03c2d7: processing e :/builds/moz2_slave/m-cen-w32/build/obj-firefox/browser/installer/../../dist/fir efox\AccessibleMarshal.dll on https://dev-master01.build.scl1.mozilla.com:8080 Traceback (most recent call last): File "e:/builds/moz2_slave/m-cen-w32/tools/release/signing/signtool.py", line 127, in <module> main() File "e:/builds/moz2_slave/m-cen-w32/tools/release/signing/signtool.py", line 111, in main errors.extend(remote_signdir(options, url, f, fmt)) File "e:\builds\moz2_slave\m-cen-w32\tools\release\signing\signing.py", line 5 32, in remote_signdir if not remote_signfile(options, url, fullpath, fmt, dest=dest): File "e:\builds\moz2_slave\m-cen-w32\tools\release\signing\signing.py", line 4 80, in remote_signfile os.rename(tmpfile, dest) WindowsError: [Error 17] Cannot create a file when that file already exists make[2]: *** [make-package] Error 1 make[2]: Leaving directory `/e/builds/moz2_slave/m-cen-w32/build/obj-firefox/bro wser/installer' make[1]: *** [all] Error 2 make[1]: Leaving directory `/e/builds/moz2_slave/m-cen-w32/build/obj-firefox/bro wser/installer' make: *** [package] Error 2
No longer depends on: 700347
Depends on: 700347
Attached patch Sign MARs (obsolete) (deleted) — Splinter Review
Not tested yet. Should work for external (GPG) singing of MAR files.
Attached patch support l10n signing, some bugfixes (obsolete) (deleted) — Splinter Review
Changes in this patch: * Inherit en-US env for l10n builds * Port MercurialBuildFactory signing server work to BaseRepackFactory * Set win32_basedir, because we need it for PYTHONPATH to work on Windows
Attachment #572491 - Attachment is obsolete: true
Attachment #572492 - Attachment is obsolete: true
Attachment #572491 - Flags: feedback?(catlee)
Attachment #572492 - Flags: review?(catlee)
Attached patch change signing format variable format (obsolete) (deleted) — Splinter Review
This patch changes the separator in the signing format variables to a space, to make it easy to filter out things in the build system.
Blocks: 701087
Attached patch Sign MARs (deleted) — Splinter Review
Using MOZ_SIGN_PACKAGE_CMD
Attachment #572813 - Attachment is obsolete: true
Comment on attachment 573199 [details] [diff] [review] change signing format variable format I have an updated buildbot-configs patch on my github that supports different signing servers for dep/nightly builds, and different signing servers for production/preproduction/config: https://github.com/bhearsum/buildbot-configs/compare/master...signing-server
Attachment #573199 - Attachment is obsolete: true
Comment on attachment 573196 [details] [diff] [review] support l10n signing, some bugfixes And an updated buildbotcustom patch here: https://github.com/bhearsum/buildbotcustom/compare/master...signing-server
Attachment #573196 - Attachment is obsolete: true
Attached patch Hooks for signing script in build process (obsolete) (deleted) — Splinter Review
planning on landing this on elm first
Attachment #566851 - Attachment is obsolete: true
Attachment #577613 - Flags: review?(ted.mielczarek)
Depends on: 706660
Depends on: 706832
Attachment #578633 - Flags: review?(bhearsum)
Attached patch buildbot configs to enable signing servers (obsolete) (deleted) — Splinter Review
Attachment #578664 - Flags: review?(rail)
Attachment #577613 - Attachment is obsolete: true
Attachment #577613 - Flags: review?(ted.mielczarek)
Attachment #578664 - Attachment is obsolete: true
Attachment #578670 - Flags: review?(rail)
Attachment #578670 - Flags: review?(catlee)
Attachment #578664 - Flags: review?(rail)
Attachment #578670 - Flags: review?(catlee) → review+
Attached patch signing tools! (deleted) — Splinter Review
Attachment #578675 - Flags: review?(rail)
Attachment #578675 - Flags: review?(bhearsum)
This applies on top of catlee's buildbotcustom patch.
Attachment #578676 - Flags: review?(catlee)
Attachment #578676 - Flags: review?(catlee) → review+
Attachment #578675 - Flags: review?(bhearsum) → review+
Comment on attachment 578670 [details] [diff] [review] enable signing server for debug builds; disable pre-signed updater on elm Almost there!
Attachment #578670 - Flags: review?(rail) → review+
Comment on attachment 578675 [details] [diff] [review] signing tools! YAY!
Attachment #578675 - Flags: review?(rail) → review+
awesome!
Attachment #578633 - Flags: review?(bhearsum) → review+
Attachment #578633 - Flags: checked-in+
Attachment #578675 - Flags: checked-in+
(In reply to Chris AtLee [:catlee] from comment #14) > I believe that we can make use of the mono tools to do signing under linux. > This requires a little tweaking to the certificates to work: > > openssl x509 -in MozAuthenticode.spc -inform DER -text > MozAuthenticode.crt > makecert MozAuthenticode.crt MozAuthenticode.spc sorry, this should be 'cert2spc MozAuthenticode.crt MozAuthenticode.spc'
Attachment #578676 - Flags: checked-in+
Attachment #578670 - Flags: checked-in+
Attached patch turn on only on elm for now (deleted) — Splinter Review
let's limit possible damage to elm/oak for now
Attachment #578810 - Flags: review?
Attachment #578811 - Flags: review?
Attachment #578810 - Flags: review? → review+
Attachment #578811 - Flags: review? → review+
Attachment #578918 - Flags: review?(rail)
Attachment #578810 - Flags: checked-in+
Attachment #578811 - Flags: checked-in+
Attachment #578918 - Flags: review?(rail) → review+
Attachment #578918 - Flags: checked-in+
Attached patch turn on signing on m-c (deleted) — Splinter Review
Attachment #579232 - Flags: review?(rail)
Comment on attachment 579232 [details] [diff] [review] turn on signing on m-c r+ from rail over irc
Attachment #579232 - Flags: review?(rail)
Attachment #579232 - Flags: review+
Attachment #579232 - Flags: checked-in+
currently sha1sum calculations are done in the same process and greenlet as the web server. this effectively blocks the server until the sha1sum is complete, which can take a while for larger files. this patch makes sha1sum() use a pool of workers to do the calculation in separate processes, also allowing the web server to continue serving requests.
Attachment #579350 - Flags: review?(bear)
Comment on attachment 579350 [details] [diff] [review] do sha1sum calculations out of process this is a lot cleaner than I've had to do in the past with my own code - looks good
Attachment #579350 - Flags: review?(bear) → review+
Attached patch enable signing on inbound (deleted) — Splinter Review
Attachment #579781 - Flags: review?(rail)
Attachment #579782 - Flags: review?(rail)
Attached patch signing server enhancements (deleted) — Splinter Review
this patch adds a few nice-to-have things: * -d option to daemonize the process (using python-daemon library, in vendor directory) * --stop/--restart/--reload commands. for restart we ask for the passphrases before stopping the old server so that downtime is minimal * backdoor support
Attachment #579828 - Flags: review?(bhearsum)
Attachment #579828 - Flags: review?(bear)
Attachment #579781 - Flags: review?(rail) → review+
Attachment #579782 - Flags: review?(rail) → review+
Comment on attachment 579828 [details] [diff] [review] signing server enhancements 1. I think you need to get the log object from multiprocessing.get_logger() if your calling log from inside a worker 2. wouldn't it be better if for the reload that you check for a valid pid and only proceed if one is found
Attachment #579828 - Flags: review?(bear) → review+
Attachment #579828 - Flags: review?(bhearsum) → review+
Attachment #579781 - Flags: checked-in+
Attachment #579782 - Flags: checked-in+
Blocks: 708656
Attachment #579350 - Flags: checked-in+
Attachment #579828 - Flags: checked-in+
I reconfig-ed with the landed changes from this bug today.
Attached patch sign ALL the branches! (deleted) — Splinter Review
This enables dep signing for all win32 builds. m-c, m-a, m-i, elm and oak nightlies get to use nightly signing keys.
Attachment #582526 - Flags: review?
Attachment #582526 - Flags: review? → review+
Attachment #582526 - Flags: checked-in+
Calling this done.
Status: REOPENED → RESOLVED
Closed: 14 years ago13 years ago
Resolution: --- → FIXED
Depends on: 716431
(In reply to Chris AtLee [:catlee] from comment #36) > (In reply to Chris AtLee [:catlee] from comment #14) > > I believe that we can make use of the mono tools to do signing under linux. > > This requires a little tweaking to the certificates to work: > > > > openssl x509 -in MozAuthenticode.spc -inform DER -text > MozAuthenticode.crt > > makecert MozAuthenticode.crt MozAuthenticode.spc > > sorry, this should be 'cert2spc MozAuthenticode.crt MozAuthenticode.spc' and make the first one: openssl x509 -in MozAuthenticode.spc -inform DER -outform PEM > MozAuthenticode.crt
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: