We hit an issue with some partner builds during the 3.5 release that caused partners (Yahoo, specifically) to be unable to sign the builds. The problem arises when we repack the partner builds after doing installer signing. This somehow affects the package in such a way that signcode.exe refuses to sign it. This is a serious issue for partners who sign their own builds. There's two ways to fix it: 1) Do full repacks of partner builds rather than just adding files to them 2) Do partner builds with only internally signed builds, not internal+installer. #1 is what we're doing to fix up the partner builds for 3.5. For the long term fix, we can probably augment our process as follows: * Do partner repacks at the same time as l10n * Sign them at the same time, too, making use of the new signing scripts to make the time increase trivial. This issue should block 1.9.1.1/3.5.1. Does that all sound right to you, Catlee?

catlee's going to attach his script here when he's done the full repackaging work for 3.5 and I'll integrate that code into the partner-repack script. That may not be how we fix this permanently, but at least kev can make use of it in the interim if necessary.

FWIW, I don't consider this blocking since we have a viable workaround with catlee's repack script.

it'd be good to get it in for 3.5.2, though, as there's effort required at a couple points. catlee's script has to be run and the repacks have to be re-signed. this means either applying the script after the repacks have been generated or waiting for generation then downloading the repacks, and then running the repack-repack script. that kind of marginalizes having the repacks generated as part of the build process, and if we could get the repacks signed as part of the process, it will save some effort (and I realize the effort saved needs to take into account what it'd take to implement the fix and do the value prop).

Pulling from the unsigned/ dir for win32 seems to work. I just had a successful repacking run on dm-partnerdist01 with it. We are running into a new problem though. As the number of partner repacks increases, the overhead for unpacking and repacking is becoming a real issue. For example, prior to the current round of yahoo repack additions, repacking all partners took about 6-8 minutes. The current test run just took 30 minutes. We can improve this somewhat by creating a graph of all planned partner/locale/platform combinations, and then unpacking each required source build only once. This is only really an issue for linux (bz2) and mac (dmg). Not something I plan to fix tonight, but soon perhaps.

kev: despite our email exchange today, I still think using the unsigned builds is the way to go. I would rather keep the signing logic in the signing scripts and not do too many back-flips to unpack/repack already-signed builds. However, due to the increasing number of partner builds, I don't think we can batch sign them together with the regular builds. I would rather break it up into two signing pieces. Here's the release step progression I envision: en-US builds -> l10n repacks -> partner repacks -> signing for en-US/l10n -> signing for partners This way RelEng can start up the rest of the release process once the core builds (en-US + l10n) are signed, and then kick off the signing for the partner builds. Signing for the core builds could even start while the partner repacks are in progress. catlee has a new signing script that speeds up the signing process significantly. From talking to him yesterday, I *think* we should be able to use that script for partner signing as well, perhaps even carrying forward the same cache that was used for the core build signing for added speed.

Going to do a test run this week of signing the partner builds using the new combined process. Hoping to get to it tomorrow, buildduty-willing.

I ran the partner builds through catlee's new combined signing script on Friday afternoon, but unfortunately it didn't work straight out of the box. Only about half of the Windows builds ended up signed. I'll be digging into the logs to figure out why and trying again this week.

Huh, not sure how I screwed it up last week, but today's run was flawless. The script signed 151 builds (internals and installers) in 23.5 minutes. For the record, here's the process I used today: * followed the setup steps for CombinedSigning, creating a new directory and copying over the Makefile * rsync the partner builds hierarchy directly under unsigned-build1/ * ran individual make targets: ** make setup ** (skipped download, see above) ** make stubs ** make sign-files I didn't run "make create-sigs" because I don't think we do that now, but I think it would be trivial to do so now. Tomcat: do you want me to upload a selection of these signed builds so you can verify the signing?

Here's the patch I used to generate the unsigned win32 partner repacks in my previous testing.

Comment on attachment 395488 [details] [diff] [review] Pull unsigned win32 builds for repacking >diff --git a/scripts/partner-repacks.py b/scripts/partner-repacks.py >+ if isWin(platform): Looks fine, just some stray tabs to zap in this line. r+ with fix on checkin.

Comment on attachment 395488 [details] [diff] [review] Pull unsigned win32 builds for repacking http://hg.mozilla.org/build/partner-repacks/rev/ef3a01b8b95b