Mostly this is in updates, do support the changes described in bug 552924 comment 26. This affects Firefox 4 and onwards, so mozilla-central and derived repos plus mozilla-2.0 when we start using it. We'll need to support nightly updates as soon as the change lands from the parent bug. If it is included in the 4.0b3 release then we'll need to support release updates at the time of 4.0b4.

Initial support for nightlies provided by adding symlinks: cd /opt/aus2/incoming/2/Firefox for d in mozilla-central mozilla-2.0 tracemonkey electrolysis; do ln -s Darwin_Universal-gcc3 $d/Darwin_ppc-gcc3-u-ppc-i386 ln -s Darwin_Universal-gcc3 $d/Darwin_i386-gcc3-u-ppc-i386 done This will redirect users of the existing ppc+i386 universal to the right snippets once bug 552924 lands. We'll need to change these snippets once we turn the x64_64 build into i386+x86_64 universal and want to switch users over to it. We can change the name of the directories that snippets to better names once everything shakes out. For releases we need to do more work for a long term fix. Assuming 552924 is in b5 then we at least need symlinks for b5 -> whatever's next.

Darwin_i386-gcc3-u-ppc-i386 replaced with Darwin_x86-gcc3-u-ppc-i386, per bug 591817.

For mozilla-central, Darwin_x86-gcc3-u-ppc-i386 now points to the Darwin_x86_64-gcc3 bucket where the snippets from the new universal are going. As do Darwin_x86-gcc3-u-i386-x86_64 Darwin_x86_64-gcc3-u-i386-x86_64 We should do the same for TraceMonkey and E10s when they do a universal nightly in the new style.

With all the fun from bug 600289 and friends we now have nightlies publishing into Darwin_x86_64-gcc3 and the following symlinks Darwin_x86-gcc3-u-ppc-i386 -> Darwin_x86_64-gcc3 Darwin_x86-gcc3-u-i386-x86_64 -> Darwin_x86_64-gcc3 Darwin_x86_64-gcc3-u-i386-x86_64 -> Darwin_x86_64-gcc3 PPC users on the old universal (Darwin_ppc-gcc3-u-ppc-i386) are no longer offered updates. I've set up the same mappings for tracemonkey, electrolysis and mozilla-2.0.

I've set up these mappings for mozilla-1.9.1 and mozilla-1.9.2 Darwin_ppc-gcc3-u-ppc-i386 -> Darwin_Universal-gcc3 Darwin_x86-gcc3-u-ppc-i386 -> Darwin_Universal-gcc3 to support the change on those branches in bug 552924 (attachment 478623 [details] [diff] [review]).

This is more critical now that we're shipping a release (3.6.13) with these updater changes. Currently, we have to manually modify the snippets to have Mac updates work for 3.6.13 users. It'll hurt even more in 3.6.14.

I'll be working on this very soon.

I haven't been able to verify this against existing snippets yet, but it _seems_ ok at a first glance. Needs a ton more testing, of course.

Did some verification by regenerating 3.6.12 -> 3.6.13 snippets. Everything in the test channels was a match, except for the sums and filesizes of partial MARs, which were regenerated as part of my test run. Done with a limited set of locales. I'm going to do a full test run on 3.5, 3.6, and 4.0 next.

After a bunch more testing I believe that this is complete and ready for review. Summary of changes: - Map "mac64" platform to "osx" in Bouncer - Add platform -> filesystem mapping, to support what patcher thinks of as "mac64" living in the "mac" directory - Adjust AUS2_PLATFORMS data structure to be a hash whose values are lists; add appropriate mac/mac64 Darwin ABIs. The rest is just adjusting patcher to cope with the changes. It looks bigger than it is; all of the large hunks are just adding a loop and re-indenting. With this patch applied, I re-run a full set of updates for 4.0b8. I diffed those against the ones we're shipping b8 with. The results were acceptable, with the following differences: - My test run had real partial snippets for Windows. In the real Beta 8, we overwrote them with completes. - My test run had no snippets for ja/ja-JP-mac. This will be fixed in bug 619522. - My test run did not generate old universal -> new universal snippets. This is Really Hard to do correctly in patcher, and I got the OK from drivers to stop doing it. (The "Only in /home/cltbld/...." lines in the diff are about this.) - My test run generated snippets for "Darwin_x86_64-gcc3-u-i386-x86_64" and "Darwin_x86-gcc3-u-i386-x86_64" for 3.7a5 through 4.0b7 and the original run did not, which had to be fixed by hand later. This is the thing we're fixing in this bug! These ABIs are only used starting on beta 5, but it doesn't hurt to have them for older releases. This patch only applies to the UPDATE_PACKAGING_R12 tag. I have a similar patch for the _R11 branch of development.

This is very very similar to the _R12 version. The only difference is that the 'mac64' bouncer platform isn't adjusted, because it's irrelevant on this branch, and the details of one of the large hunks is different, because of bug 514040. Like the other patch, I tested this by running updates. In this case, I did: 3.5.x -> 3.5.16, 3.6.x -> 3.6.13, and 3.5.16 -> 3.6.13 MU. The only differences were: - My test run generated snippets for the "Darwin_x86-gcc3-u-ppc-i386" ABI for all older 3.5 and 3.6 releases, the real one did not. Again, this what we're fixing here. Versions older than 3.5.16/3.6.13 don't need them, but it doesn't hurt to have them (and makes this patch a lot simpler...). - My test run did not have any snippets for 3.6.13build1/2 -> build3 nor 3.5.16build1 -> build2. This is OK because these are generated after patcher runs, by a different script. To land this, I'll need to create a branch whose parent is UPDATE_PACKAGING_R11, for all of the patcher/ directory. After landing I plan to retag with UPDATE_PACKAGING_R11_1. I'll be tagging _R12 similarly.

The only things affected by this are update verify, and generate-candidate-build-updates.py. I didn't do any testing on the former, but the patch to is is extremely simple and safe. For the latter, I ran it against the 3.5.16/3.6.13 snippets I generated yesterday, and re-ran the diffs. The results were the same as before, except the "missing 3.6.13/3.5.16" lines went away, which means I did it right!

Besides the tools addressed in the posted patches, our nightly update logic, contained mostly in process/factory.py also assumes a 1:1 mapping of platform:update ABI. I don't think it's worth fixing it for a few reasons: - Adding new ABIs for nightly updates is a one-off change on AUS, we don't need to fix it every time new updates are published - The patch needed is fairly difficult, and very risky to all nightly updates - This version of AUS is intended to be going away in the next 3 months.

Comment on attachment 499113 [details] [diff] [review] support multiple ABIs in patcher, UPDATE_PACKAGING_R12 version > @@ -354,8 +370,11 @@ sub SubstitutePath > my $platform = $args{'platform'} || 'UNDEFINED'; > ........... > my %bouncer_platforms = GetBouncerPlatformStrings(); > my $bouncer_platform = $bouncer_platforms{$platform}; > + > + my %filepath_platforms = GetFilepathPlatformStrings(); > + my $filepath_platform = $filepath_platforms{$platform}; > > - $string =~ s/%platform%/$platform/g; > + $string =~ s/%platform%/$filepath_platform/g; A nit. These changes should issue warnings (with -w) for SubstitutePath calls for $licenseUrl = SubstitutePath(...) because you don't pass platform. r+ if you pass platform. License is not supposed to be different per platform and contain %platform% placeholder, but just to be consistent with other calls.

Comment on attachment 499120 [details] [diff] [review] support multiple ABIs in patcher, UPDATE_PACKAGING_R11 version See my comments for the previous patch.

Comment on attachment 499295 [details] [diff] [review] fix 1:1 platform:ABI assumptions in tools repo > - update_platform = buildbot2updatePlatform(platform) > + update_platforms = buildbot2updatePlatform(platform) A nit. Wouldn't it be better to rename buildbot2updatePlatform to buildbot2updatePlatforms because it returns a list of platforms? Great progress!!!

Comment on attachment 499113 [details] [diff] [review] support multiple ABIs in patcher, UPDATE_PACKAGING_R12 version Checking in MozAUSLib.pm; /cvsroot/mozilla/tools/patcher/MozAUSLib.pm,v <-- MozAUSLib.pm new revision: 1.14; previous revision: 1.13 done Checking in patcher2.pl; /cvsroot/mozilla/tools/patcher/patcher2.pl,v <-- patcher2.pl new revision: 1.41; previous revision: 1.40 done cvs co -r UPDATE_PACKAGING_R12 mozilla/client.mk cd mozilla make -f client.mk MOZ_CO_PROJECT=all checkout cvs up -A tools/update-packaging cvs up -AdP tools/release cvs up -AdP tools/patcher cvs -q tag UPDATE_PACKAGING_R13 2>&1 | tee ../tag.log grep -v '^T' ../tag.log cvs tag -b UPDATE_PACKAGING_R13_minibranch client.mk cvs up -r UPDATE_PACKAGING_R13_minibranch client.mk edited client.mk, bumped to UPDATE_PACKAGING_R13 commited client.mk /cvsroot/mozilla/client.mk,v <-- client.mk new revision: 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1; previous revision: 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1 cvs -q tag UPDATE_PACKAGING_R13 client.mk W client.mk : UPDATE_PACKAGING_R13 already exists on version 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1 : NOT MOVING tag to version 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1 foo-ix-blah:mozilla bhearsum$ cvs -q tag -F UPDATE_PACKAGING_R13 client.mk T client.mk Also created an UPDATE_PACKAGING_R13 tag on m-c as follows: hg tag -r UPDATE_PACKAGING_R12 -m "Create UPDATE_PACKAGING_R13 tag for bug 583671. r=rail. a=npotb DONTBUILD" UPDATE_PACKAGING_R13

Comment on attachment 499120 [details] [diff] [review] support multiple ABIs in patcher, UPDATE_PACKAGING_R11 version Before applying and commiting this patch, I had to create the UPDATE_PACKAGING_R11_minibranch for patcher, as follows: cvs co -d patcher -r UPDATE_PACKAGING_R11 mozilla/tools/patcher cd patcher cvs tag -b UPDATE_PACKAGING_R11_minibranch . I then applied and commited the patch as usual: Checking in MozAUSLib.pm; /cvsroot/mozilla/tools/patcher/MozAUSLib.pm,v <-- MozAUSLib.pm new revision: 1.13.2.1; previous revision: 1.13 done Checking in patcher2.pl; /cvsroot/mozilla/tools/patcher/patcher2.pl,v <-- patcher2.pl new revision: 1.38.2.1; previous revision: 1.38 done And then did the following to create a new tag: cvs co -r UPDATE_PACKAGING_R11 mozilla/client.mk cd mozilla make -f client.mk MOZ_CO_PROJECT=all checkout cvs up -AdP -r UPDATE_PACKAGING_R11_minibranch tools/patcher cvs -q tag UPDATE_PACKAGING_R11_1 2>&1 | tee ../tag.log grep -v '^T' ../tag.log cvs up -r UPDATE_PACKAGING_R11_minibranch client.mk edited client.mk, bumped to UPDATE_PACKAGING_R11_1 commited client.mk Checking in client.mk; /cvsroot/mozilla/client.mk,v <-- client.mk new revision: 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.2; previous revision: 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1 cvs -q tag UPDATE_PACKAGING_R11_1 client.mk W client.mk : UPDATE_PACKAGING_R11_1 already exists on version 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1 : NOT MOVING tag to version 1.314.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.1.2.2 cvs -q tag -F UPDATE_PACKAGING_R11_1 client.mk T client.mk As mentioned, I updated m-c with the new tag, like so: hg tag -r UPDATE_PACKAGING_R11 UPDATE_PACKAGING_R12

Oops, forgot to address the comment about SubstitutePath before I landed, fixed in follow-up landings to MozAUSLib.pm, and I moved the tags.

(In reply to comment #20) > Comment on attachment 499295 [details] [diff] [review] > fix 1:1 platform:ABI assumptions in tools repo > > > - update_platform = buildbot2updatePlatform(platform) > > + update_platforms = buildbot2updatePlatform(platform) > > A nit. Wouldn't it be better to rename buildbot2updatePlatform to > buildbot2updatePlatforms because it returns a list of platforms? > > Great progress!!! Definitely. I'll fix that on check-in.

Comment on attachment 499295 [details] [diff] [review] fix 1:1 platform:ABI assumptions in tools repo changeset: 1915:035c7877c38c

Had to fix up the _R11_1 tag a little bit because I forgot to include mozilla/tools/release in it. In my tagging work directory I did: cd tools cvs co -d release mozilla/tools/release cvs up -AdP -r UPDATE_PACKAGING_R11 release cd release cvs tag -b UPDATE_PACKAGING_R11_minibranch . cvs tag UPDATE_PACKAGING_R11_1 .

We need this ABI so that Mac PPC users will still receive updates.

So, other than the missing ppc ABI, for which I just posted the fix for, this was 100% good in my staging 3.6.14 run. Identical snippets were generated for the correct ABIs, update verify passed. That, combined with my previous verifications makes me very confident that this is nearly ready to flip in production. I just wanted to wait for Rail to finish his 4.0 staging run before calling it good and bumping the UPDATE_PACKAGING tags in the production configs.

Comment on attachment 500103 [details] [diff] [review] add Darwin_ppc-gcc3-u-ppc-i386 ABI Landed this. On UPDATE_PACKAGING_R13: Checking in MozAUSLib.pm; /cvsroot/mozilla/tools/patcher/MozAUSLib.pm,v <-- MozAUSLib.pm new revision: 1.16; previous revision: 1.15 done [] bhearsum@voot:~/tmp/1/patcher$ cvs tag UPDATE_PACKAGING_R13 MozAUSLib.pm W MozAUSLib.pm : UPDATE_PACKAGING_R13 already exists on version 1.15 : NOT MOVING tag to version 1.16 [] bhearsum@voot:~/tmp/1/patcher$ cvs tag -F UPDATE_PACKAGING_R13 MozAUSLib.pm T MozAUSLib.pm And for UPDATE_PACKAGING_R11_1: Checking in MozAUSLib.pm; /cvsroot/mozilla/tools/patcher/MozAUSLib.pm,v <-- MozAUSLib.pm new revision: 1.13.2.3; previous revision: 1.13.2.2 done [] bhearsum@voot:~/tmp/2/patcher$ cvs tag UPDATE_PACKAGING_R11_1 MozAUSLib.pm W MozAUSLib.pm : UPDATE_PACKAGING_R11_1 already exists on version 1.13.2.2 : NOT MOVING tag to version 1.13.2.3 [] bhearsum@voot:~/tmp/2/patcher$ cvs tag -F UPDATE_PACKAGING_R11_1 MozAUSLib.pm T MozAUSLib.pm

Comment on attachment 500104 [details] [diff] [review] add ppc ABI to the tools library changeset: 1948:3247678ba39e

Worked fine while testing mac64->mac transition in bug 600931. No more pain with manual snippet manipulations! /(| ( : __\ \ _____ (____) `| (____)| | (____).__| (___)__.|_____

Everything is checked in, both my and Rail's tests passed, we're all done here, woohoo!