User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 Build ID: 20150828185855 Steps to reproduce: Since bug #672509 the Gecko SDK is not built anymore as part of the xulrunner build (which has been stopped as of Firefox 42), but as part of Firefox's build. However, only the nightly Aurora and Mozilla Central builds happen to produce the SDK. It is not made available for Beta releases nor Stable releases of Firefox. If my analysis is correct, this means that mozilla does not produce anymore builds of the SDK for beta and stable. Expected results: I have seen various posts of people looking for the SDK and I would like mozilla to consider releasing it again for beta and stable releases of Firefox. It is currently missing for Firefox 42 and 43.

In addition to that, the wiki page https://developer.mozilla.org/en-US/docs/Gecko_SDK is obsolete since it has not been updated.

I can't find them either, which quite surprises me. I see that MOZ_AUTOMATION_SDK=1 is set in logs such as http://ftp.mozilla.org/pub/firefox/candidates/43.0-candidates/build1/logs/release-mozilla-release-linux64_build-bm72-build1-build3.txt.gz, which should cause them to get built...

(In reply to Ben Hearsum (:bhearsum) from comment #2) > I can't find them either, which quite surprises me. I see that > MOZ_AUTOMATION_SDK=1 is set in logs such as > http://ftp.mozilla.org/pub/firefox/candidates/43.0-candidates/build1/logs/ > release-mozilla-release-linux64_build-bm72-build1-build3.txt.gz, which > should cause them to get built... Any idea about this, glandium?

The MOZ_AUTOMATION variables only get used if building with './mach build', which is what mozharness runs. Beta & release builds are still using explicit calls from buildbot, like 'make -f client.mk build', 'make package', 'make buildsymbols', etc. Did we have a buildbot step for 'make sdk' at one point? If so, it's probably easiest to add that back in for beta/release, rather than try to convert them to mozharness.

(In reply to Michael Shal [:mshal] from comment #4) > The MOZ_AUTOMATION variables only get used if building with './mach build', > which is what mozharness runs. Beta & release builds are still using > explicit calls from buildbot, like 'make -f client.mk build', 'make > package', 'make buildsymbols', etc. > > Did we have a buildbot step for 'make sdk' at one point? If so, it's > probably easiest to add that back in for beta/release, rather than try to > convert them to mozharness. I think we did as part of XULRunner builds, but not Firefox... The tricky part might be that we need to upload them too, not just build them. Maybe that will "just work" once they're on disk?

From IRC: 13:39 <~bhearsum> mshal: any idea if MOZ_AUTOMATION_SDK will be necessary to get "make upload" to pick up the sdk? 13:40 < mshal> bhearsum: I don't believe so - it should just drive whether or not 'mach build' runs 'make sdk' 13:40 <~bhearsum> ah, okay... 13:40 <~bhearsum> this seems like it _should_ be easy to hack in then... 13:41 < mshal> bhearsum: the SDK should get picked up by the upload-files.mk if it exists before 'make upload': https://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/upload-files.mk#743 Flipping packageSDK _might_ do the trick...it should cause "make -f client.mk sdk" to be run, per https://github.com/mozilla/build-buildbotcustom/blob/master/process/factory.py#L1596

Hi, I don't want to sound too impatient, but we'd *really* need the SDKs for Gecko 42, 43 and 44 beta, if someone can make them built. Also, status of the bug is still UNCONFIRMED, I suppose it could be changed now that it's been confirmed.

mshal: can you take a look at this?

Mihai had already volunteered for this in a releng email. Mihai, feel free to ping me if you have any questions!

Please accept my apologies for delay here, have been PTO last week. I have been greedy and selfish when took this bug; now I'll have to defer as I've got a lot on my plate with the releaseduty stuff this week and the next. @mshal: if you have time to help here, it would be great. If not I'll reassign it to me and have a look at it by the end of next week. Sorry again.

Is this all that's needed? I was able to verify in a staging "release-mozilla-release-linux_build" and "release-mozilla-beta-linux_build" that the make_sdk step shows up. I believe that should cause it to get built, and since it's before 'make upload' it should also get uploaded. Are these the right buildertypes to check? I can't completely verify the builds in staging since I haven't found a way to actually run them to completion.

Comment on attachment 8710100 [details] [diff] [review] packagesdk Review of attachment 8710100 [details] [diff] [review]: ----------------------------------------------------------------- This will turn make_sdk on for Thunderbird and Fennec as well, so it might break those release builds unless the command works fine on them. The safest thing to do here is probably default to False, and set it to True in Firefox release configs.

Dug a bit for the context of this in IRC and found the following, hope it helps. 15:59:54 <mtabara> bhearsum: re: 1233005. flipping the packageSDK as easy as changing https://github.com/mozilla/build-buildbotcustom/blob/master/process/factory.py#L811 default value to True, or by passing it along when calling for MercurialBuildFactory? 16:00:01 <~bhearsum> mtabara: so, i don't think you can change the default value of packageSDK - that will probably have a lot of unwanted consequences 16:00:23 <~bhearsum> so setting it to True for Firefox builds in release automation is likely what you want 16:00:45 <~bhearsum> the trick there is that you want to make sure it's false for Fennec and Thunderbird releases still...so you may need to add a new item to the release configs 16:10:47 — mtabara digs to find usages of MercurialBuildFactory 16:10:50 <~bhearsum> okay, so ReleaseBuildFactory (a subclass of MercurialBuildFactory) is where packageSDK will need to be set: https://github.com/mozilla/build-buildbotcustom/blob/master/process/release.py#L585 16:11:20 <~bhearsum> that code gets run for Firefox, Fennec, and Thunderbird builds - so you need to make sure it's only true for Firefox 16:11:49 <~bhearsum> one way to do that would be to set a variable in the release configs (such as https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release-firefox-mozilla-release.py.template), and set packageSDK to true/false based on that 16:12:39 <~bhearsum> dump_master.py is probably a good way to do initial verification, have you used it before? 16:12:48 <mtabara> nope 16:13:53 <~bhearsum> okay, what it does is dump the entire generated buildbot config to a file. what we typically do is run it once before making changes, then run it again afterwards - you can see the actual effect your changes make by diffing the before/after output 16:14:18 <~bhearsum> https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques talks about it

Now based off of config settings.

Comment on attachment 8710182 [details] [diff] [review] packagesdk.patch Review of attachment 8710182 [details] [diff] [review]: ----------------------------------------------------------------- Thanks Mike!

Comment on attachment 8710182 [details] [diff] [review] packagesdk.patch https://hg.mozilla.org/build/buildbotcustom/rev/5d15ab1c5ddd

Comment on attachment 8710183 [details] [diff] [review] bbconfigs-sdk.patch https://hg.mozilla.org/build/buildbot-configs/rev/9f0795fa7184

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9f0795fa7184

In production: https://hg.mozilla.org/build/buildbotcustom/rev/5d15ab1c5ddd

Something is broken with the landed patch for release builds. Please see bug 1242374.

Do the previous commits mean that the SDK has been built with Firefox 44? If so, where can we find it? Also, will the SDK be made available for Firefox 42 and 43?

(In reply to Samuel Verschelde from comment #22) > Do the previous commits mean that the SDK has been built with Firefox 44? If > so, where can we find it? Yeah, the SDK was built for 44.0. It didn't make its way to the releases directory (but should for the next release). You can find it in http://ftp.mozilla.org/pub/firefox/candidates/44.0-candidates/build3/win32/en-US/ and the other en-US directories. > Also, will the SDK be made available for Firefox 42 and 43? No plans to backfill these now that 44.0 is released. What's the reasoning behind wanting them at this point?

(In reply to Ben Hearsum (:bhearsum) from comment #23) > Yeah, the SDK was built for 44.0. It didn't make its way to the releases > directory (but should for the next release). You can find it in > http://ftp.mozilla.org/pub/firefox/candidates/44.0-candidates/build3/win32/ > en-US/ and the other en-US directories. Thanks. > > Also, will the SDK be made available for Firefox 42 and 43? > > No plans to backfill these now that 44.0 is released. What's the reasoning > behind wanting them at this point? We're maintaining a product which allows the use of any of all different versions of gecko since 1.9.1 so that companies can select the specific version they need for vendor or internal web apps with compatibility issues. Honestly, in the case of version 42 and 43, the probability that users need specifically those versions is low. Either they need the latest, or an old one for some old unmaintained internal web app. Having a hole in the versions is not very good for image, though. Another reason is that sometimes you need to build against a specific version to test for a behaviour change or bug, and having the SDK pre-built is handier than having to built it yourself.

Just noticed that the mac sdk is still screwed up. It ends up at http://ftp.mozilla.org/pub/firefox/candidates/44.0-candidates/build3/Firefox-44.0.en-US.mac-x86_64.sdk.tar.bz2.

(In reply to Ben Hearsum (:bhearsum) from comment #25) > Just noticed that the mac sdk is still screwed up. It ends up at > http://ftp.mozilla.org/pub/firefox/candidates/44.0-candidates/build3/Firefox- > 44.0.en-US.mac-x86_64.sdk.tar.bz2. I filed bug 1243750 for this, to avoid piling on additional work here. At least it's published now. I also filed bug 1243740 with a patch to make sure the SDKs are automatically synced to /releases/ in the future (I manually synced them for 44.0 and 45.0b1 just now).

There will probably be the need for updating https://developer.mozilla.org/en-US/docs/Gecko_SDK too, or create another, more recent, page dedicated to the SDK which explains where to find it?

I got a report filed for mozdownload (https://github.com/mozilla/mozdownload/issues/353) which doesn't allow us anymore to download the real binary for the platform, because the SDK takes precedence now. Why do we save the SDK under https://archive.mozilla.org/pub/mozilla.org/firefox/candidates/44.0-candidates/build3/linux-i686/en-US/firefox-44.0.sdk.tar.bz2 right next to the build now? We never did that before. Can't we use a folder like [..]/candidates/44.0-candidates/build3/linux-i686/sdk/? We do not ship it with different locales so I really don't see why it should be put under en-US only.

(In reply to Henrik Skupin (:whimboo) from comment #28) > I got a report filed for mozdownload > (https://github.com/mozilla/mozdownload/issues/353) which doesn't allow us > anymore to download the real binary for the platform, because the SDK takes > precedence now. > > Why do we save the SDK under > https://archive.mozilla.org/pub/mozilla.org/firefox/candidates/44.0- > candidates/build3/linux-i686/en-US/firefox-44.0.sdk.tar.bz2 right next to > the build now? We never did that before. Can't we use a folder like > [..]/candidates/44.0-candidates/build3/linux-i686/sdk/? We do not ship it > with different locales so I really don't see why it should be put under > en-US only. It goes there because that's where the build system puts it. Not sure how easy it is to adjust...it may inherit a lot of package naming from Firefox now that its built with it. I don't think it's a high priority to move this...

So this bug says that we haven't build the SDK since Firefox 42. When I check the release folder for e.g. the 41.0 release I cannot find any SDK package under the same location. Where has it been placed before? Something has definitely changed here.

(In reply to Henrik Skupin (:whimboo) from comment #30) > So this bug says that we haven't build the SDK since Firefox 42. When I > check the release folder for e.g. the 41.0 release I cannot find any SDK > package under the same location. Where has it been placed before? Something > has definitely changed here. Yes, it changed. We used to build it as part of XULRunner, but we moved it over to Firefox when we killed that. Used to be published to directories such as: http://ftp.mozilla.org/pub/xulrunner/releases/41.0/sdk/

I see. With Xulrunner we had a single folder for all builds which is pretty nice. So if we are really not going to change that I will have to get mozdownload fixed to not download the SDK if you want the actual binary. Btw there are no SDK builds on Mac: http://archive.mozilla.org/pub/firefox/releases/44.0/mac/en-US/

(In reply to Henrik Skupin (:whimboo) from comment #32) > I see. With Xulrunner we had a single folder for all builds which is pretty > nice. So if we are really not going to change that I will have to get > mozdownload fixed to not download the SDK if you want the actual binary. I think this is because the build system has specific logic when MOZ_APP_NAME == xulrunner. See also: https://bugzilla.mozilla.org/show_bug.cgi?id=1243750#c1 > Btw there are no SDK builds on Mac: > http://archive.mozilla.org/pub/firefox/releases/44.0/mac/en-US/ bhearsum commented about this in #c25 and #c26 - please add your thoughts to bug 1243750.

I believe we have SDKs for linux/mac/win i386 & x86_64 now, so we can close this. Keep me posted if there are further issues.