We should make gaia and gecko tarballs/zips available. In other words, b2g-26.0a1.en-US.android-arm.tar.gz and gaia.zip . They're generated from publicly available code and contain no proprietary bits.

For all device builds?

Hmm, good question. Maybe it would be better if we stuck to the emulator and publicly available devices. So, Hamachi, Unari, and Inagi. (and Nexus 4 when that starts)

Just to be explicit then, which devices shouldn't we publish gaia/gecko for?

To err on the side of caution, let's not publish for Leo and Helix at the moment.

This is very exciting news. I believe this will be a significant improvement over the current situation for many in our community who have devices but are unable to track the tip of development where bug finding and fixing is likely to be easiest and more impactful. How is this prioritized and is there anything that I can do to help move this forward.

Here's my situation: I vet community members to Gaia UI Tests (in Python), have them start with Desktop, and then in warranted cases, give them an Inari + send them builds, nearly daily -- I'd like to change it (here) so they (and anyone else who bootloader-flashes a ZTE Open, etc.) can find and test daily builds. :mwu, per my understanding, we also shouldn't publish for Hamachi/Buri, no? (Even though those devices, I've confirmed, are otherwise identical to their shipping counterpart.) Or is there a completely Mozilla RIL-only build for that, too? (As you know, we flash with comm-RIL repacks, but flash with -n).

When dealing with third parties (acquiring top tier content) for Firefox OS, we frequently need to have developers using cutting edge b2g features. It's extremely frustrating because we can't point them to download images and instructions; instead we must ask them to ship us the phone so we can update them, then mail them back. This makes the content acquisition process take significantly longer than it needs to. I've been working on an add on to help developers quickly flash back and forth between 1.0, 1.1, 1.2, 1.3+. https://github.com/nickdesaulniers/fxos-version-control It's not ready yet, I had a working prototype that I've started rewriting to be a cross platform version of the initial prototype (where everything was hard coded). It would be great to ship an add on for Firefox that would flash your phone for you. It could even recover phones that were soft bricked if we provide instructions for how to manually reboot each phone into its bootloader. The biggest blocker is that device images are private, except for geeksphones. The other is that bootloaders are locked on devices shipping to end users. Both could probably be fixed via branding agreements.

Hey guys, now that we're transitioning to Buri devices, this issue got crucial. Any updates?

I think the same concerns from bug 917621 would apply here - that is, we may not be able to redistribute these files because they contain compiled copies of audio/video codecs.

(In reply to Chris AtLee [:catlee] from comment #9) > I think the same concerns from bug 917621 would apply here - that is, we may > not be able to redistribute these files because they contain compiled copies > of audio/video codecs. Not a concern. These files only contain free codecs - VP8/Theora/Vorbis/etc.

Is that a green light for publishing our builds (for released devices)?

I wouldn't have opened this bug if I thought there were issues. I don't know what people are waiting on.

Let's do this!

Do we need this for the per-checkin builds, or just for nightly builds? Do we need to have the engineering builds publicly available as well?

This has some implications for disk space usage on the ftp server. Probably not going to be a problem, but please check with me before we go live with it.

Nightly is likely enough if we need to watch the disk space usage. Engineering builds can be pretty useful, so I would like them as well if possible.

https://github.com/escapewindow/mozharness/compare/gecko-gaia-upload Ready for testing. Pretty sure the b2g/nightly and b2g/tinderbox-builds dirs on stage.m.o will Just Work; I'm a lot less sure about the b2gtry stuff, which may need a user account configured on stage.

Looks like there's no named package-gecko target to package gecko for the emulator, so there will be no gecko package for emulator{,jb}.

Comment on attachment 8359553 [details] [diff] [review] in-tree configs It'll be easier to test once this lands, because mapper.

Able to test dep builds without mapper... There's a b2gtry on stage.m.o, yay! I'll have to tweak the upload paths to match the desktop stage.m.o paths, boo.

Comment on attachment 8359554 [details] [diff] [review] nuke the unused emulator-ics and unagi config dirs Review of attachment 8359554 [details] [diff] [review]: ----------------------------------------------------------------- I'd rather finish up bug 916134 and nuke 'emulator'. Nuking unagi is fine.

Comment on attachment 8359553 [details] [diff] [review] in-tree configs Pushed to b2g-inbound: https://hg.mozilla.org/integration/b2g-inbound/rev/82a4e7daeb13

Is there a public URL now for these builds?

Oops, forgot to [leave open]

I think this might work, but I still need to rework the script to use the new config format: https://github.com/escapewindow/mozharness/commit/78e98477cb24775fa2d36aa1dc05e00206307825

Nick: On a high level, do you see anything objectionable in this patch? I am porting some upload.py logic into b2g_build.py... I thought that might be easiest. I'm thinking this may take quite a bit of testing, and I'd rather find out I'm on the wrong path earlier.

Also viewable on github: https://github.com/escapewindow/mozharness/compare/gecko-gaia-upload

Comment on attachment 8360779 [details] [diff] [review] (needs testing) public_upload_wip This cleanup is awesome to see! >diff --git a/configs/b2g/releng-beta.py b/configs/b2g/releng-beta.py >+ "upload": { >+ "default": { >+ "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"), >+ "ssh_user": "b2gbld", >+ "upload_remote_host": "pvtbuilds2.dmz.scl3.mozilla.com", >+ "upload_remote_path": "/pub/mozilla.org/b2g/tinderbox-builds/%(branch)s-%(target)s/%(buildid)s", I think Dustin would like to do away with pvtbuilds2, so we could swap to using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a path prefix of '/mnt/pvt_builds' for anything that should stay private though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub). >diff --git a/configs/b2g/releng-emulator.py b/configs/b2g/releng-emulator.py >+ "post_upload_cmd": "post_upload.py --tinderbox-builds-dir ${branch}s-%(target)s -p b2g -i %(buildid)s --revision %(revision)s --release-to-tinderbox-dated-builds", "post_upload_nightly_cmd": "post_upload.py --tinderbox-builds-dir %(branch)s-%(target)s -b %(branch)s -p b2g -i %(buildid)s --revision %(revision)s --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated", >+ }, >+ }, Missing newline prior to post_upload_nightly_cmd. >diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py >+ def _do_postupload_upload(self, upload_dir, ssh_key, ssh_user, remote_host, ... >+ for dirpath, dirname, filenames in os.walk(upload_dir): >+ for f in filenames: >+ path = '%s/%s' % (dirpath, f) os.path.join() for portability ? >+ postupload_key = 'post_upload_ngithly_cmd' Typo in nightly. >+ if os.path.exists(dirs['abs_public_upload_dir']) and self.config['upload'].get('public'): Is it possible we might get here with an empty dir, or will gaia always get uploaded ?

(In reply to Nick Thomas [:nthomas] from comment #31) > Comment on attachment 8360779 [details] [diff] [review] > (needs testing) public_upload_wip > > This cleanup is awesome to see! Thanks! > >diff --git a/configs/b2g/releng-beta.py b/configs/b2g/releng-beta.py > >+ "upload": { > >+ "default": { > >+ "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"), > >+ "ssh_user": "b2gbld", > >+ "upload_remote_host": "pvtbuilds2.dmz.scl3.mozilla.com", > >+ "upload_remote_path": "/pub/mozilla.org/b2g/tinderbox-builds/%(branch)s-%(target)s/%(buildid)s", > > I think Dustin would like to do away with pvtbuilds2, so we could swap to > using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a > path prefix of '/mnt/pvt_builds' for anything that should stay private > though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub). I think we assume the remote path is equivalent to the path in the URL, so I may need to create a /pvt softlink to /mnt/pvt_builds. Puppet? > >diff --git a/scripts/b2g_build.py b/scripts/b2g_build.py > >+ def _do_postupload_upload(self, upload_dir, ssh_key, ssh_user, remote_host, > ... > >+ for dirpath, dirname, filenames in os.walk(upload_dir): > >+ for f in filenames: > >+ path = '%s/%s' % (dirpath, f) > > os.path.join() for portability ? I'm explicitly using '/' because this is the directory / filepath that will be passed over ssh to the post_upload.py call, and I assume we will never have a windows upload server. I may be wrong, but I hope not. Maybe I'll add a comment. > >+ if os.path.exists(dirs['abs_public_upload_dir']) and self.config['upload'].get('public'): > > Is it possible we might get here with an empty dir, or will gaia always get > uploaded ? Gaia's always added. I nuke the public upload dir during clobber, and only recreate it by copy_to_upload_dir() calls, so I think we shouldn't have an empty dir.

(In reply to Aki Sasaki [:aki] from comment #32) > > I think Dustin would like to do away with pvtbuilds2, so we could swap to > > using pvtbuilds.pvt.build.mozilla.org. Need to be careful about adding a > > path prefix of '/mnt/pvt_builds' for anything that should stay private > > though (pvtbuilds.pvt.build.m.o:/pub is actually http://ftp.m.o/pub). > > I think we assume the remote path is equivalent to the path in the URL, so I > may need to create a /pvt softlink to /mnt/pvt_builds. Puppet? There actually is one already, and in your patch we're using pvtbuilds.pvt whenever the path starts with /pvt. So that leaves the private builds in /pub case. I'm actually don't understand what the distinction is between pub and pvt any more, and the access controls are all wacky (yay, legal issues). Perhaps it's time to rationalise that too.

Testing's going well for private tinderbox-builds (with softlink!) Getting this for post_upload.py public nightly though: 13:04:24 INFO - Copy/paste: ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa dev-stage01.srv.releng.scl3.mozilla.com "post_upload.py --tinderbox-builds-dir mozilla-central-hamachi-eng -b mozilla-central -p b2g -i 20140117152452 --revision bbe08810e87b --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated /tmp/tmp.uSXhlgN814/ \"/tmp/tmp.uSXhlgN814//sources.xml\" \"/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip\" \"/tmp/tmp.uSXhlgN814//gaia.zip\" \"/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.tar.gz\"" 13:04:24 INFO - sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'mozilla-central-hamachi-eng', '-b', 'mozilla-central', '-p', 'b2g', '-i', '20140117152452', '--revision', 'bbe08810e87b', '--release-to-tinderbox-dated-builds', '--release-to-latest', '--release-to-dated', '/tmp/tmp.uSXhlgN814/', '/tmp/tmp.uSXhlgN814//sources.xml', '/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip', '/tmp/tmp.uSXhlgN814//gaia.zip', '/tmp/tmp.uSXhlgN814//b2g-29.0a1.en-US.android-arm.tar.gz'] 13:04:24 INFO - Traceback (most recent call last): 13:04:24 INFO - File "/usr/local/bin/post_upload.py", line 551, in <module> 13:04:24 INFO - func(options, upload_dir, files) 13:04:24 INFO - File "/usr/local/bin/post_upload.py", line 201, in ReleaseToLatest 13:04:24 INFO - CopyFileToDir(f, upload_dir, latestPath) 13:04:24 INFO - File "/usr/local/bin/post_upload.py", line 103, in CopyFileToDir 13:04:24 INFO - tmp_fd, tmp_path = tempfile.mkstemp(dir=dest_dir) 13:04:24 INFO - File "/usr/lib64/python2.6/tempfile.py", line 293, in mkstemp 13:04:24 INFO - return _mkstemp_inner(dir, prefix, suffix, flags) 13:04:24 INFO - File "/usr/lib64/python2.6/tempfile.py", line 228, in _mkstemp_inner 13:04:24 INFO - fd = _os.open(file, flags, 0600) 13:04:24 INFO - OSError: [Errno 13] Permission denied: '/home/ftp/pub/b2g/nightly/latest-mozilla-central/tmpnykvkH' 13:04:24 ERROR - Return code: 1 13:04:24 ERROR - failed to run post_upload.py --tinderbox-builds-dir mozilla-central-hamachi-eng -b mozilla-central -p b2g -i 20140117152452 --revision bbe08810e87b --release-to-tinderbox-dated-builds --release-to-latest --release-to-dated! I think I just need to track down an issue with my cmdline... maybe the trailing slash?

Oh, /home/ftp/pub. Hm.

(In reply to Aki Sasaki [:aki] from comment #35) > Oh, /home/ftp/pub. Hm. I think this is a ffxbld/b2gbld issue, since stage uses ffxbld and pvtbuilds uses b2gbld, but dev-stage01 can't have both users own the same directory tree. Did a recursive chmod g+w on b2g/nightly and am trying again.

Phew, that took a lot longer than I expected. Successful staging dep run: http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_emulator_dep/builds/6/steps/run_script/logs/stdio Successful staging nightly run: http://dev-master01.build.mozilla.org:8052/builders/b2g_mozilla-central_hamachi_eng_nightly/builds/6/steps/run_script/logs/stdio

Comment on attachment 8362066 [details] [diff] [review] gecko-gaia-upload Review of attachment 8362066 [details] [diff] [review]: ----------------------------------------------------------------- I didn't look at this in great detail because I ended up doing that for the earlier feedback request, and the interdiff was small. There is one big issue though ... ::: configs/b2g/releng-emulator.py @@ +17,5 @@ > + "upload": { > + "default": { > + "ssh_key": os.path.expanduser("~/.ssh/b2gbld_dsa"), > + "ssh_user": "b2gbld", > + "upload_remote_host": "pvtbuilds.pvt.build.mozilla.org", The effect of taking this as-is would be to put private bits on the public ftp server. If you log in to both pvtbuilds2 and pvtbuilds.pvt and compare /pub/mozilla.org/b2g/nightly/ you'll see what I mean (the former has private stuff, the later is public b2g desktop builds). My suggestion to swap to this was actually a bad idea, at least in the short-term, so lets just stick with pvtbuilds2 for now, in all these config files where it's already used.

Interdiff: https://github.com/escapewindow/mozharness/commit/a1cdc5d694ef87ddd4cfc7f62eb3eccce75705ed

Comment on attachment 8363065 [details] [diff] [review] back to pvtbuilds2 r+. Please watch carefully after this lands in case we missed something.

Comment on attachment 8363065 [details] [diff] [review] back to pvtbuilds2 Landed on default, kicked off emulator builds on cypress. http://hg.mozilla.org/build/mozharness/rev/9e6b874c256b

Ok. The uploads look good on Cypress, and we have softlinks on pvtbuilds tinderbox-builds. We are getting this message I'd like to patch (just skip it for now; we can parse post_upload.py output later if needed). Not urgent: 18:12:50 INFO - Copy/paste: ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_dsa stage.mozilla.org "post_upload.py --tinderbox-builds-dir cypress-generic-debug -p b2g -i 20140120083431 --revision 45a22c4901bb --release-to-tinderbox-dated-builds /tmp/tmp.7XJbZeIsPm/ /tmp/tmp.7XJbZeIsPm//gaia.zip /tmp/tmp.7XJbZeIsPm//sources.xml /tmp/tmp.7XJbZeIsPm//gaia-tests.zip /tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip /tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.tests.zip" 18:12:51 INFO - sys.argv: ['/usr/local/bin/post_upload.py', '--tinderbox-builds-dir', 'cypress-generic-debug', '-p', 'b2g', '-i', '20140120083431', '--revision', '45a22c4901bb', '--release-to-tinderbox-dated-builds', '/tmp/tmp.7XJbZeIsPm/', '/tmp/tmp.7XJbZeIsPm//gaia.zip', '/tmp/tmp.7XJbZeIsPm//sources.xml', '/tmp/tmp.7XJbZeIsPm//gaia-tests.zip', '/tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip', '/tmp/tmp.7XJbZeIsPm//b2g-29.0a1.en-US.android-arm.tests.zip'] 18:12:51 INFO - http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/gaia.zip 18:12:51 INFO - http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/sources.xml 18:12:51 INFO - http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/gaia-tests.zip 18:12:51 INFO - http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/b2g-29.0a1.en-US.android-arm.crashreporter-symbols.zip 18:12:52 INFO - http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/cypress-generic-debug/1390235671/b2g-29.0a1.en-US.android-arm.tests.zip 18:12:53 INFO - Return code: 0 18:12:53 INFO - Upload successful: http://stage.mozilla.org//tmp/tmp.7XJbZeIsPm/

(I'll merge to production tomorrow morning.)

(In reply to Aki Sasaki [:aki] from comment #43) > (I'll merge to production tomorrow morning.) I just did that: https://hg.mozilla.org/build/mozharness/rev/8e4b5b9fbe71

Comment on attachment 8363399 [details] [diff] [review] silence_public_upload https://hg.mozilla.org/build/mozharness/rev/1c3cd27cbbb5

Bustage fix: https://hg.mozilla.org/build/mozharness/rev/a7182b58a7b3

I think we're done here.

Is there a public url for these builds now (if so, where)?

http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/

/nightly/ doesn't seem to have all device builds, only emulator builds and hamachi builds. Is this intentional? Should we be pointing people to /tinderbox-builds/ then?

The logs have the upload locations. * not all device builds upload depend builds to tinderbox-builds * not all device builds have nightly builds * since this was just enabled, we haven't done a full set of nightlies across all branches yet What device/branch are you specifically concerned about?

I'd like to see combinations of: Devices: [unagi, inari, hamachi, mako (n4)] FxOS versions: [1.0, 1.1, 1.2, 1.3, master (1.4)] We send many devices to third party app developers and have been in a pinch when they need to update devices. It would be great to point them to a specific dir for each device so they could quickly find a build for what version of FxOS they're looking for. I'd also like to write an add on that can do this automatically when a phone is connected, so having a known dir to look for will help.

(In reply to Nick Desaulniers [:\n] from comment #52) > I'd like to see combinations of: > > Devices: [unagi, inari, hamachi, mako (n4)] > FxOS versions: [1.0, 1.1, 1.2, 1.3, master (1.4)] > > We send many devices to third party app developers and have been in a pinch > when they need to update devices. It would be great to point them to a > specific dir for each device so they could quickly find a build for what > version of FxOS they're looking for. > > I'd also like to write an add on that can do this automatically when a phone > is connected, so having a known dir to look for will help. I think we're going to have to manually keep that updated, especially since 1.4 is going to move to mozilla-aurora then to b2g30_v1_4 later. Maybe we should have a b2g equivalent of http://nightly.mozilla.org/ ? Also, unagi builds are no longer supported.

Ok, this is eating a ton of disk space on ftp.m.o... overall ~4TB aiui, and it's putting all infrastructure at risk. I'm thinking about: * turning off all public try uploads * turning off all public depend uploads except for emulator (nightly only for devices) Any objections?

Sorry, I mispoke on IRC. There is 3TB of space used by /pub/mozilla.org/b2g/tindebox-builds/*gecko (eg mozilla-inbound-linux32_gecko some 270GB), but I don't see any gecko/gaia archives there. I'm sure they're contributing elsewhere, because there's a clear downward trend on the free space which starts on the 21st or 22nd, but don't have numbers yet.

Backed out due to killing ftp.m.o: http://hg.mozilla.org/build/mozharness/rev/1c8df71deec8 http://hg.mozilla.org/build/mozharness/rev/b296efca7838 http://hg.mozilla.org/build/mozharness/rev/69c24d23e5e1

(In reply to Aki Sasaki [:aki] from comment #54) > Ok, this is eating a ton of disk space on ftp.m.o... overall ~4TB aiui, and > it's putting all infrastructure at risk. > > I'm thinking about: > * turning off all public try uploads > * turning off all public depend uploads except for emulator (nightly only > for devices) > > Any objections? Nick: I'm willing to do the above, which should limit the amount of additional load on ftp.m.o, but only if we think it won't kill ftp.m.o again anyway. Do you have any more data? If the above won't help, maybe this bug should be morphed into an S3 upload or something. ... also, I'm not finding the ftp filling up bug to link here: do you have it handy?

Pretty much the same, but: * combined the two previous patches * updated the new fota configs * removed the public upload config from the try config file * only pushes public bits if self.query_is_nightly() The other change we might want is to add nightly builds for more devices + emulator.

[12:35] <hwine> aki: re https://bugzilla.mozilla.org/show_bug.cgi?id=914111 - check in with catlee before you deploy, since that will impact the ipsec traffic issue

should be fine if it's only pushing bits nightly

Comment on attachment 8367530 [details] [diff] [review] public_upload_nightly_only Sorry, I've expended all my tokens and will have to look at this first thing next week.

Comment on attachment 8367530 [details] [diff] [review] public_upload_nightly_only I agree with catlee that this should be OK on ftp.m.o. Sorry for the delay.

Comment on attachment 8367530 [details] [diff] [review] public_upload_nightly_only https://hg.mozilla.org/build/mozharness/rev/552c85b84fe9

Ok, we're uploading to ftp.m.o again, but only for nightlies (not depend builds, not try)

Aaaaand relanded bustage fix http://hg.mozilla.org/build/mozharness/rev/a7182b58a7b3 : http://hg.mozilla.org/build/mozharness/rev/89068c9307db

Aaaaaaaaaaaand relanded try bustage fix http://hg.mozilla.org/build/mozharness/rev/b66a53a7144d