Bug 1116074 is to support emulator-lollipop local build. Eventually, we will want to have automatic tests for emulator-lollipop on treeherder. And this is the bug to track tasks to make emulator-L run on TPBL.

Can we bump the priority on this please? At the moment, Nexus 5-L builds are the only Lollipop-based builds we run in automation. However, since B2G device image builds can't be run on Try, that leaves developers with no way to test their patches for possible L-only bustage short of pushing to inbound and crossing their fingers. This isn't a hypothetical concern either, such a situation happened today. Note that getting tests running on the builds is orthogonal to this discussion. I just think that we need to give developers an ability to verify their changes on Try before pushing, and an emulator build would go a long way to satisfying that.

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #1) > Can we bump the priority on this please? At the moment, Nexus 5-L builds are > the only Lollipop-based builds we run in automation. However, since B2G > device image builds can't be run on Try, that leaves developers with no way > to test their patches for possible L-only bustage short of pushing to > inbound and crossing their fingers. This isn't a hypothetical concern > either, such a situation happened today. > > Note that getting tests running on the builds is orthogonal to this > discussion. I just think that we need to give developers an ability to > verify their changes on Try before pushing, and an emulator build would go a > long way to satisfying that. Hi Ryan, Currently Seinlin and Edgar are working on bug 1116074. Once that gets fixed, we will try to make a L build on try as soon as possible; and as you said, that doesn't need to happen after tests on L are ready.

Comment on attachment 8567057 [details] [diff] [review] Bug 1126695: Added B2G config for emulator-l (on m-c) [checkin: comment 39] Sorry, I'm not a peer on this code.

I looked at bug 979450 and bug 984066, and copied what was done there.

Comment on attachment 8567059 [details] [diff] [review] Bug 1126695: Added B2G L emulator (on tbpl) Review of attachment 8567059 [details] [diff] [review]: ----------------------------------------------------------------- LGTM, but it's far more critical to get this fixed on Treeherder than TBPL at this point. To patch Treeherder, you need to modify: https://github.com/mozilla/treeherder-service/blob/master/treeherder/etl/buildbot.py#L273 https://github.com/mozilla/treeherder-ui/blob/master/webapp/app/js/values.js (both thPlatformNameMap and thPlatformOrder) You can ask edmorley to review. Also, if I can ask a favor of you, while you're editing values.js, can you also add android-4-4-armv7-api11 to thPlatformOrder in the right spot? I forgot to do that when I added support for Android 4.4 tests recently.

Comment on attachment 8567058 [details] [diff] [review] Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) [checkin: comment 41] I don't think I'm an appropriate reviewer for this.

OK, I'll supply a TH patch as well.

Comment on attachment 8567061 [details] [diff] [review] Bug 1126695: Added emulator-l to build configs (on buildbot_configs) Review of attachment 8567061 [details] [diff] [review]: ----------------------------------------------------------------- I've applied this patch to my local buildbot, and it fails to "make checkconfigure" with this error: File ".../master/b2g_config.py", line 1945, in <module> BRANCHES['try']['platforms']['emulator-l']['slaves'] = TRY_SLAVES['mock'] KeyError: 'emulator-l' You should probably add 'emulator-l' and 'emulator-l-debug' in 'try' section, around line 1639

Comment on attachment 8567063 [details] [diff] [review] Bug 1126695: Added emulator-l to trychooser (on tools) [checkin: comment 40] Should work.

Changes since v1: - added emulator-l to try branches (near line 1650) - enabled nightly builds with m-c and v2.2 (near line 1760) - remove emulator-{l,l-debug} for Gecko versions before 37 (near line 1950) I talked to RyanVM about the nightly builds. The Gecko version cleanups near line 1950 are similar to emulator-kk's cleanups on anything before b2g 1.4. emulator-l should only be build on b2g 2.2 or later.

As discussed with RyanVM, JB/L opt builds shall run on every push; while JB/L debug builds shall only run periodically. I hope this patch already does the job.

Comment on attachment 8567058 [details] [diff] [review] Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) [checkin: comment 41] Review of attachment 8567058 [details] [diff] [review]: ----------------------------------------------------------------- the only thing missing is the b2g-lollipop tag on the platform_hardware_ril repo. please make sure to create that branch on that repo before landing this patch

Comment on attachment 8567112 [details] [diff] [review] [02] Bug 1126695: Run run JB,L opt on pushes; and JB,L debug periodically This needs changes to the Try config or we're gonna have a bad time. https://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/b2g_config.py#l1883

Comment on attachment 8567059 [details] [diff] [review] Bug 1126695: Added B2G L emulator (on tbpl) TBPL is code-frozen now, so we'll just want the Treeherder piece, not the TBPL one.

Changes since v1: - override try to build pushes but not periodically; as advised by Ryan

Comment on attachment 8567091 [details] Github pull request for treeherder-ui [checkin: comment 28] These Treeherder PRs look great - I'll merge the service one straight away. If you could make the addition that Ryan mentioned on the UI one (adding a missing entry from a previous bug), squash/rebase/force-push, and comment on the PR (so I get notified; commits themselves don't trigger email) I'll merge that too :-)

Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/9bcc8aaa743e4a215107e25bc7548233fb7f3f83 Bug 1126695: Added emulator-l to build bot https://github.com/mozilla/treeherder-service/commit/442b5cc086b7acf80bd46f29ee9f8f38106a91b3 Merge pull request #390 from tdz/bug-1226695 Bug 1126695 - Add support for B2G emulator-l jobs

(In reply to Chris AtLee [:catlee] from comment #20) > Comment on attachment 8567058 [details] [diff] [review] > Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) > > Review of attachment 8567058 [details] [diff] [review]: > ----------------------------------------------------------------- > > the only thing missing is the b2g-lollipop tag on the platform_hardware_ril > repo. please make sure to create that branch on that repo before landing > this patch Edgar, can you take care of this?

Comment on attachment 8567091 [details] Github pull request for treeherder-ui [checkin: comment 28] Updated Github pull request - added entries for android-{4-4,5-0}-armv7-api11

Commits pushed to master at https://github.com/mozilla/treeherder-ui https://github.com/mozilla/treeherder-ui/commit/a628187e9f94b8fb9b0b7012058f8b0acec38993 Bug 1126695: Added android-{4-4,5-0}-armv7-api11 and emulator-l to Treeherder UI The entries for android-{4-4,5-0}-armv7-api11 were not added in bug 1112693, so we do that now. https://github.com/mozilla/treeherder-ui/commit/a50c62524c3db0e477052c1c2afcbc7ffc91c8cd Merge pull request #380 from tdz/bug-1126696 Bug 1126695: Added emulator-l to Treeherder UI

Comment on attachment 8567124 [details] [diff] [review] [02] Bug 1126695: Run run JB,L opt on pushes; and JB,L debug periodically (v2) Review of attachment 8567124 [details] [diff] [review]: ----------------------------------------------------------------- I cannot apply this patch locally... hg pull -u; hg update -C default; hg import --no-commit https://bugzilla.mozilla.org/attachment.cgi?id=8567124 pulling from ssh://hg.mozilla.org/build/buildbot-configs searching for changes no changes found 1 files updated, 0 files merged, 0 files removed, 0 files unresolved applying https://bugzilla.mozilla.org/attachment.cgi?id=8567124 patching file mozilla/b2g_config.py Hunk #2 FAILED at 1287 Hunk #3 FAILED at 1355 Hunk #5 FAILED at 1937 I cannot apply this patch in fresh clone of buildbot-configs, could you regenerate it? Thanks!

Comment on attachment 8567099 [details] [diff] [review] [01] Bug 1126695: Added emulator-l to build configs (on buildbot_configs) (v2) Looks like the wrong patch got uploaded. This is the m-c patch (attachment 8567057 [details] [diff] [review]).

Changes since v1: - please see comment #18 There are too many repos here and I'm getting confused. This should be the correct patch.

Changes since v2: - rebased onto [01]

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #26) > (In reply to Chris AtLee [:catlee] from comment #20) > > Comment on attachment 8567058 [details] [diff] [review] > > Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) > > > > Review of attachment 8567058 [details] [diff] [review]: > > ----------------------------------------------------------------- > > > > the only thing missing is the b2g-lollipop tag on the platform_hardware_ril > > repo. please make sure to create that branch on that repo before landing > > this patch > > Edgar, can you take care of this? Hi Chris and Thomas, platform_hardware_ril is moved to new branching model [1], we are going to branch according to ril version, so there will be no b2g-lollipop, but b2g-ril_v* for example. And currently platform_hardware_ril is only supporting ril version used in kitkat, there is a bug tracking lollipop status [2]. Before the platform_hardware_ril is ready for lollipop, emulator-l fetches hardware/ril codes from aosp directly in b2g-manifest [3]. 1. Is it okay for B2G bumper script with above configuration? 2. Once platform_hardware_ril is ready for lollipop, b2g-manifest needs a revise to fetches code back from platform_hardware_ril, do we need to do anything in B2G bumper? Thank you. [1] https://wiki.mozilla.org/B2G/Emulator#Branch_Model [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1134982 [3] Please see https://github.com/mozilla-b2g/b2g-manifest/pull/286

The bumper only requires that all the repositories / references in the manifests be mirrored onto git.m.o. It doesn't care about the actual branch names. When I looked at your initial patch, the bumper process failed because the branch specified in the manifest didn't exist yet in the repositories.

Nexus 5-L builds were enabled on b2g37_v2_2 today. Can we please make sure that Emulator-L builds are also run there? Same setup as trunk should be fine.

(In reply to Chris AtLee [:catlee] from comment #34) > The bumper only requires that all the repositories / references in the > manifests be mirrored onto git.m.o. It doesn't care about the actual branch > names. > > When I looked at your initial patch, the bumper process failed because the > branch specified in the manifest didn't exist yet in the repositories. I see, thanks for this clarification.

(In reply to Edgar Chen [:edgar][:echen] from comment #33) > > platform_hardware_ril is moved to new branching model [1], we are going to > branch according to ril version, so there will be no b2g-lollipop, but > b2g-ril_v* for example. > And currently platform_hardware_ril is only supporting ril version used in > kitkat, there is a bug tracking lollipop status [2]. > Before the platform_hardware_ril is ready for lollipop, emulator-l fetches > hardware/ril codes from aosp directly in b2g-manifest [3]. ^^^^ Sorry it should be `caf` not `aosp`. And the caf/platform/hardware/ril is mirrored onto git.m.o already, http://git.mozilla.org/?p=external/caf/platform/hardware/ril.git;a=summary. Thank you. > > [1] https://wiki.mozilla.org/B2G/Emulator#Branch_Model > [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1134982 > [3] Please see https://github.com/mozilla-b2g/b2g-manifest/pull/286

Now that bug 1116074 landed, emulator-l builds are supported.

https://hg.mozilla.org/integration/b2g-inbound/rev/ea2b7d6407c4 https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/94530180141f

Comment on attachment 8567063 [details] [diff] [review] Bug 1126695: Added emulator-l to trychooser (on tools) [checkin: comment 40] https://hg.mozilla.org/build/tools/rev/82e3589bc31f

Comment on attachment 8567058 [details] [diff] [review] Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) [checkin: comment 41] https://hg.mozilla.org/build/mozharness/rev/4e0a080517ed

Comment on attachment 8567823 [details] [diff] [review] [01] Bug 1126695: Added emulator-l to build configs (v2) [checkin: comment 42] https://hg.mozilla.org/build/buildbot-configs/rev/89a7634d0a6f

Comment on attachment 8567824 [details] [diff] [review] [02] Bug 1126695: Run run JB,L opt on pushes; and JB,L debug periodically (v3) [checkin: comment 42] https://hg.mozilla.org/build/buildbot-configs/rev/fbc297beb01e

Comment on attachment 8567058 [details] [diff] [review] Bug 1126695: Added emulator-l to B2G bumper script (on mozharness) [checkin: comment 41] v2.2 as well: http://hg.mozilla.org/build/mozharness/rev/a28d8f302d52

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #41) > https://hg.mozilla.org/build/mozharness/rev/4e0a080517ed (In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #44) > http://hg.mozilla.org/build/mozharness/rev/a28d8f302d52 In production.

Darn it, I meant to put a leave-open on this until the buildbot parts were in production and then forgot to.

The jobs got enabled in production today and we ran into bug 1120680. Easy fix. https://hg.mozilla.org/mozilla-central/rev/600f44fd317c

At least on b2g37, these are hitting bustage. There's nobody around to fix it right now, so I've hidden them across all trees. https://treeherder.mozilla.org/logviewer.html#?job_id=52453&repo=mozilla-b2g37_v2_2

With bug 1137651 landed, we're hitting this bustage now: https://treeherder.mozilla.org/logviewer.html#?job_id=1419459&repo=b2g-inbound

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #50) > With bug 1137651 landed, we're hitting this bustage now: > https://treeherder.mozilla.org/logviewer.html#?job_id=1419459&repo=b2g- > inbound This should get fixed by bug 1137654. IIRC :mwu is at the MWC this week. I'll look for another reviewer.

Getting closer: https://treeherder.mozilla.org/logviewer.html#?job_id=1430153&repo=b2g-inbound

Let's go through the stack trace 09:03:16 INFO - Running command: ['./repo', 'forall', '-c', 'git show-ref -q --head HEAD || rm -rfv $PWD'] in /builds/slave/b2g_b2g-in_emu-l_dep-000000000/build 09:03:16 INFO - Copy/paste: ./repo forall -c "git show-ref -q --head HEAD || rm -rfv $PWD" This command fails. repo checks the heads of a git repository exists or otherwise deletes the local checkout (?) 09:03:16 INFO - Traceback (most recent call last): 09:03:16 INFO - File "/builds/slave/b2g_b2g-in_emu-l_dep-000000000/build/.repo/repo/main.py", line 506, in <module> 09:03:16 INFO - _Main(sys.argv[1:]) 09:03:16 INFO - File "/builds/slave/b2g_b2g-in_emu-l_dep-000000000/build/.repo/repo/main.py", line 482, in _Main 09:03:16 INFO - result = repo._Run(argv) or 0 09:03:16 INFO - File "/builds/slave/b2g_b2g-in_emu-l_dep-000000000/build/.repo/repo/main.py", line 161, in _Run 09:03:16 INFO - result = cmd.Execute(copts, cargs) 09:03:16 INFO - File "/builds/git-shared/repo/repo/subcmds/forall.py", line 200, in Execute 09:03:16 INFO - mirror = self.manifest.IsMirror 09:03:16 INFO - File "/builds/git-shared/repo/repo/manifest_xml.py", line 358, in IsMirror 09:03:16 INFO - return self.manifestProject.config.GetBoolean('repo.mirror') Here, repo tries to figure out if the manifest repository mirrors an existing git repository. 09:03:16 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 112, in GetBoolean 09:03:16 INFO - v = self.GetString(name) 09:03:16 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 129, in GetString 09:03:16 INFO - v = self._cache[_key(name)] 09:03:16 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 246, in _cache 09:03:16 INFO - self._cache_dict = self._Read() 09:03:16 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 253, in _Read 09:03:16 INFO - self._SaveJson(d) 09:03:16 INFO - File "/builds/git-shared/repo/repo/git_config.py", line 283, in _SaveJson 09:03:16 INFO - if os.path.exists(self.json): 09:03:16 INFO - AttributeError: 'GitConfig' object has no attribute 'json' 09:03:16 ERROR - Return code: 1 Repo tries to read the 'mirror' property from a cache file. As the cache doesn't exist, it tries to create the file, which fails. I ran the repo command locally and grepped for repo configs. The result is ./manifests.git/.repo_config.json ./repo/.git/.repo_config.json

Running ./repo forall -c "git show-ref --head HEAD" locally doesn't produce any errors. The only info I could find online is [1]. It talks about incorrect file permissions. Ryan, I think this is a problem on the build server. Could you check if the cache files are there and manifest directories are writable by the build job? [1] https://code.google.com/p/git-repo/issues/detail?id=190

(In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #54) > Ryan, I think this is a problem on the build server. Could you check if the > cache files are there and manifest directories are writable by the build job? I don't have access to the slaves, so no. Try asking in #releng?

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #55) > (In reply to Thomas Zimmermann [:tzimmermann] [:tdz] from comment #54) > > Ryan, I think this is a problem on the build server. Could you check if the > > cache files are there and manifest directories are writable by the build job? > > I don't have access to the slaves, so no. Try asking in #releng? Peter, could you take a look? See comment 53 and below for the context.

-> buildduty

Sorry, yesterday was a nightmare. I'll have a look today to see if it's something obvious. Standard proviso: I can't debug an actual build problem, but am happy to loan an instance to someone who can. (https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave)

(In reply to Chris Cooper [:coop] from comment #58) > Sorry, yesterday was a nightmare. I'll have a look today to see if it's > something obvious. Hi Chris, any news here?

I had trouble tracking down a slave that still had the build directory intact. The spot instances often get recycled before investigation is possible. I just found this Emulator-L build from today: https://treeherder.mozilla.org/#/jobs?repo=b2g-inbound&revision=ab7f61f08c42&exclusion_profile=false The build is still failing, but not on an Attribute Error. That particular check-in is a manifest bump, so maybe the AttributeError is fixed in one of the new revs included in the bump?

Emulator-L builds have a lot of logspam like the below as well: 16:47:00 INFO - prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-ar: /lib64/libz.so.1: no version information available (required by prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8/bin/arm-linux-androideabi-ar)

Ted, the makefile rules around tests names 'steeplechase' fail on emulator L in the build of comment 60. Can you help debugging? The full log is at [1]. The failed command is [2]. [1] https://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/b2g-inbound-emulator-l-debug/1425605419/b2g_b2g-inbound_emulator-l-debug_dep-bm72-build1-build2.txt.gz [2] https://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#531

I'm guessing you must have --disable-webrtc in the mozconfig right now? The only steeplechase tests we have are behind a MOZ_WEBRTC check: https://dxr.mozilla.org/mozilla-central/source/dom/media/moz.build#73 You could safely wrap the guts of the stage-steeplechase target in ifdef MOZ_WEBRTC to work around this: stage-steeplechase: make-stage-dir ifdef MOZ_WEBRTC <...> endif

Bug 1137515 literally landed and bounced on inbound today. So maybe we just wait for that to stick and see where that gets us. Though it'd probably be a decent safety move to do the ifdef wrap for any future issues that might arise.

Opt builds are green now that bug 1137515 has landed, so webrtc being disabled was indeed the issue. I filed bug 1142494 for only packaging steeplechase tests if webrtc is enabled.

Emulator L builds are green and running on all expected trees. I've unhidden them. I think we're done here! :)

Commits pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/27c1608e6a38de306176edcef0fc21189606dd30 Bug 1126695: Added android-{4-4,5-0}-armv7-api11 and emulator-l to Treeherder UI The entries for android-{4-4,5-0}-armv7-api11 were not added in bug 1112693, so we do that now. https://github.com/mozilla/treeherder/commit/056a128a53a992fafc6afd42740d2a90504ad813 Merge pull request #380 from tdz/bug-1126696 Bug 1126695: Added emulator-l to Treeherder UI