+++ This bug was initially created as a clone of Bug #688187 +++ This is to document the plan for WebRTC repo management and how we plan to handle patches for upstream and merging patches from upstream.

Comments copied from bug 688187: Here's a pretty good picture of how to import updates to an external vendor lib in mercurial. Looks fairly straightforward. It won't directly track renames, etc, but that's probably ok (that's hard to do across different SVC systems). http://www.selenic.com/pipermail/mercurial/2009-April/025304.html Randell Jesup [:jesup] 2011-09-23 12:44:25 PDT Ok, summarizing from #media: (And remembering O'Duin's exhortation to document all decisions in writing). This is the outcome of myself, rillian and dmose chatting. Tentative plan - 1) Start with webrtc in the tree on our branch repo, but built as a standalone lib for now. 2) To enable easy tracking of upstream, use a structure like this: a) Changes are checked into our trunk. If they're not moz-specific (make stuff, etc) we add [upstream-merge-needed] to whiteboard. b) upstream-merge-needed patches are merged to a separate head that has a clean copy of the library with the changes we want to go back to Google; whiteboard is changed to [upstream-pending] c) downstream pulls are done via a script; it puts them on a "vendor-branch" head which is then merged to the upstream-pending head. Conflicts are dealt with manually. [upstream-pending] patches now in the upstream should be marked [upstreamed]. Then the upstream-pending head is merged to trunk. Comments? It should be easy to get lists of pending upstream patches, etc. Biggest place it can fall down is patches in trunk not getting [upstream-merge-needed]; we could make a tool to catch those. Comment 9 Randell Jesup [:jesup] 2011-09-23 12:59:13 PDT Oh, yeah, we'll work out with Google if they want batches of patches for upstreaming or done one at a time as generated. For us, some level of batching for most patches makes sense. We might want to add a 4th upstream state: [upstream-waiting] after it's submitted and not yet in a downstream

Ralph Giles (:rillian) 2011-09-23 13:07:57 PDT Certainly we should update the bug here with links to google's rietveld instance when the upstream status for a patch changes. I'd suggest, therefore, that our webrtc patch bugs not be closed until they're accepted or rejected upstream.

So, mark as fixed when we check into trunk; but don't have QA/etc close them until upstream accepts them?

I think we shouldn't even mark them fixed until upstream has also reviewed on their side. But I don't know who QA/etc would be for our branch.

Hmmm. Not sure how leaving them 'open' would work with our process here in bugzilla. I can check with a few people. We generally close bugs when fixed in trunk; for example a bug found in Aurora is closed when the fix is in trunk; we use flags to mark the handling for aurora. (We *could* use flags here, but we wouldn't really want them global, so I think whiteboard tags makes more sense - effectively on-demand private flags.) I think Matt is our QA contact; chatted briefly at All-Hands. Per discussion in #media, we'll want to avoid perturbing their internal directory structure if we can - it makes merges in either direction a pain. For pieces we want to "break out" (AEC, ice, etc) we'll make the build system create a stand-along .a or .so to pull in elsewhere, I think.

I'm preparing the initial import onto alder (our project repo), setting up a 'vendor branch' per the discussion above, and building a tool for merging updates. Before pushing the import, I'm going to test the tool and sequence on a dummy repo using the real webrtc files. I plan to pull it into media/webrtc, and because it has a bunch of dependencies on living in a directory named "trunk" I'm planning on have all the code live in media/webrtc/trunk. Also, we'll have other webrtc code which can live in media/webrtc and subdirs of it. I plan to name the vendor tag "webrtc-import-last", the pending upstream tag "webrtc-pending". I also plan to push a tag at the import point: "webrtc-import" to make rollbacks and diffs easy. I plan to exclude all .svn/* files from import. Here's the .hg/hgrc you'll need: [paths] default = http://hg.mozilla.org/projects/alder/ default-push = ssh://hg.mozilla.org/projects/alder/ (Adding the rest of the team for heads-up)

(In reply to Randell Jesup [:jesup] from comment #6) > I plan to pull it into media/webrtc, and because it has a bunch of > dependencies on living in a directory named "trunk" I'm planning on have all What depends on this? > the code live in media/webrtc/trunk. Also, we'll have other webrtc code > which can live in media/webrtc and subdirs of it. Traditionally, our 1st party code has lived in content/media.

You can't even pull the code with sync if you use the --name option to change the directory name.

Ok, after talking with sfink and derf on #developers last night, and looking more at pages about vendor branches in Mercurial, I'm leaning towards either a true branch (though it clutters up the commits a little, it avoids mistakenly leaving the tip on one of my sidelines), or bookmarks, which are like branches but don't clutter metadata. If we use bookmarks, when we commit to the bookmark head I think we'll have to do a commit to the development head to make it the tip again, which is a pain and prone to race conditions and errors. I trialed bookmarks on a local dummy repo, and the sequence for merging updates from upstream works - the one caveat is that you must force it to create a new head for the bookmark before trying to merge to it. (I.e. commit a do-nothing commit to upstream-pending before merging to it). Because of the problems about tip handling (unless 2 branches for handling vendor tracking is ok), it might make sense to handle the vendor branch (webrtc-import-last) and upstream-pending on a separate repo. The branches might make more sense if they could be shared by *all* the external libraries we import (or at least new ones) - and I suspect they can be shared, but it needs thought. This will need a little more thought and reading. I hope I can have it ready by Monday, or ready for approval of the scheme from some the other stakeholders in tree management.

Ok - things aren't so simple: Talked with ted and kyle in #developers. hg sub-repos may make sense (to a private mirror or local hg repo that implements the scheme above). Note that per hg docs they recommend not having sub-repos under real ones, since the sub-repo can block attempts to push to the main repo. Also, it's even bigger than I remember (maybe they added more tests): 480K ./media/webrtc/trunk/third_party_mods 71036K ./media/webrtc/trunk/test 24964K ./media/webrtc/trunk/src 4580K ./media/webrtc/trunk/tools 408K ./media/webrtc/trunk/build 200K ./media/webrtc/trunk/peerconnection 4K ./media/webrtc/trunk/.svn 4828K ./media/webrtc/trunk/testing 43224K ./media/webrtc/trunk/third_party 149864K ./media/webrtc/trunk and that's NOT built. >1/2 of it is the test and testing dirs. We only have about 550MB of code in mozilla-central today... So, a little more thought and prep is going to be needed, even if we go with this overall idea. Factors to consider: a) ease of taking updates from upstream b) ease of collecting and submitting updates to upstream c) affect on mozilla tree and build management d) risks of fubars e) affect on development times for developers: tree pulls, build times, etc Or we deal with it as an pure external lib with no mods. This has a bunch of downsides from our point-of-view; I think we want it in our tree. But I'm open to argument.

(In reply to Randell Jesup [:jesup] from comment #11) > So, a little more thought and prep is going to be needed, even if we go with > this overall idea. I think we should just try something, and work from there. We're on a branch, so we can't really screw anyone else up yet. If it's untenable, we'll notice pretty quickly, and we can figure out what to do based on actual experience.

+1 for just trying something. When we land feature branches, do we import the whole development history, or just merge a patches with the working code? If the later we don't have to worry about the code size.

Could be either (import history or just code). After talking with people on #mercurial, I think I'm going to simplify it for now by not making an upstream-pending bookmark. We can add it later if we want. I'll have a bookmark that we can use to "trim" down the release for import. I'll probably import from the 'trim' bookmark, but I may not bother to trim yet. Later on we can hg remove any files we don't want in our m-c build repo. What I may do when we have patches to upstream is to hold them in a (revision-controlled, shared) queue (mq) in the merge repo. "hg init --mq" -- though for now, the merge repo can probably just be local, as there's minimal important info in it. So I'll start with it local. Or we could just leave them in the m-c (alder for now) repo, and use whiteboard to manage. So this simplifies things.

Updated update script - most of the "modify the repos" and "push to alder" commands are currently echoed for the user to run, until I'm comfortable its working well.

My webrtc_import repo is now hosted at hg.mozilla.org/users/rjesup_wgate.com/webrtc_import (The address should change when my checkin credentials are updated to my current email address) Sequence to build an import repo: hg init webrtc_import mkdir media/webrtc cd media/webrtc mkdir webrtc_update cd webrtc_update gclient config http://webrtc.googlecode.com/svn/trunk gclient sync --force cd trunk hg tag -l old-tip hg bookmark webrtc-import-last rm -rf trunk mv webrtc_update/trunk trunk mv webrtc_update/.g* . rmdir webrtc_update hg addremove --exclude "**.svn" --similarity 75 trunk hg add media/webrtc/.g* hg commit -m 'Webrtc import 2011-09-27 09:00 EDT' hg bookmark webrtc-pending hg bookmark webrtc-trim echo > media/webrtc/dummy foo hg add media/webrtc/dummy hg commit -m "Add dummy file to force bookmark to make a head" hg remove media/webrtc/dummy hg commit -m "Remove dummy file"

Import process here didn't work in the end - too much explosion of history in hg. We use a manual process with reapplying all patches (ad a single mass) on top of a pull of new code from upstream