https://mxr.mozilla.org/mozilla-central/source/testing/mozbase/generate_diff.py generates a diff wrt m-c from a tagged version of particular mozbase packages. However, there is nothing that ensures (or even makes likely) that the version of all mozbase packages on m-c is even internally compatible. generate_diff.py should check for this, using the dependency information exposed from setup_development.py. If necessary, functionality may be imported from versionbump.py to help

I've looked at this a bit and have come to the following opinions: - the version of setup_development.py on m-c will need updating. I would probably ticket this separately, but it is likely that fixing will involve fixing setup_development.py concurrently, so might as well keep it here - as to the above, much of the code from setup_development.py will have to be copy+pasted or setup_development.py reworked such that the information gathering parts of main() are exposed. I prefer the latter - since we require setuptools for mozbase, we will need to run this in a virtualenv; to this end, I think we should probably rework generate_diff.py to be a mach command. Nothing's ever easy. That said, I'm inclined to do it correctly vs slapping another unreliable layer on top of things. What I am aiming for, in abstract, is a tool (read: python script) that can divine especially dependency information from a set of packages/setup.pys. While right now I mostly carea about this problem, maybe 90+% of this problem is not mozbase-specific.

To give some background on why I care about this bug right now is that currently the procedure for mirroring mozbase to m-c is basically "write a bug and assign it to jhammel". There has been some complaints that other folks can't do it and it isn't documented. In fact, it isn't documented (well, I haven't documented it) because it is not robust. Nor if I do it am I guaranteed to screw things up. So, let's fix it.

I may also try to monkeypatch setuptools in sys.modules instead. While, in general, this won't be reliable, it will work with mozbase packages.

In the interest of time, and of not scope creeping too much, I think I will use this sort of thing to gather dependency info from generate_diff.py. It is not as robust as what setup_development.py does, in general, but this should work for all mozbase packages. I do think that in the longer term generate_diff.py should be converted to a mach command and also that Mozilla (and the world) wants for a tool to inspect python packages, but for now let's just brute force all the things.

It ain't elegant....the least elegant part being the huge overlap in intention between this and setup_development.py. But it does seem to work, at least if you have pkg_resources installed globally (seems to be the case in python 2.7). Again, I'd really like to make the plethora of knowledge in setup_development.py, versionbump.py, and generate_diff.py into one useful tool. But I'm kinda blanking at the moment and assuredly that's a larger effort.

Comment on attachment 697716 [details] [diff] [review] check consistency when generating the diff This all looks pretty reasonable to me.

It all seems bat-sh** crazy to me, and i wrote it! ;) That said, I don't think I'm doing anything particularly unnecessary or am forgetting necessary parts I wished to tackle. Pushed to inbound: https://hg.mozilla.org/integration/mozilla-inbound/rev/fe4e809058e0 I will update https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Mirroring with usage instructions. Together with setup_development.py and versionbump.py, generate_diff.py is probably the last main essential piece we need to keep version sanity. None of this software is fool-proof or wonderful, mind you, but at least it is something to build on.

Unfortunately, because this script writes in indelible ink, it is hard to test. More unfortunately, I left debugging code in. Let's kill it

pushed to m-c: https://hg.mozilla.org/mozilla-central/rev/5bb309998e70

documented: https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Mirroring Note that this is all pretty experimental but i do encourage everyone to use it

What was the intention with: # ensure that the versions to mirror are compatible with what is in m-c old_package_info = current_package_info.copy() (http://hg.mozilla.org/mozilla-central/file/23c23b472a4f/testing/mozbase/generate_diff.py#l331) We currently don't use old_package_info after assignment - wondering if I can just clean this up?

(In reply to Ed Morley (Mostly away until 21st October) [:edmorley UTC+1] from comment #12) > What was the intention with: > > # ensure that the versions to mirror are compatible with what is in > m-c > old_package_info = current_package_info.copy() > (http://hg.mozilla.org/mozilla-central/file/23c23b472a4f/testing/mozbase/ > generate_diff.py#l331) > > We currently don't use old_package_info after assignment - wondering if I > can just clean this up? Guessing that it was debugging code that I forgot to remove. Please feel free to remove.

(In reply to Jeff Hammel [:jhammel] from comment #13) > Guessing that it was debugging code that I forgot to remove. Please feel > free to remove. Ty; bug 929157 :-)