In bug 1253110 we'll be producing archives with binaries files. Instead of hard-coding a list of binaries (like we do with artifact builds today), let's derive the list of binaries from moz.build. The first step is to produce a machine readable file containing binaries metadata.

Review commit: https://reviewboard.mozilla.org/r/37993/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/37993/

Now when we print instances in a debugger, we'll see something like: <StaticLibrary: other-licenses/snappy/libother-licenses_snappy.a> <StaticLibrary: toolkit/library/StaticXULComponentsEnd/libStaticXULComponentsEnd.a> <SharedLibrary: toolkit/library/XUL> <StaticLibrary: toolkit/library/libxul_s.a> <HostProgram: config/nsinstall_real> <Program: memory/replace/logalloc/replay/logalloc-replay> <SimpleProgram: mfbt/tests/TestArrayUtils> ... instead of the the class name and memory address. Review commit: https://reviewboard.mozilla.org/r/37995/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/37995/

This will make it easier for binaries only archive generation to consult the list of binaries that are relevant to packaging. This does overlap somewhat with compile databases. Perhaps someday the logic could converge. While I was here, I cleaned up writing of the all-tests.json file to avoid an intermediate string variable. Review commit: https://reviewboard.mozilla.org/r/37997/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/37997/

It just occurred to me that we could potentially use this code for updating the dist/bin install manifest. We could also potentially write out make rules to create binaries in their final location under dist/bin. That is scope bloat, however.

Comment on attachment 8726448 [details] MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37997/diff/1-2/

Comment on attachment 8726446 [details] MozReview Request: Bug 1253436 - Remove unused imports; r=glandium https://reviewboard.mozilla.org/r/37993/#review35273

Comment on attachment 8726447 [details] MozReview Request: Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium https://reviewboard.mozilla.org/r/37995/#review35275

Comment on attachment 8726448 [details] MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium https://reviewboard.mozilla.org/r/37997/#review35277 ::: python/mozbuild/mozbuild/backend/common.py:204 (Diff revision 2) > + def to_dict(self): > + SHARED_ATTRS = ('basename', 'lib_name', 'import_name', 'soname', > + 'install_target', 'relobjdir') > + shared = [] > + for l in self.shared_libraries: > + shared.append({k: getattr(l, k) for k in SHARED_ATTRS}) > + > + PROGRAM_ATTRS = ('program', 'KIND', 'install_target', 'relobjdir') > + programs = [] > + for p in self.programs: > + programs.append({k.lower(): getattr(p, k) for k in PROGRAM_ATTRS}) > + > + return { > + 'shared_libraries': shared, > + 'programs': programs, > + } I'd rather use a JSONEncoder subclass, that could be repurposed later.

I'd love to figure out a more general solution here to capture the parsed moz.build data for use in scripts/mach commands/etc instead of continuing to accumulate one-off json files. That's not to say we shouldn't land this patch, since it's useful, but it would be great to figure out a better long-term plan.

Comment on attachment 8726446 [details] MozReview Request: Bug 1253436 - Remove unused imports; r=glandium Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37993/diff/1-2/

Comment on attachment 8726447 [details] MozReview Request: Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37995/diff/1-2/

Comment on attachment 8726448 [details] MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37997/diff/2-3/

Comment on attachment 8726448 [details] MozReview Request: Bug 1253436 - Write out a machine readable file with binaries metadata; r?glandium https://reviewboard.mozilla.org/r/37997/#review36461 > In the spirit of perfect is the enemy of done, I think this should be deferred to a follow-up bug. Please file that followup bug, otherwise, it's never going to happen. ::: python/mozbuild/mozbuild/frontend/data.py:379 (Diff revision 3) > + def to_dict(self): > + return {k: getattr(self, k) for k in self.DICT_ATTRS} You might as well move this method to the base class. It will fail on all classes with no DICT_ATTRS, which is fine.

https://hg.mozilla.org/integration/mozilla-inbound/rev/c4a868c76959d3a2e13af6ca5adb0d629dcb2b34 Bug 1253436 - Remove unused imports; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/84849ad026c9ba1bbf71c93172b0a03440e51bec Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/4167dfdf10457360c9c94ee6e55b03ef1b92c16d Bug 1253436 - Write out a machine readable file with binaries metadata; r=glandium

Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/04a9056213bd for https://treeherder.mozilla.org/logviewer.html#?job_id=23744325&repo=mozilla-inbound.

https://hg.mozilla.org/integration/mozilla-inbound/rev/f1b77ed0c0c4a5d40d4ba854db9ae506d906b070 Bug 1253436 - Remove unused imports; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/e1533121f2e3adbe89e4a799c714d065a44ee5c3 Bug 1253436 - Add __repr__ to BaseLibrary and BaseProgram; r=glandium https://hg.mozilla.org/integration/mozilla-inbound/rev/bd4a4166af52256da8e99d90668bb40a82801694 Bug 1253436 - Write out a machine readable file with binaries metadata; r=glandium

Fixed the case of 'kind' vs 'KIND' to unbust it.

https://hg.mozilla.org/mozilla-central/rev/f1b77ed0c0c4 https://hg.mozilla.org/mozilla-central/rev/e1533121f2e3 https://hg.mozilla.org/mozilla-central/rev/bd4a4166af52