The test resolver calls into this backend to find tests / manifests:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/gen_test_backend.py#44

The problem is that certain manifests are behind build conditions in the moz.build files. E.g, if I don't have a local buildconfig, then only a tiny subset of tests will be found. Even with a full build, there are manifests that are e.g platform specific.

We need a way to parse moz.build files and find all possible test manifests no matter what (for a few use cases now). This will likely involve parsing the moz.build the same way we do for the SPHINX_TREES variable:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/frontend/reader.py#909

There's a comment in there even suggesting we make the function more general purpose so it can be re-used with other variables.. so the suggestion is at least not too crazy.

The alternative is to be strict about not allowing test manifest definitions behind build configuration if statements (and instead do the skipping in manifestparser/mozinfo). I haven't looked into this, but I imagine it would actually be a lot harder than parsing the AST.

Hi ahal, is this something you're planning to work on, or are you looking for assistance here?

Was planning to tackle it at some indefinite point in the future.. though if you know how to solve it and are offering, I wouldn't turn down help ;).

The BuildReader's 'find_sphinx_variables' function is hardcoded to look for the
SPHINX_TREES and SPHINX_PYTHON_PACKAGE_DIRS variables. But it's otherwise
implemented to find any arbitrary variable (as long as they are a simple list
or dict).

This change simply moves the variable name(s) to the function call. The comment
about possibly wanted to use a higher level AST library to parse other kinds of
variables still applies, but for now this change is good enough to suit my
needs.

https://hg.mozilla.org/mozilla-central/rev/3cb0c1d7e716