One of the strengths of mozharness is anything can go into a config file. One of the weaknesses of mozharness is there's nothing to validate a config file as good, other than well-formedness (json or python). We should be able to define certain config variables as required, or as optional. Whether they're strings or integers or lists or tuples or dicts. If they're lists or tuples, and they're part of a "choice" optparse option, we should be able to define that enum of legal choices. Variables/options that are unknown can be marked as ignore, warning, or fatal depending on levels of strictness. If one optional variable is set, it might trigger the availability or required-ness of other variables. I'm not sure how best to do this, but it's definitely a want. We'd need to be able to define this per script or script family, and mark config files as belonging to those scripts or script families.

Bonus points for dtd/schema inheritance, so all MercurialScript or VirtualenvMixin based scripts can get those config file options for free. Extra credit for writing this in mozharness + mozharness-config style language, since I know one common complaint for DTDs is that you have to write a different parser and know a different syntax for DTDs.