http://hg.mozilla.org/build/talos/file/195293d7404c/talos/test.py has a lot of test definition. However, these are currently used only as a basis for configuration. When it comes to actually running and reporting the test results, we pass around a test_config dictionary that is read (basically) from the configuration. However, it may be to our advantage to offload more of this to the test (and here I'm loosening the word a bit) "definition". This could make for more flexibility in how tests are done if we give more flexibility into allowing tests to do different things. Overall, I'm a bit skeptical of this as long-term architecture. Configuration should be configuration and logic should be logic. While this could (and IMHO should) be done in such a way as to separate the configuration definition layer from any sort of "logic". However, if it helps us solve problems today, it could be a good exercise in bootstrapping. In any case, it feels like passing around test_config in one place and using test (definition) classes in another place is a bit wonky. I realize this is a fairly vaguely written bug as I think we'll have to find a real problem in order to make this practicable, but noting for tracking.

As an example of something where this does make a practical difference, take bug 767226 . If we passed around full test objects vs a test config dictionary, we could do something like test.getResults() to use the special v8 filter and get AVERAGE vs values from the test definition as well Instead, we have to resort to the following magic: * we report the scores for each benchmark * we put special logic in filter.py that *assumes* that the order observed == the order in the manifest * we add another check in reporting for the v8 test to report the average All and all, this is fragile and will make the code kinda awful. I thought about fixing this first, but really that will delay landing. That said, we will have to do things like this again. We already do enough awful; let's not make it worse

adding :parkouss on here as this might be a useful enhancement to the talos code base.

I think this has been around for many years and is probably out of scope given the current talos.