Code coverage is collected when running tests in a test suite. In an effort to run these tests quickly, automation splits a suite into chunks and runs them in parallel on a number of machines. This means code coverage profiles must be collected and assembled to create a "complete" coverage profile. This problem extends beyond any one suite: We have multiple suites with overlapping coverage profiles. We have multiple platforms with overlapping profiles. Plus, each suite can cover lines in multiple languages. A "truly complete" profile is one that includes coverage from all these combinations. The assembly of partial profiles to create more-complete profiles has been built. The remaining part is knowing when a profile is "complete". Here are some ways we can be missing profiles: * If chunks 1, 2 an 3 are completed for a suite, how do we know there is no chunk 4? * If a test is turned off (skipped) how does that affect coverage? * If a test is failing, how does that affect coverage? * In he case of gcno files, how do we know a build did not generate an incomplete subset, and mistakenly reports less lines? * If a suite is specifically not run, how do report the incomplete coverage? * If a test/chunk/suite will not complete, can we extract value from the few profiles we did get? 1) What other ways can we be missing profiles? 2) What do we need to know to solve these problems? What is the schema? 3) How do we collect the information required to fill the schema? 4) When we are missing profiles, what solutions are there, and where are they implemented? 5) Can we mitigate missing coverage?

Some comments on what may be required for obtaining the metadata needed: * If chunks 1, 2 an 3 are completed for a suite, how do we know there is no chunk 4? We will need to use the task-graph.json file from the Gecko decision task which has metadata about the entire task that will be run. It also has information about the total number of chunks that are going to be run. * If a test is failing, how does that affect coverage? What we need here will depend upon whether or not we are collecting code coverage after a failure. If we do, it may or may not decrease coverage (or maybe even increase if a failure somehow causes more code to be run). Otherwise, it will always decrease the coverage percentage. * If a suite is specifically not run, how do report the incomplete coverage? Using 'test-sets.yml', discussed in bug 1337243. * If a test/chunk/suite will not complete, can we extract value from the few profiles we did get? For jsdcov running the xpcshell test suite, we do not collect coverage for failing tests, but for mochitest-* we still collect it.