Open Bug 1460061 Opened 7 years ago Updated 2 years ago

Determine if line hit counts should be considered when diffing baseline coverage from test coverage.

Categories

(Testing :: Code Coverage, enhancement)

enhancement

Tracking

(Not tracked)

People

(Reporter: sparky, Unassigned)

References

(Blocks 1 open bug)

Details

This is a follow-up bug to determine if hit counts should be considered when diffing the test coverage with the baseline coverage. Some analysis needs to be done to determine: (1) How large is the gain in coverage when we consider hit counts? (2) Is this gain due to variability, if so, how much? (i) Determine amount of variability in baseline. (ii) Determine if there exists gains which are greater than the variability. If there are gains in the coverage that are significant and cannot be due to variability this means that there must be test coverage that was obtained for lines where hit counts were compared. So, the next question needed to be answered is: (3) If there is a test coverage gain, is this coverage significant?
Blocks: 1301170
Depends on: 1459926
Thinking about this more, I have a proposition. Instead of trying to figure out if we need them or if we don't, we could give a score to each line that could tell us how intermittent it is. In essence, I propose that we use something like: | confidence_score = ( (<test_line_count> - <baseline_line_count>) / <baseline_line_count>) | This formula is a common way of describing 'task' related changes. We can add some heuristic rules to gear it to our needs, like if <baseline_line_count> == 0, then set score = None (or something else), to show that the line is unique for the test. We don't have to care about lines unique the the baseline, so if <test_line_count> == 0, we simply ignore the line. Otherwise, the score would be a percentage of how far away we are from the baseline_line_count. If it is 0, then the line has no differences between the test and baseline. As it approaches -infinity, the test_line_count decreases which means that the baseline hit the line more. And as it approaches +infinity, the test_line_count increases which means that the baseline hit the line less often then the test. Now, if we want to find all lines which are hit at least 25% more times in the test than the baseline, we can use the given proportion to find them, i.e. score >= 0.25. And all lines that are unique to the test and those that are hit twice as often would result in searching for lines with a score == None or score >= 1. There are many benefits to using this, from what I can tell, such as being able to choose how we want to look at the data well after baseline corrections are done.
Blocks: 1471577
No longer blocks: 1301170
No longer depends on: 1459926
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.