some thoughts on what to look for:

• Measure weekly, but real test is on a 4 week window
• Frequency of xpcshell annotations are similar as before (i.e. we didn’t miss anything)
• Only exception is if we disable a few tests that are problematic but were previously spread out among many bugs
• Frequency of xpcshell tests disabled are similar as before
• Number of bugs fixed by developers (in a given week how many intermittent bugs are fixed by developers, does this change as a result of this)

the metric I see the look at is bugs fixed by developers. Here are some bugzilla queries:
long description with autoland reference

summary with test_

ideally we can create a query or tool which can give us a rate of fixed-with-commit xpcshell intermittent-failure bugs. Given lower frequency, I think monthly cadence will be where we can see a pattern. This should be split out into two data points:

• fixed-by-commit::manifest
• fixed-by-commit::test-or-product

another note here is to look at the last year of data. There are some periods of time (June, December, etc.) where PTO is heavy and other priorities might be higher than fixing intermittents.