I'd argue that bug comments should be submitted (weekly or daily, not too particular here) in resolved bugs even if we don't hit WEEKLY_THRESHOLD (but at least once in the time range).

This would be easy to do - I just need a set of specific requirements. eg which repos (or all), which resolved states (all, or just fixed?) etc.

Completely forgot about this one. I think a start would be to drop WEEKLY_THRESHOLD to 1 for all resolved bugs on all repos. If that gets too spammy we can dial it back or add different thresholds for fixed vs incomplete vs others. But I think in pretty much all cases, we should make a comment for any failures in any resolved bugs, and then people can verify that the failures would warrant reopening the bug. Ccing other sheriffs for their input.

yeah i agree with wes, we should drop the WEEKLY_THRESHOLD to 1 - if this is too much/less etc we can adjust this anyway i guess

I don't suppose you could explain the rational for this change, so I can make sure I follow? :-) At the moment I see the possible scenario, which I guess this bug is meant to help: 1) A new intermittent failure occurs, and a bug is filed 2) That intermittent failure occurs < 5 times a week (the current threshold), so a bug comment is never left 3) After X months of inactivity the bug now longer counts in the open_recent Treeherder bug suggestions bucket, and instead falls in the all_others bucket. 4) If that intermittent failure occurs again, then in the Treeherder UI, one of the following happens: (a) if there are other bugs in open_recent (probably unlikely), then the all_others bucket is collapsed by default, and the user has to press "show more" before selecting (which is annoying) (b) if there are no other open_recent bugs, then the all_others bucket is expanded by default, so no more annoying to use than if the bug were classed as open_recent 5) Independently of 3-4, the owners of the component may also close the bug as WORKSFORME or INCOMPLETE Which issue is this trying to solve in the above? I'd posit that opening the bug isn't actually helpful in many cases. The activity is so low that no one is going to fix it anyway - and in the New World (where the bug number isn't the canonical identifier for each intermittent failure), bugs are only going to be filed for the most actionable subset of them anyway.

It's always a question to find the right balance between has the patch fixed the problem, or not. It's not always easy to see immediately especially for low intermittent failure rates of the test anyway. But not sure how often this would occur. At least it would be helpful to let the bug assignee know about still happening failures in that area. I had a situation today with bug 1364245 comment 20. I already requested uplift for a feature addition in Marionette the fix for this bug is necessary. Looking at OF I now see still happening failures after my patch landed. So at first sight I was a bit worried. Talking to Carsten and Wes on IRC we figured out that all those failures are not really related because they are happing on try, or other integration branches the Sheriffs do not maintain. From my perspective it would be great to get earlier feedback (maybe the same day) if failures are still happening on our main integration branches. Not sure if there is a way to check the landing status, so that no false positives are getting send out because the necessary merge didn't happen yet.

Rationale is mostly for the "did the patch fix the failure" case. And to answer comment 1, I would go with mozilla-inbound/central/autoland as the repos we care about (not every fix gets uplifted so release branches aren't important, try/project branches could be based on months-old code, so also not critical). States could probably be only FIXED if we're only caring about whether a patch resolved things. And maybe only for failure instances where <failure classification time> is more than 24 hours after the bug's last_resolved (I think that's the field name) timestamp, to make sure the patch has had ample time to get merged around. Agreed that this all stops mattering when we enter the glorious bugless future, but we haven't reached nirvana yet, years later, and this might help in the meantime. :)

Alternatively, maybe Treeherder-via-Orangefactor could expose the bug's resolved timestamp in the classification panels (in the tooltip?), and the human doing the classification can make the call to comment in the bug?

(In reply to Wes Kocher (:KWierso) from comment #7) > could expose the bug's resolved timestamp I was going to say I don't think Bugzilla exposes the resolved date, only the modified date, however there's actually `cf_last_resolved`, which is handy: https://bugzilla.mozilla.org/rest/bug/1179821

Weekly threshold was dropped to 1. Good enough for me.