Currently, we've the keyword "regression" to say that a bug is a regression.
But we've no way to know that a bug is not a regression.
So the idea is to have a new field "is_regression" with 4 possible values:

• "Yes": this is a regression
• "No": this isn't a regression
• "Unknown": the dev doesn't know if it's a regression but thought about it
• "---": the default value

Observations and Questions

• Looking at bug history was a way to see if a bug is a regression, but that was not a "at a glance" piece of information, so this is an improvement

• Adding a flag type for this is straightforward, but because of the way that Bugzilla displays flags, this would not be displayed alongside the regresses and regressed-by fields, so we need to update UI to make this more usable.

• Current bug handling documentation documentation changes would need to be amended for this

• Do we make this change along with the changes we're making to add regresses/regressed-by?

• We'd need to work with manager's regression triage to make sure they use the new workflow

  • But at least the Wednesday regression triage would be the right place to socialize the change

• Values for this field would be

  • +: yes is a regression, and regressed-by should be non-empty, and the committer needinfo'ed to fix
  • -: no, is not at regression, and regressed-by should be empty
  • ?: requesting confirmation that this is a regression, this would replace the regressionwindow-wanted field
  • ---: not a regression or considered a regression at this time

• We'd also need to map existing data

  • Look at bug histories to see when the regression keyword was added and removed to know when to mark the bug as -
  • Bugs for which the regression keyword was added and the regressor found would be marked as +
  • Bugs with the regression keyword and either no regressors listed or the regressionwindow-wanted is present, would be marked as ?

Before creating dependencies, I'd like some feedback on these questions and comments.

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #1)

• Do we make this change along with the changes we're making to add regresses/regressed-by?

I think this would be ideal, as they are related changes.

• We'd need to work with manager's regression triage to make sure they use the new workflow

  • But at least the Wednesday regression triage would be the right place to socialize the change

• Values for this field would be

  • +: yes is a regression, and regressed-by should be non-empty, and the committer needinfo'ed to fix
  • -: no, is not at regression, and regressed-by should be empty
  • ?: requesting confirmation that this is a regression, this would replace the regressionwindow-wanted field
  • ---: not a regression or considered a regression at this time

I wouldn't mix the 'Is Regression' and 'Regression Window Wanted' information. If a bug is a regression, it is a regression no matter if we know the regression window or not.
'Is Regression' == 'yes' would only mean that the bug is a regression. The regressed-by could be empty because there are many cases where getting a regression window is not easy / not worth it. The type of the bug should be 'defect'.
'Is Regression' == 'no' would mean that the defect is not a regression, and regressed-by must be empty.
'Is Regression' == 'unknown' would mean that the defect is aknowledged, somebody triaged it, but it isn't known whether it is a regression or not.
'Is Regression' == '---' is the default value, nobody took the time to check the defect to tell whether it is a regression or not.

• We'd also need to map existing data
  • Look at bug histories to see when the regression keyword was added and removed to know when to mark the bug as -
  • Bugs for which the regression keyword was added and the regressor found would be marked as +
  • Bugs with the regression keyword and either no regressors listed or the regressionwindow-wanted is present, would be marked as ?

I would only map existing data when we are absolutely sure, to avoid introducing noise.
So, I'd only set 'Is Regression' == 'yes' for bugs which have the 'regression' keyword.

Sound like a great summary.
We should do it but no need to block the regressed-by new field with that.

Okay, state of play on this is:

regression keyword goes away, replaced by the is-regression status flag with the values:

• yes - would only mean that the bug is a regression. The regressed-by could be empty because there are many cases where getting a regression window is not easy / not worth it. The type of the bug should be 'defect'
• no would mean that the defect is not a regression, and regressed-by must be empty
• unknown would mean that the defect is acknowledged, somebody triaged it, but it isn't known whether it is a regression or not
• --- is the default value, nobody took the time to check the defect to tell whether it is a regression or not

The regressionwindow-wanted ought to migrate to a flag as well:

• yes - yes, a regression window is needed, should be set once is regression is yes or if investigation is needed
• found - yes, a regression window has been found
• ---- default, nobody has investigated if a window is needed or if the bug is a regression

It's not super clear to me what the difference between regression:unknown and regression:--- is. There's a verified defect (otherwise the bug would be invalid), but we don't know if it's a regression. I would guess triagers aren't going to take the time to change from --- to unknown for each bug, if they mean basically the same thing.

I wonder if we can also solve the following type of problem with this flag:

• we have a real regression, but it's been shipping for multiple, multiple releases

At this point, the regression should probably just be treated as a regular bug, meaning we don't need to triage it as part of regression triage. In the past we've removed the regression keyword to get it off our queries, but Lizz made a good point that it skews reality for people trying to do research, etc. Instead we've agreed to just set a regression to fix-optional across the board so it disappears.

If we had a flag value to reflect whatever this state is, we could flip it to that and make the bug disappear from regression triage. But I have no useful suggestions on names.

What about?

• yes - would only mean that the bug is a regression. The regressed-by could be empty because there are many cases where getting a regression window is not easy / not worth it. The type of the bug should be 'defect'
• no would mean that the defect is not a regression, and regressed-by must be empty
• regression-window-needed - we believe this is a regression, but we want to know what regressed it. The regressed-by field must be empty (then we can get rid of the regressionwindow-wanted keyword)
• --- is the default value, nobody took the time to check the defect to tell whether it is a regression or not

and adding

• known - we have a real regression, but it's been shipping for multiple releases

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #4)

The regressionwindow-wanted ought to migrate to a flag as well:

Great idea making the regression window a field too!

(In reply to Mike Taylor [:miketaylr] from comment #5)

It's not super clear to me what the difference between regression:unknown and regression:--- is. There's a verified defect (otherwise the bug would be invalid), but we don't know if it's a regression. I would guess triagers aren't going to take the time to change from --- to unknown for each bug, if they mean basically the same thing.

The main goal here was to have a way to say "this is a defect, but it isn't easy to tell whether it is a regression". That is, from the current state of the bug, we can't say if it is a regression or not. "---" instead is the "empty" value, which means nobody thought about checking whether the defect is a regression or not. Maybe it is and the reporter even said it, but nobody set the field.
I see your point though, maybe people won't really use it. We could add it, and then remove it in the future if it proves to be unused.

I wonder if we can also solve the following type of problem with this flag:

• we have a real regression, but it's been shipping for multiple, multiple releases

At this point, the regression should probably just be treated as a regular bug, meaning we don't need to triage it as part of regression triage. In the past we've removed the regression keyword to get it off our queries, but Lizz made a good point that it skews reality for people trying to do research, etc. Instead we've agreed to just set a regression to fix-optional across the board so it disappears.

If we had a flag value to reflect whatever this state is, we could flip it to that and make the bug disappear from regression triage. But I have no useful suggestions on names.

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #6)

What about?

• yes - would only mean that the bug is a regression. The regressed-by could be empty because there are many cases where getting a regression window is not easy / not worth it. The type of the bug should be 'defect'
• no would mean that the defect is not a regression, and regressed-by must be empty
• regression-window-needed - we believe this is a regression, but we want to know what regressed it. The regressed-by field must be empty (then we can get rid of the regressionwindow-wanted keyword)
• --- is the default value, nobody took the time to check the defect to tell whether it is a regression or not

and adding

• known - we have a real regression, but it's been shipping for multiple releases

Wouldn't the field become too complex to use if we add too many alternative values, mixing the description of the bug with prioritization/release decisions?

(In reply to Marco Castelluccio [:marco] from comment #7)

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #4)

The regressionwindow-wanted ought to migrate to a flag as well:

Great idea making the regression window a field too!

We already have regression range as a flag but it doesn't have the "wanted" state:

Has Regression Range: no/yes/irrelevant

I assume we would be replacing the the keyword and flag.

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #6)

What about?

• yes - would only mean that the bug is a regression. The regressed-by could be empty because there are many cases where getting a regression window is not easy / not worth it. The type of the bug should be 'defect'
• no would mean that the defect is not a regression, and regressed-by must be empty
• regression-window-needed - we believe this is a regression, but we want to know what regressed it. The regressed-by field must be empty (then we can get rid of the regressionwindow-wanted keyword)
• --- is the default value, nobody took the time to check the defect to tell whether it is a regression or not

and adding

• known - we have a real regression, but it's been shipping for multiple releases

This makes sense to me. Interested if other people have feedback, I can appreciate Marco's comment about complexity.

Great summary. Thanks Emma!
I have some doubts about known.
I am concerned that the difference between yes and known is going to be too subtle. With just this value, I won't know what is the difference.

Maybe we should be more explicit with something like
old (existing for several years)

(In reply to Matthew N. [:MattN] (PM me if requests are blocking you) from comment #8)

We already have regression range as a flag but it doesn't have the "wanted" state:

Has Regression Range: no/yes/irrelevant

I assume we would be replacing the the keyword and flag.

I had forgotten about that one (again, too many fields) and there are 706 open bugs using it (https://mzl.la/2uOeDMD).

Options here would be:

1. Get rid of regressionwindow-wanted keyword, update Has Regression Range for bugs using the old keyword (https://mzl.la/2KbhIkv), don't have a regression-window-needed value for Is Regression
2. Get rid of regressionwindow-wanted keyword, get rid of Has Regression Range, use the values of the two to set Is Regresssion to regression-window-needed

And also to use old or be capricious and use ancient and undying for regressions for which we've known about for at least three releases.

I would like to wrap up the conversation on this by Tuesday the 9th of April, 2019 so I can start on the implementation.

Thank you all.

Point of information:

Using a regression for which we have known about for at least three releases to mean status_firefoxNN, status-firefoxNN-1, status-firefoxNN-2 is equal to affected or wontfix, we get ~45 bugs (https://mzl.la/2TW51d2).

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #11)

(In reply to Matthew N. [:MattN] (PM me if requests are blocking you) from comment #8)

We already have regression range as a flag but it doesn't have the "wanted" state:

Has Regression Range: no/yes/irrelevant

I assume we would be replacing the the keyword and flag.

I had forgotten about that one (again, too many fields) and there are 706 open bugs using it (https://mzl.la/2uOeDMD).

Options here would be:

1. Get rid of regressionwindow-wanted keyword, update Has Regression Range for bugs using the old keyword (https://mzl.la/2KbhIkv), don't have a regression-window-needed value for Is Regression
2. Get rid of regressionwindow-wanted keyword, get rid of Has Regression Range, use the values of the two to set Is Regresssion to regression-window-needed

And also to use old or be capricious and use ancient and undying for regressions for which we've known about for at least three releases.

I would like to wrap up the conversation on this by Tuesday the 9th of April, 2019 so I can start on the implementation.

Thank you all.

I would prefer 1, as it would reduce the number of different info we are mixing in 'Is Regression'. We are already mixing something related to the description of the bug ("is it a regression") with prioritization info ("is it a regression we care about?"), so I wouldn't also add the regression range info in there.
Also, if we go with 2, we'd need many combinations ("new regression, regression range known", "old regression, regression range known", "new regression, regression range wanted", "old regression, regression range wanted", and so on).

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #11)

I would like to wrap up the conversation on this by Tuesday the 9th of April, 2019 so I can start on the implementation.

Can we get a summary of the changes before implementation starts? It's not clear what path has been chosen for some of the cases and knowing the whole picture (with how the flags could interact/conflict) is useful.

(In reply to Matthew N. [:MattN] (away until Apr. 14) from comment #14)

Can we get a summary of the changes before implementation starts? It's not clear what path has been chosen for some of the cases and knowing the whole picture (with how the flags could interact/conflict) is useful.

Proposed Change

Remove regression keyword, and replace with is_regression status flag with values:

• yes - yes, bug is a regression
  • check the values of:
    • has_regression_range (see below) and
    • regressed_by
      to see if the cause of the regression has been identified
• no - no, bug is not a regression
• --- - default status, bug either has not been checked, or is not a regression

Remove regressionrange-wanted keyword, use the existing has_regression_range status flag.

That field has the values

• --- - default
• yes - yes, a regression range has been found
  • check the value of regressed_by
• no - no, a regression range needs to be found
• irrelevant - bug is a regression, but the changeset which introduced it does not need to be found

Migration

Bugs with regression keyword

• Bug has regressionrange-wanted keyword?
  • Set is_regression to yes
  • Set has_regression_range to no
  • Remove regression keyword
  • Remove regressionrange-wanted keyword
• Bug has has_regression_range set to yes
  • Set is_regression to yes
  • Bug has non-empty regressed_by field
    • Set has_regression_range to yes
  • Bug has empty regressed_by field
    • Bug has empty blocked_by field
      • Set has_regression_range to no
    • Bug has a single blocked_by
      • Set regressed_by to the value of blocked_by
    • Bug has multiple blocked_by
      • Human review required
• Bug has has_regression_range set to no?
  • Set is_regression to yes
• Bug has has_regression_range set to irrelevant?
  • Set is_regression to yes

Bugs without regression keyword

• Bug has regressionrange-wanted keyword?
  • Set is_regression to yes
  • Set has_regression_range to no
• Bug has has_regression_range set to yes
  • Set is_regression to yes
  • Bug has empty regressed_by field
    • Bug has empty blocked_by field
      • Human review required
    • Bug has a single blocked_by
      • Set regressed_by to the value of blocked_by
    • Bug has multiple blocked_by
      • Human review required

Next step will be notifying the relevant mailing lists about this change.

Bugs with regression and regressionwindow-wanted keywords

Bugs with regression keyword only

Bugs with regressionwindow-wanted only

Bugs without regression or regressionwindow-wanted keywords

Just a couple of observations on the migration:

• I wouldn't do anything yet with blocked_by/regressed_by, as we can't be sure that blocked_by was used to note a regression. I think it's better to have some missing information than wrong information, or we risk tools making wrong assumptions. We can work on migrating some old blocked_by to regressed_by separately, if we manage to find rules that are 100% valid.

• If 'has_regression_range' is 'yes' but 'blocked_by' and 'regressed_by' are empty, I wouldn't change 'has_regression_range' to 'no'. I think human review is required here as I'm afraid of overwriting human changes, so I would avoid this but add a autonag check (https://github.com/mozilla/relman-auto-nag/issues/546).

• Less important: some people use "regressionrange-wanted" even when it is not clear yet if the bug is a regression, just as a way of asking the reporter to check if it is a regression. So, we can't be 100% sure that 'regressionwindow-wanted' implies 'is regression'. This is less important though, as in my experience this case is pretty rare.

I have read the comments in the past few days plus Emma’s doc, but it seems the UX complexity and data inconsistency/redundancy won’t be solved with the current proposal. Given that we have added Regressed by field, the situation will rather be worse. As proposed earlier, having 1 flag should be enough.

Proposed change

Is Regression flag

• ---: default status
• range-wanted: replaces regressionwindow-wanted keyword
• range-irrelevant
• yes: replaces regression keyword
• no: we don’t have this now

(Don’t add Has Regression Range flag)

Automation

• When a bug is added to Regressed by field, change Is regression to yes
• When a bug is removed from Regressed by field, change Is regression to no unless it’s range-irrelevant

Or, this might sound more natural and less redundant:

⚠️ Updated the following comment as I’ve remembered that we already have Has Regression Range flag

Proposed change

Extend Has Regression Range flag and rename it to Regression Range

• ---: default status
• wanted (currently no): replaces regressionwindow-wanted keyword
• identified (currently yes): replaces regression keyword
• irrelevant
• not regression (new value)

(Having identified or irrelevant means the bug is a regression, and we have Regressed by field as well, so don’t add Is Regression flag)

Automation

• When a bug is added to Regressed by field, change Regression Range to identified
• When a bug is removed from Regressed by field, change Regression Range to not regression unless it’s irrelevant

(As I mentioned in Emma’s proposal doc) is_regression can be added as a bug’s read-only property, which will be true when Regression Range is identified or irrelevant. The will give important info on the UI or via the API while mitigating the inconsistency due to manual edits. We have already done the same thing for the is_open (Resolution is ---) and is_confirmed (Status is not UNCONFIRMED) properties of a bug.

Have to be manually updated:

• Regression Range: wanted, irrelevant, not regression

Can be automatically determined:

• Regression Range: identified (based on Regressed by field)
• Is Regression: true/false (based on all the above factors)

The proposed change in comment 15 made sense to me, I find boolean(ish) flags easier to understand than the new names suggested in comment 19.

The name doesn’t matter; if yes is better than identified, we can use it. What I want to say here is:

• Having 2 flags is a source of inconsistency as shown in the Legal Values chart in Emma’s proposal doc
• Given that we already have Regressed by field, the inconsistency will be worse
• The issue can be mitigated by
  • Putting all values in 1 flag
  • Letting Bugzilla set values that can be automatically calculated

I know the regression info has been very inconsistent so I want to avoid the further chaos.

I agree that one field is better as long as it's not confusing to use.

Boolean-ish values make sense for some of the values with has_regression_range as the name of the field.

• has_regression_range is --- is default
• has_regression_range is no means it's a regression and don't know the commit that caused it
• has_regression_range is yes means it's a regression and we know the commit that caused it
• has_regression_range is not needed means it's a regression and we don't care what caused it
• has_regression_range is n/a means it's not a regression

I'm suggesting not needed instead of irrelevant.

I think the rule for the generated is_regression field should be:

• if has_regression_range is no, yes, or not needed then the generated field is_regression should be true.
• if has_regression_range is ---, or n/a then the generated field should be false.

We should also be careful about reclassifying a bug as not a regression if we unset the regressed_by field. That can mean it's still a regression, but we were incorrect as to the cause and still don't know it.

Question

I went back and read Mike's comment where we started the discussion as to how do we represent bugs which we know are regressions and don't need to identify the cause, and I'd like to know how many bugs that is?

If we allow has_regression_range = no to mean is_regression = true, then that takes care of that case, and simplifies the field. In the case of a long-standing regression which we want to fix, but don't need to know the cause, we can note in the bug that we don't need to find the cause.

:marco, :jcristau and :mtaylor does the above address your concerns?

My feeling is that two fields here would make things more clear cut and less confusing, but I defer to your judgement.

To give you an example, in the two fields case, if a bug has 'is regression' set to 'no', I can be 100% confident that the person who sets the field actually means to say that the bug is not a regression.
With the single field, if a bug has 'has_regression_range' set to 'no' or 'not needed', I can't be so sure that the person actually meant that it is a regression: maybe it is not a regression, and so clearly there is no regression range (or a regression range is not needed).

Basically with a single field with these possible values, people will need training to get it right (and maybe they will forget about the nuances). With two fields things would be so obvious that people can't really get it wrong.

(In reply to Emma Humphries, Bugmaster ☕️🎸🧞‍♀️✨ (she/her) [:emceeaich] (UTC-8) needinfo? me from comment #23)

I agree that one field is better as long as it's not confusing to use.

Boolean-ish values make sense for some of the values with has_regression_range as the name of the field.

• has_regression_range is --- is default
• has_regression_range is no means it's a regression and don't know the commit that caused it
• has_regression_range is yes means it's a regression and we know the commit that caused it
• has_regression_range is not needed means it's a regression and we don't care what caused it
• has_regression_range is n/a means it's not a regression

I'm suggesting not needed instead of irrelevant.

I think the rule for the generated is_regression field should be:

• if has_regression_range is no, yes, or not needed then the generated field is_regression should be true.
• if has_regression_range is ---, or n/a then the generated field should be false.

In the '---' case, I would make 'is_regression' be '---'. '---' means unknown, so if 'has_regression_range' was not filled, 'is_regression' should be considered as not filled too.

We should also be careful about reclassifying a bug as not a regression if we unset the regressed_by field. That can mean it's still a regression, but we were incorrect as to the cause and still don't know it.

Agreed. If regressed_by is unset we could move 'has_regression_range' from yes to no.

I've had another idea if we want to be able to have a single field but eliminate the confusion.

Field 'Is Regression', with possible values:

• Yes, range known
• Yes, range needed
• Yes, range not needed
• No
• '---'

(These are the same values proposed in comment 23, just with a different field name and options wording to make them clear-cut).

Kudos for the simple solution!

That’s almost the same as my Comment 18. The problem is, values cannot have a human-readable label at this time. So yes-range-known is good but Yes, range known may not work. I’ll figure out if it can be solved.

We can work around that. Thanks to everyone for continuing to work the problem.

I think Marco's Comment #26 looks pretty great (which seems like a simplification of Emma's ideas in Comment #23).

(In reply to Marco Castelluccio [:marco] from comment #17)

• Less important: some people use "regressionrange-wanted" even when it is not clear yet if the bug is a regression, just as a way of asking the reporter to check if it is a regression. So, we can't be 100% sure that 'regressionwindow-wanted' implies 'is regression'. This is less important though, as in my experience this case is pretty rare.

In my experience this isn't very rare. The data potentially supports this (unless some people assume regressionwindow-wanted implies regression): there are currently 337 bugs with regressionwindow-wanted and 120 of them (35.6%) don't have the regression keyword: https://mzl.la/2VNNsxB

Often someone reports a bug and doesn't indicate if it's a regression or not and it may not be trivial for me to check for myself so I add the regressionwindow-wanted keyword and ask the reporter. It's very common for reporters to not reply to questions so having a way to indicate regression ranges are wanted (from QA or community members, like Alice White) is useful, otherwise I would just have to leave the field set to '---' and the bug will go stale if it doesn't sound urgent enough. In the past QA would look for bugs with regressionwindow-wanted (later only with the qaurgent keyword) but I think QA no longer queries for bugs requesting regression ranges and that hinders triage (though this should be revisited IMO).

With the proposal from comment 26, there isn't a way to distinguish "defects that may or may not be a regression" from "defects that haven't yet been triaged" so people who want to help find regression ranges don't have a way to query for the bugs.

Maybe we can add "Unsure, range needed" to the list of values? Or how do we want to signal that bug triage is stalled until we get a regression window?

Adding a field such as unsure, range needed or yes, range needed can allow bugs to get into an in-actionable state. We'd need an AutoNag rule for it. That said, I'll accept it.

I'm fine with that.

This has never been added to production. WONTFIX?

Will discuss this on Monday's BMO rountable.

I think this field would still be pretty useful in the long term.

agreed