There is a large discrepancy between people who report updates disabled via settings.update.enabled vs. UPDATE_CHECK_CODE_NOTIFY. In an attempt at changing the way we detect users who have updates disabled in the Spark script for our update orphaning dashboard, I found that roughly twice as many people report having updates disabled (~40,000 users) via settings.update.enabled vs. UPDATE_CHECK_CODE_NOTIFY codes 12 and 13 (~20,000 users). A secondary question is why about 100 users report code 12 for UPDATE_CHECK_CODE_NOTIFY but don't report settings.update.enabled = false. Robert, as the expert here, is this easy for you to explain away?

What is the total number in the *NOTIFY data set? Not sure if this applies but are you taking into account how telemetry will provide a count and report numbers greater than 1?

Here is the run from this morning: http://nbviewer.jupyter.org/urls/s3-us-west-2.amazonaws.com/telemetry-public-analysis-2/update-orphaning-weekly-analysis/data/Update%20orphaning%20analysis%20using%20longitudinal%20dataset.ipynb The 1% longitudinal dataset contained data for 514,873 unique users. 70,362 reported being out of date. 20,047 reported having updates disabled via settings.update.enabled. Of the remaining 50,315 orphaned users, 18 users report code 12 for UPDATE_CHECK_CODE_NOTIFY. If I ignore settings.update.enabled and only look at the codes for UPDATE_CHECK_CODE_NOTIFY for the 70,362 orphaned users, only 5,566 report code 12 and 0 report code 13. I don't think reporting values greater than 1 applies here, since we look at individual users over time and any value other than 0 will be a match for any particular code. The only reason why we wouldn't report a user as code 12 is if the same users also reported any code between 1-11. I didn't think this was possible.

The 18 users could have changed the pref. Do you agree with ignoring the discrepancy of 18 reports? Not surprised that there aren't any 13's in the report. Those would be administrative installs where the administrator took the time to lock the update pref to disabled. What are the UPDATE_CHECK_CODE_NOTIFY values for the people that report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY?

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #3) > The 18 users could have changed the pref. Do you agree with ignoring the > discrepancy of 18 reports? Yes, I could agree with that. > What are the UPDATE_CHECK_CODE_NOTIFY values for the people that report > UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY? 5,758 of the people who are out of date report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY. This is an array with either 0 or 1 values as far as I can tell. I haven't dug into this, I just filtered based on users who report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY at all. These users report the following UPDATE_CHECK_CODE_NOTIFY values: {"1":14,"3":5,"6":1,"8":5,"9":124,"10":3,"12":5566,"14":2,"22":5,"23":27,"29":6} In other words, all users who report code 12 also report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY.

In comment #0 you wrote there are around 40,000 reports of UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY. Does your comment #4 include all of those? I'm confused because I asked for the UPDATE_CHECK_CODE_NOTIFY values for these users that report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY and the numbers in comment #4 don't add up anywhere near 40,000. Also noticed that you are saying in comment #4 there are 5,758 users that report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY which is significantly less than the 40,000. Perhaps a value of 0 for UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY in telemetry means it is enabled. Whatever the case, if the numbers in comment #4 is for the UPDATE_CHECK_CODE_NOTIFY values for these users that report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY then the discrepancy is no where near what was stated in comment #0... right?

Comment 0 mentioned numbers that I previously found and wrote down, using an old longitudinal dataset and this triggered the filing of this bug (granted, with some delay). Starting with comment 2, I've been using the longitudinal dataset that was generated on 7/4/2016. As you can see, the situation is even worse with 20,047 users reporting updates disabled via settings.update.enabled vs. only 5,566 via UPDATE_CHECK_CODE_NOTIFY. You are correct however that users reporting UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY don't add up to 20,047 users. There is clearly another discrepancy between users reporting settings.update.enabled = false and users reporting UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY. Sorry about the confusion.

At this point there are "5,758 of the people who are out of date report UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY" per comment #4 and the UPDATE_CHECK_CODE_NOTIFY values for these reports in comment #4 show that there isn't a large discrepancy. Do you agree with this assessment?

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #7) > At this point there are "5,758 of the people who are out of date report > UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY" per comment #4 and the > UPDATE_CHECK_CODE_NOTIFY values for these reports in comment #4 show that > there isn't a large discrepancy. > > Do you agree with this assessment? Yes. I'm assuming there are followup questions or analysis to perform to figure out why settings.update.enabled shows significantly more people having updates disabled than either via UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY or UPDATE_CHECK_CODE_NOTIFY?

How did you get 5,758 people reporting UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY (e.g. app.update.enabled = false) in comment #4 vs. getting 20,047 reporting UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY (e.g. app.update.enabled = false) in comment #2? These numbers should be the same... right?

Oh, you are using a non app update pref telemetry vs. the UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY. I don't know enough about the internals of telemetry to answer why those two values for the same pref differ but that would definitely be a good place for you to start to investigate why they are different. I suspect it could be due to the way they are triggered and possibly only checking for the existence of UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY vs the value... possibly other not well understood things about telemetry as well.

BTW: the UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY value is recorded at the time the update check is performed and should very closely match up with the value for the check as was shown in comment #4

You might want to check what platforms those systems are. Could be distros with app update disabled reporting to telemetry or something else like that.

As I've felt all along I suspect that in the end this issue will end up being an exercise in understanding telemetry along with the clients vs. a problem that is actionable with app update.

I just compared the telemetry for settings.update.enabled and update_not_pref_update_enabled_notify. Including both the update_not_pref_update_enabled_notify and update_ping_count_notify in the result set I added the following in a new cell below the update_disabled_mapper in the existing notepad def update_ping_update_disabled_mapper(d): status, ping = d if ping is None or ping.update_ping_count_notify is None: return ("none-ping", ping) index = 0 for ping_count in ping.update_ping_count_notify: if ping_count > 0: if ping.update_not_pref_update_enabled_notify is not None and ping.update_not_pref_update_enabled_notify[index] > 0: return ("update-disabled", ping) return ("update-enabled", ping) index += 1 return ("none-ping", ping) update_ping_update_disabled_statuses = out_of_date_statuses.map(update_ping_update_disabled_mapper) update_ping_update_disabled_results = update_ping_update_disabled_statuses.countByKey() update_ping_update_disabled_json_results = json.dumps(update_ping_update_disabled_results, ensure_ascii=False) update_ping_update_disabled_json_results Here is a comparison of the results From the existing update_disabled_mapper '{"none-update-enabled":40,"update-enabled":131569,"update-disabled":27441}' From the new update_ping_update_disabled_mapper '{"update-enabled":127415,"none-ping":4400,"update-disabled":27235}' I also checked for clients that never send an update ping which comes out to 4210 So, the difference is | settings | app update | Diff enabled | 131569 | 127415 | 4154 disabled | 27441 | 27235 | 206 none | 40 | 4400 | -4360 (150 when no update pings is accounted for) The disabled value is the value of update_not_pref_update_enabled_notify for the index of the first update ping and no value or a value of 0 is the disabled value. Note: the value of update_not_pref_update_enabled_notify is recorded at the time of the update check so it should be more accurate when evaluating the UPDATE_CHECK_CODE_NOTIFY telemetry value. For example, if an update is downloading, has been downloaded, is staged, etc. and the pref is changed one of those codes will be reported to UPDATE_CHECK_CODE_NOTIFY and not CHK_PREF_DISABLED.

I think with the above this can be closed.

Some additional info. This only includes the out of date data. In summary: Match : 97.14% Missing update data : 2.76% Missing settings data : 0.03% Possibly due to timing : 0.07% So, the match between settings.update.enabled and update_not_pref_update_enabled_notify is extremely close except for the case where there is no ping data and even then the percentage of no ping data is a small percentage overall. I suspect that the missing update data are possibly 3rd party builds (e.g. tenfourfox, etc.) but I haven't investigated yet. Also, trying to compare the value of UPDATE_CHECK_CODE_NOTIFY to settings.update.enabled is like comparing apples and oranges. UPDATE_CHECK_CODE_NOTIFY is an enumerated value and settings.update.enabled is a boolean, UPDATE_CHECK_CODE_NOTIFY and UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY are collected at the same time and settings.update.enabled is collected at a different time, UPDATE_CHECK_CODE_NOTIFY reports several other conditions before it reports codes 12 or 13 though it would be reasonable if UPDATE_CHECK_CODE_NOTIFY was a bitmask but alas' it is not which is why there is the UPDATE_NOT_PREF_UPDATE_ENABLED_NOTIFY telemetry histogram, etc. | Count | Percent | Status | -------------------------------------------------+--------+---------+--------+ update-enabled-update-ping-none-update-pref | 126805 | 79.73% | Match | update-disabled-update-ping-update-disabled | 27191 | 17.10% | Match | update-enabled-none-update-ping | 4210 | 2.65% | NoData | update-enabled-update-ping-update-enabled | 510 | 0.32% | Match | update-disabled-none-update-ping | 186 | 0.12% | NoData | update-disabled-update-ping-none-update-pref | 51 | 0.03% | Timing | update-enabled-update-ping-update-disabled | 44 | 0.03% | Timing | none-update-enabled-update-ping-none-update-pref | 40 | 0.03% | NoData | update-disabled-update-ping-update-enabled | 13 | 0.01% | Timing | -------------------------------------------------+--------+---------+--------+ | 159050 | update-enabled-update-ping-none-update-pref ------------------------------------------- settings.update.enabled[0] is True, there have been update pings, and update_not_pref_update_enabled_notify is None which signifies that the app.update.enabled preference has not been recorded by app update telemetry as false during an update check. This is a match between settings.update.enabled and update_not_pref_update_enabled_notify values since there were update pings and when update_not_pref_update_enabled_notify is None the value for the app.update.enabled preference is true. update-disabled-update-ping-update-disabled ------------------------------------------- settings.update.enabled[0] is False, there have been update pings, and the last time there was an update ping the update_not_pref_update_enabled_notify value was greater than 0 which signifies that the app.update.enabled preference was false at that time. This is a match between settings.update.enabled and update_not_pref_update_enabled_notify values since there were update pings and when update_not_pref_update_enabled_notify has a value the app.update.enabled preference is false. update-enabled-none-update-ping ------------------------------- settings.update.enabled[0] is True and there haven't been any update pings. Without update pings there is nothing to match and as with all pings that don't contain an update ping this needs to be investigated just as many of the other telemetry values. update-enabled-update-ping-update-enabled ----------------------------------------- settings.update.enabled[0] is True, there have been update pings, and the last time there was an update ping the update_not_pref_update_enabled_notify value was 0 which signifies that the app.update.enabled preference was true at that time. This is a match between settings.update.enabled and update_not_pref_update_enabled_notify values. update-disabled-none-update-ping -------------------------------- settings.update.enabled[0] is True and there haven't been any update pings. Without update pings there is nothing to match and as with all pings that don't contain an update ping this needs to be investigated just as many of the other telemetry values. update-disabled-update-ping-none-update-pref -------------------------------------------- settings.update.enabled[0] is False, there have been update pings, and update_not_pref_update_enabled_notify is None which signifies that the app.update.enabled preference has not been recorded by app update telemetry as false during an update check. This is not a match since there should be a value greater than 0 for update_not_pref_update_enabled_notify instead of None. This may be due to settings.update.enabled and update_not_pref_update_enabled_notify being collected at different times. update-enabled-update-ping-update-disabled ------------------------------------------ settings.update.enabled[0] is True, there have been update pings, and the last time there was an update ping the update_not_pref_update_enabled_notify value was greater than 0 which signifies that the app.update.enabled preference was false at that time. This is not a match since the value of update_not_pref_update_enabled_notify should be 0. This may be due to settings.update.enabled and update_not_pref_update_enabled_notify being collected at different times. none-update-enabled-update-ping-none-update-pref ------------------------------------------------ settings.update.enabled[0] is None, there have been update pings, and update_not_pref_update_enabled_notify is None which signifies that the app.update.enabled preference has not been recorded by app update telemetry as false during an update check. Without a value for settings.update.enabled[0] there is nothing to match and it might be worthwhile investigating this though it isn't needed for app update analysis since we should be using update_not_pref_update_enabled_notify, etc. which is recorded at the same time as the update check and thereby doesn't suffer from the preference changing in between the time the settings.update.enabled value is recorded and the time when the update check occurs. update-disabled-update-ping-update-enabled ------------------------------------------ settings.update.enabled[0] is False, there have been update pings, and the last time there was an update ping the update_not_pref_update_enabled_notify value was 0 which signifies that the app.update.enabled preference was true at that time. This is not a match since there should be a value greater than 0 for update_not_pref_update_enabled_notify instead of 0. This may be due to settings.update.enabled and update_not_pref_update_enabled_notify being collected at different times.

Here is the modified and adhoc update_disabled_mapper I used to get this data using the update orphaning notebook. As with the previous code I had to include update_ping_count_notify and update_not_pref_update_enabled_notify in the results. It also has "just in case" code to be certain nothing slips through the cracks. def update_disabled_mapper(d): status, ping = d if ping is None: return ("none-ping", ping) if ping.enabled is None: if ping.update_ping_count_notify is None: return ("none-enabled-none-update-ping", ping) if ping.update_not_pref_update_enabled_notify is None: return ("none-enabled-update-ping-none-update-pref", ping) index = 0 while index < len(ping.update_ping_count_notify): if ping.update_ping_count_notify[index] > 0: if ping.update_not_pref_update_enabled_notify[index] == 0: return ("none-enabled-update-ping-update-enabled", ping) return ("none-enabled-update-ping-update-disabled", ping) index += 1 return ("none-enabled-update-ping-none-greater-than-0", ping) if ping.enabled[0] is None: if ping.update_ping_count_notify is None: return ("none-enabled0-none-update-ping", ping) if ping.update_not_pref_update_enabled_notify is None: return ("none-enabled0-update-ping-none-update-pref", ping) index = 0 while index < len(ping.update_ping_count_notify): if ping.update_ping_count_notify[index] > 0: if ping.update_not_pref_update_enabled_notify[index] == 0: return ("none-enabled0-update-ping-update-enabled", ping) return ("none-enabled0-update-ping-update-disabled", ping) index += 1 return ("none-enabled0-update-ping-none-greater-than-0", ping) if ping.enabled[0] == True: if ping.update_ping_count_notify is None: return ("update-enabled-none-update-ping", ping) if ping.update_not_pref_update_enabled_notify is None: return ("update-enabled-update-ping-none-update-pref", ping) index = 0 while index < len(ping.update_ping_count_notify): if ping.update_ping_count_notify[index] > 0: if ping.update_not_pref_update_enabled_notify[index] == 0: return ("update-enabled-update-ping-update-enabled", ping) return ("update-enabled-update-ping-update-disabled", ping) index += 1 return ("update-enabled-update-ping-none-greater-than-0", ping) if ping.enabled[0] == False: if ping.update_ping_count_notify is None: return ("update-disabled-none-update-ping", ping) if ping.update_not_pref_update_enabled_notify is None: return ("update-disabled-update-ping-none-update-pref", ping) index = 0 while index < len(ping.update_ping_count_notify): if ping.update_ping_count_notify[index] > 0: if ping.update_not_pref_update_enabled_notify[index] == 0: return ("update-disabled-update-ping-update-enabled", ping) return ("update-disabled-update-ping-update-disabled", ping) index += 1 return ("update-disabled-update-ping-none-greater-than-0", ping) if ping.update_ping_count_notify is None: return ("other-update-disabled-none-update-ping", ping) if ping.update_not_pref_update_enabled_notify is None: return ("other-update-enabled-update-ping-none-update-pref", ping) index = 0 while index < len(ping.update_ping_count_notify): if ping.update_ping_count_notify[index] > 0: if ping.update_not_pref_update_enabled_notify[index] == 0: return ("other-update-enabled-update-ping-update-enabled", ping) return ("update-enabled-update-ping-update-disabled", ping) index += 1 return ("other-update-enabled-update-ping-none-greater-than-0", ping)