It's not written in the Story, but it's implicit, that this is intended to be Windows-only.

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

It's not written in the Story, but it's implicit, that this is intended to be Windows-only.

Does it have to be? Obviously WIndows is the priority given platform reach though I did not consider initially but everything apart from Installation type is platform agnostic and we could argue that installation type could be set to pkg VS dmg on macOS and package manager VS flatpak VS Snap on Linux?

(In reply to Romain Testard [:RT] from comment #2)

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

It's not written in the Story, but it's implicit, that this is intended to be Windows-only.

Does it have to be? Obviously WIndows is the priority given platform reach though I did not consider initially but everything apart from Installation type is platform agnostic and we could argue that installation type could be set to pkg VS dmg on macOS and package manager VS flatpak VS Snap on Linux?

No, it does not have to be -- I simply saw "Installation type (Stub VS full, silent VS non silent, MSI VS exe)" and noted that all of those options were Windows-only.

In terms of implementation, the first three are all possible because the Windows installer can itself write the data into a shared location (sibling to attribution data). But this ticket requires some shared state across the OS to capture "previous Firefox installs", and that I don't really know how to achieve. On Windows, if the installer is privileged then we can use the registry and HKLM. But if we're not privileged, then I'm not aware of a cross-OS-user place for this?

(In reply to Nick Alexander :nalexander [he/him] from comment #3)

(In reply to Romain Testard [:RT] from comment #2)

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

It's not written in the Story, but it's implicit, that this is intended to be Windows-only.

Does it have to be? Obviously WIndows is the priority given platform reach though I did not consider initially but everything apart from Installation type is platform agnostic and we could argue that installation type could be set to pkg VS dmg on macOS and package manager VS flatpak VS Snap on Linux?

No, it does not have to be -- I simply saw "Installation type (Stub VS full, silent VS non silent, MSI VS exe)" and noted that all of those options were Windows-only.

In terms of implementation, the first three are all possible because the Windows installer can itself write the data into a shared location (sibling to attribution data). But this ticket requires some shared state across the OS to capture "previous Firefox installs", and that I don't really know how to achieve. On Windows, if the installer is privileged then we can use the registry and HKLM. But if we're not privileged, then I'm not aware of a cross-OS-user place for this?

Thanks, very fair comments. I updated the user story to prioritize the different requirements so it reflects the fact that Windows is clearly a priority.

In mid 2020, we implemented a probe to collect profile counts (1647422) - it seems that the way we implemented (which was an intentional choice so as not to negatively impact startup perf) caused us to not record data quite a lot of the time (see the null values represented here).

With this incoming change, I want to flag this issue so that we:

1. Don't repeat the same mistake - ie, let's take care to not impact startup perf but could we look at another way to get at this data more reliably?
2. IF we find a more reliable way to handle this, would it be possible to apply that change to the NUMBER_OF_PROFILES probe as well? It's a related investigation, so if it's not a huge amount of effort it would be nice to roll that into this work (in a separate patch is fine too).

Adam, since you're picking this one up, I wanted to flag you on this issue to get your thoughts. Does this seem do-able?

(In reply to Rachel Tublitz [:rachel] from comment #5)

In mid 2020, we implemented a probe to collect profile counts (1647422) - it seems that the way we implemented (which was an intentional choice so as not to negatively impact startup perf) caused us to not record data quite a lot of the time (see the null values represented here).

You may have meant to link to a different query, that one looks at histograms.number_of_profiles, which is an old probe set to expire in 85. That may be null in many cases because it is only sent in one ping of a session. If we take the max over all pings for 84.0.1 clients in the last few days, the null count goes from 28% down to 3%.

There is a similar issue with browser.engagement.profile_count, which is the one in bug 1647422, the proportion of NULLs goes from 45% down to 9% if aggregating across a user.

9% still seems a little high, I was curious if this was really due to that many sessions never having the 20 idle seconds needed for the late idle tasks to run. I looked for telemetry from these tasks to see how often they run:

• _sendMediaTelemetry() doesn't actually record telemetry since bug 1356046, and the probes are expired anyway. (There's a bug 1527132 to consider removing or reworking it)
• The MASTER_PASSWORD_ENABLED histogram can't tell us anything as it's a flag, so there's always a value (despite what the docs say, it looks like flag histograms like MASTER_PASSWORD_ENABLED are always initialized to 0 while count histograms like NUMBER_OF_PROFILES are not.)
• I'm not sure any telemetry would correspond to RemoteSettings, PublicSuffixList, and RemoteSecuritySettings
• Corroborate.init() records to an expired histogram (I filed bug 1685234 to remove or disable this)
• BrowserUsageTelemetry.reportProfileCount() ultimately sets browser.engagement.profile_count, as noted above this is 9% null.
• OsEnvironment.reportAllowedAppSources has a similar 9% null rate.
• I didn't dig into Services.search.checkWebExtensionEngines.

With this incoming change, I want to flag this issue so that we:

1. Don't repeat the same mistake - ie, let's take care to not impact startup perf but could we look at another way to get at this data more reliably?

2. IF we find a more reliable way to handle this, would it be possible to apply that change to the NUMBER_OF_PROFILES probe as well? It's a related investigation, so if it's not a huge amount of effort it would be nice to roll that into this work (in a separate patch is fine too).

It isn't completely clear to me what the issue is, 9% seems too high for it to just be because these tasks didn't get a chance to run, but it seems likely to be the issue given that two probes set from the late idle list have similar null rates. If that's the issue, we might consider moving both the profile counter and this to _scheduleStartupIdleTasks(), which is much more likely to run early enough. We'd just have to be careful to keep it off the main thread.

I don't think we can avoid having to aggregate, though, unless we significantly change how we collect the telemetry.

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #6)
Thank you Adam, for the digging here; this is really helpful!

(In reply to Rachel Tublitz [:rachel] from comment #5)
There is a similar issue with browser.engagement.profile_count, which is the one in bug 1647422, the proportion of NULLs goes from 45% down to 9% if aggregating across a user.

Aha! That might have been the step that was missed then in the initial analysis of the original probe. Although 9% null rate is still relatively high, it's a significant improvement over what we were seeing before (ie, after aggregating across client_ids) - :Romain - I'm of the opinion that we can live with 9% for now, and let this one go for the time being. Would you agree here, or do you want us to keep investigating?

Recording information about the installation is straightforward enough, all of the P1 items (installation version, date/time, and type) can be easily written into a file and picked up by Firefox, as mentioned in point 3 of your discussions with Molly. But I'm not clear on many details, Romain can you please provide some clarifications on these points?

• Do we want to include the install path? Or a hash of it?

This is fairly sensitive as this could commonly include the user's Windows user name. If we're just interested in whether the default path is used, there is already a default_path field in the install ping that we could pass along to Firefox. (This mostly only applies for the full installer, but it sometimes can be set with the stub, see bug 1351697, and I think it can use a previous install path). It wasn't clear from the User Story what exactly was desired here.

I'd prefer to just record the default_path field, maybe with a fix for bug 1351697.

• What information do we want for previous installs?

If we just want a count of other installs, there's a field from the uninstall ping which gives a good estimate, I could expose that in other pings.

Would it be sufficient to record just whether there was a previous install that this is replacing, or a count of installs into a given path? Or do nothing client-side and query for different install times on the same client ID?

If we want all the installation info, does this include in different directories, past installs in the same directory, both? Should this include installs that had been uninstalled?

I'd prefer to just record whether the installer is paving over a previous installation, we already know this in the installer so it can be passed along easily enough, and I think that will be most of the information needed.

• Should this only be sent in one ping?

Some of the User Story suggests that it should be sent once and then deleted, but I'm having a hard time reconciling that with the desire to maintain a history of installs. If only one ping has the install info then every query that wants to refer to e.g. the installation type would need to find that ping for that client ID, and there could be multiple client IDs/profiles for a single install.

I don't have a preference here. I had been envisioning the installation info as being something included persistently in the environment, like profile.creationDate.

• Installation time is a fairly high resolution machine identifier, we've been trying to avoid this.

Even if we only send it in one ping, that ping will have a client ID on it.

• I'm only scoping this out for Windows right now.

Mac would be tricky as I don't think there's any difference between the dmg and pkg installs currently. Conceivably we could pack a file into each prepopulated with the install type. A pkg install should be able to run a script to write a timestamp, but we've been avoiding adding a script to it, I don't understand this area very well yet.

For Linux, I think we can already detect flatpak through partner.distributionId = "mozilla-flatpak" (at least for our official package). It looks like snaps report partner.distributionId = "canonical-002", though it's harder to be certain about that one (I did check on arch and it shows up that way there, too). These IDs show up in roughly the expected numbers in this query looking at all distribution IDs on Linux for one day, 17k for Snap (canonical-002) and 5k for Flatpak (mozilla-flatpak).

(In reply to Rachel Tublitz [:rachel] from comment #7)

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #6)
Thank you Adam, for the digging here; this is really helpful!

(In reply to Rachel Tublitz [:rachel] from comment #5)
There is a similar issue with browser.engagement.profile_count, which is the one in bug 1647422, the proportion of NULLs goes from 45% down to 9% if aggregating across a user.

Aha! That might have been the step that was missed then in the initial analysis of the original probe. Although 9% null rate is still relatively high, it's a significant improvement over what we were seeing before (ie, after aggregating across client_ids) - :Romain - I'm of the opinion that we can live with 9% for now, and let this one go for the time being. Would you agree here, or do you want us to keep investigating?

If I get it right, 9% is for all profiles whereas our focus is new profiles with bug 1647422 (context being identification of new short lived profiles artificially decreasing new user retention). Leif helped on https://jira.mozilla.com/browse/DS-1365 where, when ignoring multiple NULL entries per client_id we seem to get that 16% to 26% of new profiles reporting null values - which is much higher than the anticipated 10% to 12% of new profiles now encountering 20s of idle time. Focus of this bug is not directly related to the "retention measurement optimisation" issue although it seems right to maintain data collection consistency (i.e have similar null values) between profile count and installation data to help further analysis when needed (do multiple installs correlate with multiple profiles?)

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #8)

Recording information about the installation is straightforward enough, all of the P1 items (installation version, date/time, and type) can be easily written into a file and picked up by Firefox, as mentioned in point 3 of your discussions with Molly. But I'm not clear on many details, Romain can you please provide some clarifications on these points?

• Do we want to include the install path? Or a hash of it?

This is fairly sensitive as this could commonly include the user's Windows user name. If we're just interested in whether the default path is used, there is already a default_path field in the install ping that we could pass along to Firefox. (This mostly only applies for the full installer, but it sometimes can be set with the stub, see bug 1351697, and I think it can use a previous install path). It wasn't clear from the User Story what exactly was desired here.

Thanks for the context here, I was not aware of the default path adjustment when not in an admin case.
Could we then path information on whether the default path was used or not and whether the user has admin privileges? Not being an admin user is actually a potential scenario we discover leads to specific issues and collecting this state feels valuable for troubleshooting new profile issues in general.

I'd prefer to just record the default_path field, maybe with a fix for bug 1351697.

• What information do we want for previous installs?

If we just want a count of other installs, there's a field from the uninstall ping which gives a good estimate, I could expose that in other pings.

Would it be sufficient to record just whether there was a previous install that this is replacing, or a count of installs into a given path? Or do nothing client-side and query for different install times on the same client ID?

If we want all the installation info, does this include in different directories, past installs in the same directory, both? Should this include installs that had been uninstalled?

I'd prefer to just record whether the installer is paving over a previous installation, we already know this in the installer so it can be passed along easily enough, and I think that will be most of the information needed.

I was assuming we do nothing specific. I just wanted to ensure that data sent on installation N+1 does not overwrite data sent on installation X. Not sure what the best way to implement this is (scalar?) but essentially a new installation should result in the client sending installation data with a timestamp that can then be queried across installations for a single client_id.

• Should this only be sent in one ping?

Some of the User Story suggests that it should be sent once and then deleted, but I'm having a hard time reconciling that with the desire to maintain a history of installs. If only one ping has the install info then every query that wants to refer to e.g. the installation type would need to find that ping for that client ID, and there could be multiple client IDs/profiles for a single install.

I don't have a preference here. I had been envisioning the installation info as being something included persistently in the environment, like profile.creationDate.

I was thinking of something like events sent on installations but I feel you're right there with [environment] better suiting this. Is that possible to have an element like a table of install values where new installs would create a new line?

• Installation time is a fairly high resolution machine identifier, we've been trying to avoid this.

Even if we only send it in one ping, that ping will have a client ID on it.

Can we get a resolution that is high enough to identify consecutive installs? 30s resolution maybe?

• I'm only scoping this out for Windows right now.

Mac would be tricky as I don't think there's any difference between the dmg and pkg installs currently. Conceivably we could pack a file into each prepopulated with the install type. A pkg install should be able to run a script to write a timestamp, but we've been avoiding adding a script to it, I don't understand this area very well yet.

For Linux, I think we can already detect flatpak through partner.distributionId = "mozilla-flatpak" (at least for our official package). It looks like snaps report partner.distributionId = "canonical-002", though it's harder to be certain about that one (I did check on arch and it shows up that way there, too). These IDs show up in roughly the expected numbers in this query looking at all distribution IDs on Linux for one day, 17k for Snap (canonical-002) and 5k for Flatpak (mozilla-flatpak).

Sounds good, Windows is the main concern at the moment.

Thanks for the replies Romain.

(In reply to Romain Testard [:RT] from comment #9)

Focus of this bug is not directly related to the "retention measurement optimisation" issue although it seems right to maintain data collection consistency (i.e have similar null values) between profile count and installation data to help further analysis when needed (do multiple installs correlate with multiple profiles?)

Are you saying that installation telemetry should use the same collection strategy as profile count? Or that both be changed to something else that will reduce the null rate?

I looked over the doc where bytesized discussed the collection methods for profile count, and I think all the same arguments still apply, so I'll plan to collect in the same way as profile count. If we decide that we can't accept that null rate, and that we can deal with doing the IO earlier (e.g. at delayed telemetry init), then we can consider moving both of these.

(In reply to Romain Testard [:RT] from comment #10)

Thanks for the context here, I was not aware of the default path adjustment when not in an admin case.
Could we then path information on whether the default path was used or not and whether the user has admin privileges? Not being an admin user is actually a potential scenario we discover leads to specific issues and collecting this state feels valuable for troubleshooting new profile issues in general.

Ok, I will include the admin_user and default_path flags, as currently reported in the install ping.

I'd prefer to just record whether the installer is paving over a previous installation, we already know this in the installer so it can be passed along easily enough, and I think that will be most of the information needed.

I was assuming we do nothing specific. I just wanted to ensure that data sent on installation N+1 does not overwrite data sent on installation X.

Could this be handled with a count of the number of times that this has been reinstalled in the same location? Are you worried about data being overwritten on the client side, or just being indistinguishable in our server-side collection?

Not sure what the best way to implement this is (scalar?) but essentially a new installation should result in the client sending installation data with a timestamp that can then be queried across installations for a single client_id.

Being able to use the timestamp as an identifier is what I'm trying to avoid. It sounds like all we want to do is to be able to tell if there are multiple installations, which a reinstall count or flag should be sufficient for.

I was thinking of something like events sent on installations but I feel you're right there with [environment] better suiting this.

Sorry for suggesting this, I hadn't read the "Timings options" doc first. We're being strongly discouraged from adding anything to the environment, so providing them as scalars (as with profile count) or events (though I'm not sure how to avoid reporting this more than once) would be best.

• Installation time is a fairly high resolution machine identifier, we've been trying to avoid this.

Even if we only send it in one ping, that ping will have a client ID on it.

Can we get a resolution that is high enough to identify consecutive installs? 30s resolution maybe?

If it can be used to identify that this is the same install, then it's a new identifier. I think that a reinstall count, or even just a reinstall flag, would be sufficient for signalling that this is not a fresh install.

I propose adding the following Telemetry scalars on Windows. They would be written to a file in the install directory by full installer after a successful install, read by Firefox when the late / best effort idle tasks run (same as browser.engagement.profile_count), and sent as Telemetry scalars when the file is present, under installation., e.g. installation.version.

• version, build_id - read out of application.ini after a successful install
• install_existed - if there was already an install at the same location (determined by checking for the exe in the target directory before install)
• profile_dir_existed - if there was already a top level Firefox profiles directory, this could be a stronger signal of multiple installs on a system
• installer_type - full or stub
• admin_user - if the installer is running as administrator
• If the full installer was run directly without the stub, also include these additional fields from the install ping:
  • silent
  • from_msi
  • default_path

Additionally I'll report other_installs like otherInstalls in the uninstall ping.

If install_existed, profile_dir_existed, and other_installs don't seem sufficient, this could be extended with the version and build ID of the previous install, and/or a count of installs in the current location could be kept. I'd want to remove the count when Firefox is uninstalled, though, so it might not be particularly useful.

If we really need to do a history of install information within a single ping from the client, I think that needs to be left for a separate effort.

Please let me know if you have any concerns with this proposal.

Romain, just so this one doesn't get lost - ni-ing you here to confirm we're good to move forward with Adam's above suggestions?

Thanks Adam, more details:

• Collection strategy: agreed that I see no better way per Kirk's doc. I pinged Leif on https://jira.mozilla.com/browse/DS-1365 for any further idea he may have but let's assume we use the same collection strategy as previous implementation
• Re - admin_user and default_path flags - thanks I updated the user storyy accordingly
• "Could this be handled with a count of the number of times that this has been reinstalled in the same location? Are you worried about data being overwritten on the client side, or just being indistinguishable in our server-side collection?" I see value in installation dates/time and information attached to previous installs (i.e is the scenario of a non admin user installing several times daily frequent - this would help shape how we can help users) so a count would not be ideal. I have no particular concern, just making sure that the need is to be able to analyze a log of installations per client. To possibilities to use the timestamp as an identifier, rather than using the actual timestamp, could we use an approximation of it that prevents uniqueness? In general is not the profile creation date using a similar timestamp?
• If we use events, each install would be an event and each event has a timstamp by design - so the timestamp issue is a non issue?
• "If it can be used to identify that this is the same install, then it's a new identifier" This is the piece I don't get, all event telemetry is attached to a client_id, the client_id is the identifier tying different installs so we're not creating a new one?

Would using events work for you using the same structure as you outlined above excluding 'install_existed' given that we can work out if other installs were previously made by looking-up earlier events?

(In reply to Romain Testard [:RT] from comment #13)

To possibilities to use the timestamp as an identifier, rather than using the actual timestamp, could we use an approximation of it that prevents uniqueness? In general is not the profile creation date using a similar timestamp?

• If we use events, each install would be an event and each event has a timstamp by design - so the timestamp issue is a non issue?
• "If it can be used to identify that this is the same install, then it's a new identifier" This is the piece I don't get, all event telemetry is attached to a client_id, the client_id is the identifier tying different installs so we're not creating a new one?

A timestamp / installation log are specific to the installation, but would be reported for all profiles the use the installation. The event would only be reported after the profile first runs, so the event's timestamp (or a profile creation date) wouldn't be correlated across profiles / client_ids. Maybe that's acceptable, see below.

In bug 1677497 there is still discussion going with Privacy about whether the download token, which would identify all profiles using a particular install, is ok for a Release population. If Privacy approves that, then it seems likely we would be ok with sending an install timestamp as well.

Would using events work for you using the same structure as you outlined above excluding 'install_existed' given that we can work out if other installs were previously made by looking-up earlier events?

I included install_existed to be accurate without relying on being able to query a single profile / client_id, but I think I'm coming to understand that cross-profile information isn't needed or even desired. I won't include it if you don't see a use for it. Maybe other_installs and profile_dir_existed aren't useful either?

Here's an idea for a useful, per-profile install timestamp:

Send the install info in an event when the profile first runs on this install, and let the time of that event stand in for "install time". The installer would generate a timestamp (or other identifier each time the installer runs), but this wouldn't be sent as part of the ping. Instead the profile would save a copy for comparison, and when it changes we'll know to send the new event.

As you said having these events will let us build an install log when we do our analysis, as long as Firefox ran on the same profile after each install. This wouldn't, however, show when exactly the installer ran, or if there were multiple reinstalls between runs of Firefox. Does that sound ok?

Depends on D103241

Depends on D103242

Comment on attachment 9199684 [details]
Installation Telemetry data review request.txt

lgtm; thanks for the clear descriptions in Events.yaml.

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in the probe definition files and the Probe Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

These round up to cat 2, interaction data.

5. Is the data collection request for default-on or default-off?

Default-on.

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data?

The probes are set to expire in 94; renewals are the responsibility of the collectors.

9. Does the data collection use a third-party collection tool?

No.

Re: my proposal in comment 14, we discussed this on Slack, highlights below:

Romain said:

A profile event per installation sounds good so long as we cover reinstall scenarios where we install over an existing profile.

and provided a set of steps as an example, I wrote up a clearer version of the steps as follows:

1. Firefox is installed to Location 1, this saves Installation Details 1.
2. Firefox runs from Location 1, creating Profile 1. This sends a ping with Installation Details 1.
3. Firefox is closed and reinstalled to Location 1, this saves Installation Details 2.
4. Firefox runs from Location 1, using the default Profile 1. This sends another ping with Installation Details 2. An analyst can find both events associated with Profile 1 to see Installation Details 1 & 2.
5. Firefox runs from Location 1 with new Profile 2. This sends a ping with Installation Details 2.
6. Firefox is uninstalled from Location 1.
7. Firefox is reinstalled in Location 1, this saves Installation Details 3.
8. Firefox runs from Location 1 using Profile 2. This sends a ping with Installation Details 3. An analyst can find both events associated with Profile 2 to see Installation Details 2 & 3.

and Romain approved. Note that Installation Details 2 is sent from both Profile 1 and Profile 2 (as was also the case in Romain's example).

I proposed the following list of fields, to which Romain agreed:

version, build_id, installer_type, admin_user, install_existed, profile_dir_existed, silent, from_msi, default_path

I modified this slightly for Events.yaml, profile_dir_existed becomes profdir_existed to fit in the name length limts, and the installer_type became the object of the Event to simplify queries (since full and stub events have somewhat different sets of extra keys).

Thanks Adam, I'm cleaning up the user story so decisions are clear.

https://hg.mozilla.org/mozilla-central/rev/7646892c8819
https://hg.mozilla.org/mozilla-central/rev/5b190cbf9e6a
https://hg.mozilla.org/mozilla-central/rev/255e61354480