In bug 1308705 we validated the search probes landed with bug 1303333. Since we uplifted the probes, we should cross check that they behave correctly on Beta. Here's the notebook and SQL query used on Nightly: [1] - https://gist.github.com/Dexterp37/078e254d3e12d57bf45d5417504d9b71 [2] - https://sql.telemetry.mozilla.org/queries/1612 We also want to cross check that the SEARCH_COUNTS and search scalar probes are behaving similarly.

Now that we have Beta data available, I took some time to run the validation notebook that we used for Nightly on Beta [1]. Key highlights: - Only 23% of the pings report search engagement data on Beta (see cell 7). - SEARCH_COUNTS follow the same trend (see cell 8). - This is somehow consistent with the data from Beta before the engagement patches landed (see cell 10) - We've got matching SEARCH_COUNTS and search engagement probes for over 99% of the pings (see cell 14) The mismatching values are due to the one-off search regression that was introduced by the search engagement patches and fixed in Beta 3. @Brendan is there anything else to do here? [1] - https://gist.github.com/Dexterp37/346226b6b4b148cc5d24e732a7ceaafe

Looks good, thanks Alessio. My only question is, beneath cell [17] you said "It looks that almost 76% of the pings don't have the SEARCH_COUNTS histogram nor the search engagement measurements. That's odd but... seems expected or, at least, not a problem in the counts." -- did you just mean it seems odd because it seems like more pings should have associated searches? Or did it seem like incorrect behavior (e.g., expecting to see the histogram even in the case of zero searches)? [[[ I don't think that this would make a big difference to the conclusion, just a note on a typo in cell [11] -- i think you meant "if len(search_counts.keys()) < 1:". As entered, "if search_counts.keys() < 1:" will always evaluate to false b/c search_counts.keys() returns a list. "Note that comparing objects of different types is legal. The outcome is deterministic but arbitrary: the types are ordered by their name. Thus, a list is always smaller than a string, a string is always smaller than a tuple, etc." (see: https://docs.python.org/2/tutorial/datastructures.html#comparing-sequences-and-other-types) ]]]

(In reply to brendan c from comment #2) > Looks good, thanks Alessio. My only question is, beneath cell [17] you said > "It looks that almost 76% of the pings don't have the SEARCH_COUNTS > histogram nor the search engagement measurements. That's odd but... seems > expected or, at least, not a problem in the counts." -- did you just mean it > seems odd because it seems like more pings should have associated searches? > Or did it seem like incorrect behavior (e.g., expecting to see the histogram > even in the case of zero searches)? Yes, sorry for being cryptic there! I meant the former, I would have expected more pings to have associated search data. But it's a personal belief disproved by the data (and that roughly matches what we saw in Nightly and in previous builds). So no, I don't think there's an incorrect behaviour lurking in the shadows! > [[[ > I don't think that this would make a big difference to the conclusion, just > a note on a typo in cell [11] -- i think you meant "if > len(search_counts.keys()) < 1:". As entered, "if search_counts.keys() < 1:" > will always evaluate to false b/c search_counts.keys() returns a list. > "Note that comparing objects of different types is legal. The outcome is > deterministic but arbitrary: the types are ordered by their name. Thus, a > list is always smaller than a string, a string is always smaller than a > tuple, etc." (see: > https://docs.python.org/2/tutorial/datastructures.html#comparing-sequences- > and-other-types) > ]]] Good catch! Yes, that doesn't change the results much. I ran the notebook again, the results still stand. Brendan, can we go on and close this bug?

Thanks for the clarification Alessio. Yes, let's go ahead and close it out.