Filing on behalf of wsmwk. 38.6.0 shows up in the recent versions list for Thunderbird, but 38.7.0 does not, suggesting something went wrong with ftpscraper. 38.7.0 shipped two days ago, and was built early on Mar 11 UTC. It's possible bug 1250142 has broken scraping but I can't see the logs to confirm that. Files the it can read: http://archive.mozilla.org/pub/thunderbird/candidates/38.7.0-candidates/build1/win32/en-US/thunderbird-38.7.0.json http://archive.mozilla.org/pub/thunderbird/candidates/38.7.0-candidates/build1/win32_info.txt

Thanks for those links. If I run it locally I should be able to debug why it's not being picked up.

When I run this locally, it finds and (pretends to) insert: INSERT BUILD ('thunderbird', '38.7.0', 'win', u'20160310150025', 'beta', 99, 'mozilla-beta', 'build1') {'ignore_duplicates': True} I'm guessing the problem here is that it thinks it's a beta release even though http://archive.mozilla.org/pub/thunderbird/candidates/38.7.0-candidates/build1/win32/en-US/thunderbird-38.7.0.json says `"moz_update_channel": "release",`. Hmmm

Well, I spoke too soon. Here's what it (pretends to) inserts: INSERT BUILD ('thunderbird', '38.7.0', 'linux', u'20160310150025', 'release', None, u'mozilla-esr38', 'build1') {'ignore_duplicates': True} 2016-03-18 16:47:24,600 DEBUG - SocorroAppBaseClass - - MainThread - is final beta version 38.7.0 INSERT BUILD ('thunderbird', '38.7.0', 'linux', u'20160310150025', 'beta', 99, 'mozilla-beta', 'build1') {'ignore_duplicates': True} INSERT BUILD ('thunderbird', '38.7.0', 'linux', u'20160310150025', 'release', None, u'mozilla-esr38', 'build1') {'ignore_duplicates': True} 2016-03-18 16:47:25,703 DEBUG - SocorroAppBaseClass - - MainThread - is final beta version 38.7.0 INSERT BUILD ('thunderbird', '38.7.0', 'linux', u'20160310150025', 'beta', 99, 'mozilla-beta', 'build1') {'ignore_duplicates': True} INSERT BUILD ('thunderbird', '38.7.0', 'mac', u'20160310150025', 'release', None, u'mozilla-esr38', 'build1') {'ignore_duplicates': True} 2016-03-18 16:47:28,522 DEBUG - SocorroAppBaseClass - - MainThread - is final beta version 38.7.0 INSERT BUILD ('thunderbird', '38.7.0', 'mac', u'20160310150025', 'beta', 99, 'mozilla-beta', 'build1') {'ignore_duplicates': True} INSERT BUILD ('thunderbird', '38.7.0', 'win', u'20160310150025', 'release', None, u'mozilla-esr38', 'build1') {'ignore_duplicates': True} 2016-03-18 16:47:33,309 DEBUG - SocorroAppBaseClass - - MainThread - is final beta version 38.7.0 INSERT BUILD ('thunderbird', '38.7.0', 'win', u'20160310150025', 'beta', 99, 'mozilla-beta', 'build1') {'ignore_duplicates': True}

In the prod database; here's what it has inserted in releases_raw:: breakpad=> select * from releases_raw where product_name ='Thunderbird' and version = '38.7.0'; product_name | version | platform | build_id | build_type | beta_number | repository | update_channel | version_build --------------+---------+--------------+----------------+------------+-------------+---------------+----------------+--------------- thunderbird | 38.7.0 | linux | 20160310150025 | beta | 99 | mozilla-beta | beta | thunderbird | 38.7.0 | mac | 20160310150025 | beta | 99 | mozilla-beta | beta | thunderbird | 38.7.0 | win | 20160310150025 | beta | 99 | mozilla-beta | beta | thunderbird | 38.7.0 | linux-i686 | 20160311030358 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160311030358 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160311030358 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160311030358 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160312030211 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160312030211 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160312030211 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160312030211 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160313030235 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160313030235 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160313030235 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160313030235 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160314030245 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160314030245 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160314030245 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160314030245 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160315030236 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160315030236 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160315030236 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160315030236 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160316030248 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160316030248 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160316030248 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160316030248 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160317030247 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160317030247 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160317030247 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160317030247 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-i686 | 20160318030426 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux-x86_64 | 20160318030426 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | mac | 20160318030426 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | win32 | 20160318030426 | nightly | | comm-esr38 | nightly | thunderbird | 38.7.0 | linux | 20160310150025 | release | | mozilla-esr38 | release | thunderbird | 38.7.0 | mac | 20160310150025 | release | | mozilla-esr38 | release | thunderbird | 38.7.0 | win | 20160310150025 | release | | mozilla-esr38 | release | (38 rows) So I think things are good there. It's just not succeeding to transfer that to the product_versions table. And I bet there's some crazy store procedure business logic rule in there that's blocking it. I'll have to return to this next week. Run out of time for now.

There's a few strange things there * the build_id for the release build is 20160310150025, which has both beta and release for build_type * the repository's are quite different between release and nightly cases * platform of linux for release case, while we have linux-i686 and linux-x86_64 for nightlies FWIW the nightly builds appear to come from dirs like http://archive.mozilla.org/pub/thunderbird/nightly/2016/03/2016-03-18-03-04-26-comm-esr38/. Maybe related, Firefox 45.0.1 shipped late last week and isn't shown by default on https://crash-stats.mozilla.com/home/products/Firefox. It does appear in the Current Versions drop down, but is below the divider line.

Pardon my ignorance of how releng works, but what does this mean? Wrong data written into these .json files? Or something socorro is scraping incorrectly?

Hard to say, as I don't really grok the way Socorro works behind the scenes. What does Socorro need to make the UI work as expected ? ie show Thunderbird 38.7.0 and Firefox 45.0.1 as the latest releases. FWIW, using .json files for releases is sufficiently new that we've never needed to ensure the contents make sense there.

Kairo, Can you see what the problem is here. Your skill set and domain knowledge fits somewhere between Nick and me :)

Thunderbird Version 38.7.1 is also missing (In reply to Peter Bengtsson [:peterbe] from comment #8) > Kairo, Can you see what the problem is here. Your skill set and domain > knowledge fits somewhere between Nick and me :) Kairo is away for more than a month. That's long time. Is there someone else we can call upon?

(In reply to Nick Thomas [:nthomas] from comment #5) > ... > Maybe related, Firefox 45.0.1 shipped late last week and isn't shown by > default on https://crash-stats.mozilla.com/home/products/Firefox. It does > appear in the Current Versions drop down, but is below the divider line. That's only because 45.0.1 hasn't been made a featured version at https://crash-stats.mozilla.com/admin/featured-versions/ which auiu Kairo might have been doing manually

Wayne, I'll try to dig into this today.

Did this manually to get 38.7.0 to appear as a valid product version: breakpad=> begin; BEGIN breakpad=> update releases_raw set repository='mozilla-release' where product_name='Thunderbird' and version = '38.7.0' and update_channel='release'; UPDATE 3 breakpad=> commit; COMMIT breakpad=> select update_product_versions(); update_product_versions ------------------------- t (1 row) breakpad=> select * from product_versions where product_name='Thunderbird' and version_string='38.7.0'; product_version_id | product_name | major_version | release_version | version_string | beta_number | version_sort | build_date | sunset_date | featured_version | build_type | has_builds | is_rapid_beta | rapid_beta_id | build_type_enum | version_build --------------------+--------------+---------------+-----------------+----------------+-------------+---------------+------------+-------------+------------------+------------+------------+---------------+---------------+-----------------+--------------- 3494 | Thunderbird | 38.7 | 38.7.0 | 38.7.0 | | 038007000r000 | 2016-03-10 | 2016-07-14 | f | release | f | f | | release | (1 row) breakpad=>

On Wednesday 3-30 manually added 38.7.0 to socorro's PG table, and I specified it as a featured version. 38.7.0 is now available as a choice in Soccorro, and it is showing ADI. We still don't know why the screen scraping process that takes data from isn't working correctly. Which is distrubing, because it won't be simple to get comparative crash rates between new builds and old builds. More to come...

Pretty sure the reason it won't go in is because of the repository. The scraping found it and saw that its repository was "mozilla-esr". But the stored procedure that goes from scrapings to releases depends on a JOIN with the release_repositories table. breakpad=> select * from release_repositories ; repository -------------------------------- mozilla-central mozilla-1.9.2 comm-central comm-1.9.2 comm-central-trunk mozilla-central-android mozilla-release mozilla-beta mozilla-aurora mozilla-aurora-android mozilla-esr10 mozilla-esr10-android b2g-release mozilla-aurora-android-xul mozilla-central-android-xul comm-aurora mozilla-central-android-api-11 mozilla-aurora-android-api-11 mozilla-central-android-api-15 mozilla-aurora-android-api-15 mozilla-beta-android-api-15 mozilla-release-android-api-15 (22 rows) So, had the repository value in the .json files for those release candidates, on archives.mozilla.org, been mozilla-release all would have been fine.

c/On Wednesday 3-30 manually added 38.7.0/On Wednesday 3-30 PETERB manually added 38.7.0/ > So, had the repository value in the .json files for those release candidates, on archives.mozilla.org, been mozilla-release all would have been fine. Right. The kicker is, I built both 38.6.0 and 38.7.0 and didn't do any different in ship-it afaik except cset. compare https://archive.mozilla.org/pub/thunderbird/candidates/38.7.0-candidates/build1/win32/en-US/thunderbird-38.7.0.json https://archive.mozilla.org/pub/thunderbird/candidates/38.6.0-candidates/build1/win32/en-US/thunderbird-38.6.0.json https://archive.mozilla.org/pub/thunderbird/candidates/38.7.0-candidates/build1/win32/en-US/thunderbird-38.7.0.txt https://archive.mozilla.org/pub/thunderbird/candidates/38.6.0-candidates/build1/win32/en-US/thunderbird-38.6.0.txt both specify mozilla-esr38. Does this need to be added to the table?

I honestly can't explain that. Perhaps the code that we used to have, that reads from the .txt files, did something different. In https://archive.mozilla.org/pub/thunderbird/candidates/38.6.0-candidates/build1/win32/en-US/thunderbird-38.6.0.txt it clearly mentions a esr38 repository. Hmm... curious. Either way, we'll need to change how the .json file looks like I think. Otherwise it'll be saved under a repository we don't recognize and it won't become a productversion.

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #9) > Kairo is away for more than a month. That's long time. Is there someone else > we can call upon? Bah, I just realized my Bugzilla message was wrong, I guess wishful thinking. I'm back today actually. :)(In reply to Peter Bengtsson [:peterbe] from comment #14) > Pretty sure the reason it won't go in is because of the repository. The > scraping found it and saw that its repository was "mozilla-esr". But the > stored procedure that goes from scrapings to releases depends on a JOIN with > the release_repositories table. OK, for one, why do we need that filtering at all? For the other, why is the ESR repository name not in there? We regularly build stuff from there, both for Thunderbird and Firefox, and we should support crash data for those.

(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #17) > (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #9) > > Kairo is away for more than a month. That's long time. Is there someone else > > we can call upon? > > Bah, I just realized my Bugzilla message was wrong, I guess wishful > thinking. I'm back today actually. :)(In reply to Peter Bengtsson [:peterbe] > from comment #14) > > Pretty sure the reason it won't go in is because of the repository. The > > scraping found it and saw that its repository was "mozilla-esr". But the > > stored procedure that goes from scrapings to releases depends on a JOIN with > > the release_repositories table. > > OK, for one, why do we need that filtering at all? For the other, why is the > ESR repository name not in there? We regularly build stuff from there, both > for Thunderbird and Firefox, and we should support crash data for those. If only I knew :) We can either dig up some answers to your great questions from the people who implemented it. Or we can just add mozilla-esr as a recognized repository. Not sure what would happen to Firefox releases though.

Peterb, can you add 38.7.2 release (released yesterday) and 45.0b4 beta (released today) into the PG tables? currently 45.0b4 isn't available as a version in the UI, and submitted crashes are coming up as 45.0b0 with processor note "release channel is beta but no version data was found - added "b0" suffix to version number", like bp-c1fefb39-f214-4735-8018-c65002160405

(In reply to Peter Bengtsson [:peterbe] from comment #18) > We can either dig up some answers to your great questions from the people > who implemented it. > Or we can just add mozilla-esr as a recognized repository. Not sure what > would happen to Firefox releases though. Let's add mozilla-esr for sure, we want Firefox ESR releases to be in Socorro as well anyhow :)

Also, can we file a followup to find out why we filter for repositories at all and remove that if it may serve no actual purpose?

On stage, I've added mozilla-esr to release_repositories. That means that the update_product_versions stored procedure SHOULD pick up 38.7.2 and 45.0b4 as recognized product versions. Let's hope it works!

Actually, by doing...:: insert into release_repositories values ('mozilla-esr38'), ('mozilla-esr45'), ('comm-esr38'), ('comm-esr45'); And...:: select update_product_versions(); Now all these Thunderbird versions appear: https://crash-stats.allizom.org/api/ProductVersions/?active=true&product=Thunderbird&start_date=%3E2016-01-01 If this looks right Wayne, I'll do the same on prod.

FWIW, this insertion fixed bug 1022884 on stage as well! \o/

Same done on prod. Reopen if that doesn't make the product versions available in prod. (but let 1h cache expire before checking)