Closed
Bug 1295268
Opened 8 years ago
Closed 8 years ago
periodic HSTS updates failing with ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so)
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(firefox49+ fixed, firefox50+ fixed, firefox51+ fixed)
People
(Reporter: keeler, Assigned: catlee)
References
Details
(Keywords: regression)
Attachments
(1 file)
(deleted),
text/x-review-board-request
|
nthomas
:
review+
catlee
:
checked-in+
|
Details |
https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm94-build1-build0.txt.gz says: INFO: Downloading all the necessary pieces to update HSTS... INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js 2016-08-13 03:05:06 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/tools/getHSTSPreloadList.js [16469/16469] -> "getHSTSPreloadList.js" [1] INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors 2016-08-13 03:05:07 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.errors [124404/124404] -> "nsSTSPreloadList.errors" [1] INFO: wget -nv --no-check-certificate http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc 2016-08-13 03:05:08 URL:http://hg.mozilla.org/mozilla-central/raw-file/default/security/manager/ssl/nsSTSPreloadList.inc [1237675/1237675] -> "nsSTSPreloadList.inc" [1] INFO: Generating new HSTS preload list... INFO: Running "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/lib:. ./xpcshell /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc" ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./libxul.so) cat: nsSTSPreloadList.errors: No such file or directory ERROR: xpcshell exited with a non-zero exit code: 1 program finished with exit code 70
Comment 1•8 years ago
|
||
[Tracking Requested - why for this release]: Once this is running again we should manually update the list copy on newer branches. The last update I see was on 2016-06-04 which was 2 days before Fx49 was merged to aurora. We might consider Fx49 OK, but 50 definitely needs an update. Note: obviously it's not great if we aren't adding this security feature to newly added sites, but the real problem is that the stale list might include sites that should be _dropped_ from the list. This may "black hole" sites (or parts of sites) that had to stop using HSTS, forcing users to switch to another browser to access those sites or pages.
status-firefox49:
--- → affected
status-firefox50:
--- → affected
status-firefox51:
--- → affected
tracking-firefox49:
--- → ?
tracking-firefox50:
--- → +
tracking-firefox51:
--- → +
Updated•8 years ago
|
Keywords: regression
Comment 3•8 years ago
|
||
(from bug 1285847 comment #0) > Broke sometime in the window 2016-07-11 and 2016-07-18: > > http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central- > linux64/mozilla-central-linux64-periodicupdate-bm72-build1-build4.txt.gz > > INFO: Running > "LD_LIBRARY_PATH=/builds/slave/m-cen-l64-periodicupdate-00000/gtk3/usr/local/ > lib:. ./xpcshell > /builds/slave/m-cen-l64-periodicupdate-00000/getHSTSPreloadList.js > /builds/slave/m-cen-l64-periodicupdate-00000/nsSTSPreloadList.inc" > ./xpcshell: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found > (required by ./libxul.so) > cat: nsSTSPreloadList.errors: No such file or directory > ERROR: xpcshell exited with a non-zero exit code: 1 > > Likely to be bug 1278456, and I think we'll need > * to update the manifest > https://dxr.mozilla.org/build-central/source/tools/scripts/ > periodic_file_updates/periodic_file_updates.manifest > to unpack gcc or clang (following the linux64 build) > * and add this sort of logic > http://hg.mozilla.org/mozilla-central/file/default/build/unix/mozconfig. > stdcxx > in > http://hg.mozilla.org/build/tools/file/default/scripts/ > periodic_file_updates/periodic_file_updates.sh
Tracking for 49. catlee can you help find someone to take this on?
Flags: needinfo?(catlee)
Assignee | ||
Comment 5•8 years ago
|
||
Will do. I really wish these tasks weren't so fragile. Do we really need to run xpcshell for this?
Flags: needinfo?(catlee)
Comment 6•8 years ago
|
||
I don't think it would be that hard to make a little Python thing that dumped it in whatever format that Firefox needs it in. TBH, it would probably best if it could just consume the Chromium pseudo-JSON directly. If not, the Mozilla HTTP Observatory already uses that data and this code could probably get you most of the way towards what you need; wouldn't be too difficult to make a little Docker container that consumed things and output the right file. https://github.com/mozilla/http-observatory/blob/master/httpobs/scanner/analyzer/utils.py#L11 It produces this kind of output: https://gist.github.com/marumari/76dc2cf7a0ee4402a856567ccc925582
Reporter | ||
Comment 7•8 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #5) > I really wish these tasks weren't so fragile. Do we really need to run > xpcshell for this? The rationale for using xpcshell is that it will process the HSTS headers the same way Firefox will. That is, sites will only be accepted as HSTS by this script if Firefox would also accept them as HSTS when visiting them during normal browsing.
Comment 8•8 years ago
|
||
I can look at this on or after August 29th when I return from vacation. If it's more urgent, someone else will need to pick it up. philor already has a good starting place in comment #3.
I'm not sure how urgent it is or how much time for testing we would need. The RC build will be Sept. 5, and we should release on Sept. 13th.
Comment 10•8 years ago
|
||
I agree with dveditz that this is pretty urgent, and pretty bad from a security perspective. Many sites rely on HSTS preloading to protect their users from MitM attacks on the first use of a website, before they've seen any HSTS headers.
Reporter | ||
Comment 11•8 years ago
|
||
One option we have is for someone to run the job manually on their machine and check the changes in to the various trees. I don't have time to take care of it today, but I can probably at least run the script on Monday.
Reporter | ||
Comment 12•8 years ago
|
||
Update on comment 11: when I run this locally it hangs, so either it only works on our infrastructure machines (well, other than when this bug is an issue), or it broke in other ways since June.
Assignee | ||
Comment 13•8 years ago
|
||
Can you try with a different / older version of Firefox? How can we make these jobs more resilient? These have historically been very fragile.
Comment hidden (mozreview-request) |
Comment 15•8 years ago
|
||
mozreview-review |
Comment on attachment 8784061 [details] Bug 1295268 - Fix HSTS updates https://reviewboard.mozilla.org/r/73634/#review71554 lgtm
Attachment #8784061 -
Flags: review?(nthomas) → review+
Assignee | ||
Comment 16•8 years ago
|
||
https://hg.mozilla.org/build/tools/rev/3600436770f2000f77ec76be02e1f06d98c3b485 Bug 1295268 - Fix HSTS updates r=nthomas
Assignee | ||
Updated•8 years ago
|
Attachment #8784061 -
Flags: checked-in+
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → catlee
Assignee | ||
Comment 17•8 years ago
|
||
FWIW, this has been working on beta and release: https://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 https://hg.mozilla.org/releases/mozilla-release/rev/440936afbdbc
Assignee | ||
Comment 18•8 years ago
|
||
With my patch landed, it's hanging on central too: https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm77-build1-build1.txt.gz
Assignee | ||
Comment 19•8 years ago
|
||
I filed 1298056 to track the hang. This isn't something RelEng can fix.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
David, can you help here and in the related bug? Is this something we can fix before 49 release?
Flags: needinfo?(dkeeler)
Sounds like there may be 2 issues: update this before 49 release and also more long term address its fragility.
Reporter | ||
Comment 22•8 years ago
|
||
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #20) > David, can you help here and in the related bug? Yes - I'll investigate. > Is this something we can fix before 49 release? I don't know.
Flags: needinfo?(dkeeler)
Assignee | ||
Comment 23•8 years ago
|
||
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21) > Sounds like there may be 2 issues: update this before 49 release and also > more long term address its fragility. The automated updates are working for beta (49). e.g. http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed August 20th.
Comment 24•8 years ago
|
||
(In reply to Chris AtLee [:catlee] from comment #23) > (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #21) > > Sounds like there may be 2 issues: update this before 49 release and also > > more long term address its fragility. > > The automated updates are working for beta (49). e.g. > http://hg.mozilla.org/releases/mozilla-beta/rev/ab63c8400843 was landed > August 20th. I don't claim to understand this bug, should status-firefox49 be "fixed"?
Flags: needinfo?(catlee)
Assignee | ||
Comment 25•8 years ago
|
||
I believe so. AFAICT, 49 is being updated regularly. It was updated again on the weekend: http://hg.mozilla.org/releases/mozilla-beta/rev/571cff42892d
Flags: needinfo?(catlee)
Reporter | ||
Comment 26•8 years ago
|
||
The blocklist is separate from the hsts and hpkp periodic updates, so that may be succeeding while the others are not. Thanks to https://hg.mozilla.org/build/tools/rev/3600436770f2000f77ec76be02e1f06d98c3b485 and bug 1298056, the hsts updates are now working again on mozilla-central: https://hg.mozilla.org/mozilla-central/rev/2cdc70c1d8e1 What we'll need to do now is uplift the changes in bug 1298056 to all channels that need it (this is being taken care of in that bug). We may also need to cherry-pick a recent version of the preload list to land on beta (since if I recall correctly, we don't run the hsts periodic updater on beta).
Comment 27•8 years ago
|
||
The current configuration is: * blocklist is updated on central, aurora, beta, release, and esr45. * HSTS and HPKP are updated on central, aurora, and esr45
David, do you still need to cherrypick from bug 1298056? Or can we resolve this?
Flags: needinfo?(dkeeler)
Reporter | ||
Comment 29•8 years ago
|
||
From what I can tell, the issue described in this bug has been addressed. Also, it looks like the preload list in mozilla-beta was successfully updated on July 30th, so I think as long as we iron out the remaining issue on esr45 (bug 1285849), we're good here (no cherry-picking necessary, I think).
Flags: needinfo?(dkeeler)
Updated•8 years ago
|
Comment 30•8 years ago
|
||
Something went wrong in the mozilla-central update this weekend in HPKP: ERROR: xpcshell exited with a non-zero exit code: 3 See http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-linux64/mozilla-central-linux64-periodicupdate-bm91-build1-build0.txt.gz. New bug ?
Comment 31•8 years ago
|
||
New bug that already exists, bug 1300305
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•