Not entirely sure what is going on with this but in bug 1374364 when firefox.exe launches the updater to apply an update the firefox.exe process is not exiting even when waiting more than the 63 seconds that nsTerminator.cpp has defined. The client waited a couple of minutes and the firefox.exe process was still around. If the client kills the firefox.exe process in task manager the update succeeds. One suggestion that has come up in the past is to have the updater kill the process. The issue with this is that the maintenance service launches the updater as the SYSTEM account and it might be possible to do evil things such as have the updater kill system level processes if this were implemented in the updater. So, if a sledgehammer approach such as killing the firefox.exe process is taken I think it would be better performed outside of the updater. cc'ing some of the people that might have input

Note: the firefox.exe process also appears to not exit on startup as well for the client in bug 1374364.

To figure out what Firefox is doing at the time of the hang, follow the instructions at https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox That will also help determine whether nsTerminator was running or what was going on.

I'm not able to reproduce so needinfoing the reporter of bug 1374364 to see if they are able to perform the steps in comment #2.

It isn't about any crash. After update, no matter whether it is through "Restart to apply updates" or through closing Nightly and launch it manually - Nightly simply doesn't start. Killing Nightly process doesn't always allow update to proceed, it sometimes causes crashes and disability of launching the browser. I need then uninstall and install Nightly again. However, sometimes it works and updated Nightly launches. If I don't kill any process, firefox.exe will remain in Task Manager until restart of the PC. Attempts to launch Nightly will cause another firefox.exe to appear every time.

There is also bug 1374097 where it appears to have happened without updating as well > Wednesday June 21 > Bug report updated > Checked about nightly Nightly up todate > Closed nightly waited 3 minutes 4 background nightly processes open 0% CPU > Tried to start nightly did not start 5 background processes open 0% CPU > Ended the 5 processes started nightly successfully > Used nightly for about 30 minutes 5 processes open closed nightly 4 closed 1 > stayed open > Waited 3 minutes 1 still open started nightly it started normally > Used nightly for about 90 minutes 10 processes open ckosed nightly 4 > processes open > Waited 3 minutes 4 processes open started nightly started nightly opened > normally

Another one of interest... bug 1374043

tkdubiel, I realize that there is currently not a crash. The request in comment 2 is about intentionally crashing firefox.exe using a special diagnostic tool. If we crash it when it's in that hung state, it should generate a crash report so that I can diagnose what's going on.

This tool launches and closes after miliseconds. When I ran it by cmd, I see only "Checking process pid and name Checking process pid and name .... ... Could not create remote thread."

I recorded the problem. I also noticed that every time closing Nightly doesn't end fierfox.exe process. However, when I don't update, there is no problem with launching Nightly after that. https://youtu.be/Z4aEhx8GODw

Filed bug 1375549 to consider not trying to launch Firefox when Firefox is in this state since it won't be successful.

This is the latest checks that I have done if you can recomend something that I can try will give it a go Friday June 23 0400 closed nightly 0403 background process one start nightly 0404 nightly started 4 process open update available 0405 download update 0408 close nightly 1 process open 0411 1 process open start nightly 0414 2 process open nightly did not start ended processes start nightly 0415 nightly started did not update I have been closing Nightly multiple time over the last 2 days and it opens and closes normally On downloading an update and trying to update nightly will not start I do not know if on trying to the update it closes all nightly processes and creates a new procees as it happens so fast or if it leaves one of the existing processes The nightly software updater process then opens stays open for about a minute then closes neither process appears to get any CPU resourses and it does not start The version that is installed at present is 56.0a1 (2017-06-20) Have tried to install two new versions since (multiple times) but neither will install and nightly does not start Saturday June 24 Use about nightly update to try and upde The same happened as yesterday

Bob, are you able to crash Firefox intentionally during this process while Firefox is hanging, and then let us know a link to your crash report? There's instructions here: https://developer.mozilla.org/en-US/docs/Mozilla/How_to_report_a_hung_Firefox

From my side I can tell you that every time I close Nightly, crash report is generated. - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Application Error" /> <EventID Qualifiers="0">1000</EventID> <Level>2</Level> <Task>100</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2017-06-26T18:38:50.573592800Z" /> <EventRecordID>11614</EventRecordID> <Channel>Application</Channel> <Computer>DESKTOP-A31R1EV</Computer> <Security /> </System> - <EventData> <Data>firefox.exe</Data> <Data>56.0.0.6386</Data> <Data>5950eed2</Data> <Data>unknown</Data> <Data>0.0.0.0</Data> <Data>00000000</Data> <Data>c0000005</Data> <Data>00007ff8b77703a4</Data> <Data>39f0</Data> <Data>01d2eeab2195e67f</Data> <Data>C:\Program Files\Nightly\firefox.exe</Data> <Data>unknown</Data> <Data>78d432ff-90a3-4482-b156-452fe3562bec</Data> <Data /> <Data /> </EventData> </Event>

Not crash report, but problem "The work (job) has been finished"

Opened about:crashes there was 4 crash reports from last year that had not been submitted plus many that had been submitted Cleared all crash reports Downloaded crashfirefox.exc Closed nightly no open process Opened nightly opened normally Executed crashfirefox a window opened it scrolled so fast I could not read anything then window closed Nightly continued working Checked about:crashes no crash report Closed nightly all processes closed Opended nightly it opened normally Executed crashfirefox as administrator got the same result as above and no crash report Downloaded nightly update Clicked restart to update 2 nightly processes were open updater opened 1 nightly process closed Executed crahfirefox whilst updater was open Executed crashfirefox after updater had closed and the nightly prosses was open Ended open nightly process Opened nightly it started normally Checked about:crashes no crash reports Just looking again what tkdubiel has above C:\Program Files\Nightly all my nightly files have always been in this directory When this problem first started I discovered that all the files were now in c:\users\bob07\local\nightly I did not change this and assumed that it was done by a nightly update It appears to have occurred at the time when the latest ver of windows 10 was installed Could it be something as simple as this

I don't think so. I have Nightly in normal directory, not in users/local.

When I started nightly this morning it updated before opening now running version 56.0a1 (2017-06-27)on closing all processes now shut down Tested this a number of times With nightly files being installed in \appdata\local\nightly instead of \program files\nightly tried to work this out When the problem first happened one of the things I did was uninstall an reinstall it Did this again this morning but the installation program that I used then and now does not give the option to change the installation directory This morning it put it into \program files so how it got into \appdata ???????????????? Thank you for your help Bob

It looks like this bug is fixed in today build. All firefox processes exit normally after closing Nightly, one remaining also exits after a while.

tkdubiel, if you are running a 64-bit version of Firefox then maybe the old crashfirefox.exe won't work (it's a 32-bit binary). Here's a 64-bit binary: https://ci.appveyor.com/api/buildjobs/oe0b55ardbgfkcuc/artifacts/x64/Release/crashfirefox.exe It might be valuable to go "back" to one of the broken builds and run this program on it just to see exactly what was going on and see if a specific Firefox checkin fixed it.

The same - "could not create remote thread"

28 June 1925 Watching youtube video got message to restart to update clicked to restart Nightly closed update window opened and appeared to complete Nightly did not start 8 processes were open ended processes Started nightly it started normally Checked version it is 56.0a1 (2017-05-27) Ran crashfirefox The URL is Wednesday https://crash-stats.mozilla.com/report/index/a1b0d521-f788-470b-b2a5-5362d0170628 You were correct have 64 bit installed and downloaded that version of crashfirefox earlier today Guess we wait until tomorrows update Or I can restore both the program files and the profile files to where they were when it successfully updated this morning and update again though do not know where to execute crashfirefox during this update Looked at how to set up a previous build I must be dumb could not work out at this stage how to do it

Today processes didn't exit and update still couldn't proceed.

However, when I closed Nightly after update, all processes exited, but one after another, not all at once.

Thursday 29 June 02:30 Started nightly loaded OK Received message to download update downloaded Exit nightly 1 process remained process still open Started nigthly 1 nightly process opened the other remained open and update process started Update process open for about 1 minute then closed Window opened updating files Window closed nightly did not start 2 processes open Started nightly 3rd opened nightly did not start Ended processes Started nightly opened normally Checked version it updated to 56.0a1 (2017-06-28) Ran crashfirefox Checked crash reports 3 new ones submitted https://crash-stats.mozilla.com/report/index/886cf9b6-354a-4d51-97e0-795970170628 https://crash-stats.mozilla.com/report/index/2c6f206b-a294-489a-bd2d-b68590170628 https://crash-stats.mozilla.com/report/index/b4f8703a-d719-4057-8dbc-4063d0170628

Friday 30 June 01:00 Started nightly loaded OK Received message to download update downloaded Exit nightly all processes closed Started nightly 1 nightly process opened update process started Update Friday process open for about 1 minute then closed 1 nightly process open 2 update processes opened both got some CPU time then both closed Window opened updating files Window closed nightly did not start 1 process open Started nightly 2nd nightly process opened got a little CPU 0.1% nightly did not start Ended processes Started nightly opened normally Ran crashfirefox restated nightly started normally Checked version it updated to 56.0a1 (2017-06-29) Checked crash reports 2 new ones submitted Https://crash-stats.mozilla.com/report/index/9ae08274-016f-4de2-ada0-0d1860170629 https://crash-stats.mozilla.com/report/index/9ae08274-016f-4de2-ada0-0d1860170629

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #6) > Another one of interest... bug 1374043 I have done tests in that bug. I wrote: https://bugzilla.mozilla.org/show_bug.cgi?id=1374043#c23 <snip> > > *** Summary to date *** > > Tested Windows 32 bit builds (2017-06-14, 15, 16 and 17). > All 'close normally'. > > Tested Windows 64 bit builds (2017-06-14, 15, 16 and 17). > > Version 56.0a1 (2017-06-14) (64-bit) > Build ID 20170614030206 > > Is like the 32 bit builds, it closes normally. > > However, > Version 56.0a1 (2017-06-15) (64-bit) > Build ID 20170615030208 > > (and 2017-06-16 and 2017-06-17) all have the 'Windows Crash' on Close. > > The Regression is in > Version 56.0a1 (2017-06-15) (64-bit) > Build ID 20170615030208 > > but it does not seem to be in the 32bit builds. I also added: https://bugzilla.mozilla.org/show_bug.cgi?id=1374043#c24 > I firmly believe that this bug [i.e. Bug 1374043] is contributing to the 'Nightly updating issues', > seen since 2017-06-15, on Windows Nightly 64 bit builds. > Possible duplicates of THAT bug include: > Bug 1373918 "Nightly is crashing at close" > > and > Bug 1376762 "Firefox Nightly (56) 20170615+ crashes on exit (Windows 7 & 10 64bit)" > > The latter, is concise - and has a Regression pointing to Bug 1372375 DJ-Leith

Probably worth re-testing this with a build on/after 2017-07-06, per a fix that landed for a related bug. If that passes, then we'd want to apply the same test to everything this bug blocks.

DJ-Leith: can you re-test with a current build to see if it's resolved?

Sorry for the slow response to the needinfo, I have been away. If I understand correctly, Bug 1372375 "EnsureWalkThreadReady takes locks during the profiler's critical section" which was blocking Bug 1357829 "Make stack walking logic in ThreadStackHelper support windows x64" introduced changes in the 2017-06-15 builds of Nightly. This had the unintended consequence of making 'Nightly hang on exit'; in the Windows 64-bit builds but NOT in the Windows 32-bit builds. This caused Windows to report "Nightly has stopped working", which I am calling a 'Windows Crash', when you closed Nightly. There were no 'Firefox crash reports'. See Bug 1374043 for more detail. As Nightly (just the Win 64-bit version) could not 'close normally', very soon (from 2017-06-16 onwards) we had many bug reports of 'Nightly can't update ...'. David Major [:dmajor] filed Bug 1378442 "Move Win64 profiler hooks to profiler_start" to do the same things as Bug 1372375 - which had caused the regression. This should be in the 2017-07-06 builds. Or possibly in the 2017-07-07 builds - I'm not 100% certain. So, I am repeating the testing that I did previously. This time, 2017-07-05, 2017-07-06 and 2017-07-07 builds, both 32-bit and 64-bit. I am using the 32-bit builds as controls. I imagine many bugs have landed since 2017-06-15. Each build is tested for 3 runs. For the Method, and my reasons, see https://bugzilla.mozilla.org/show_bug.cgi?id=1374043#c11 (and the comments below there in that bug). ====== Nightly-Test-Start-2017-07-05_32bit-NR Version 56.0a1 (2017-07-05) (32-bit) Build ID 20170705030206 Expected result (because Bug 1372375 did not regress the 32-bit build). 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe I will answer the UAC with "Cancel", I'm not sending data in these tests. Nightly closes normally each time. Actual result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. This is as expected. All is OK with the 32-bit build. ------ Nightly-Test-Start-2017-07-05_64bit-NR Version 56.0a1 (2017-07-05) (64-bit) Build ID 20170705030206 Expected result (because Bug 1372375 did regress the 64-bit build and Bug 1378442 is not in this build). 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly then has a 'Windows crash', a "Nightly has stopped working" (because Bug 1372375 did regress the 64-bit build) on all three runs, when closing of Nightly (using 'Menu, File, Exit'). During the 'crash phase', after the 'request for pingsender', there is one 'Nightly firefox.exe' still running, as seen in Windows Task Manager, until I close the 'Windows crash' dialogue. Actual result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly has a 'Windows crash', on all three runs. So, as expected, Bug 1378442 is not in this build and 'Nightly is still broken'. ====== Nightly-Test-Start-2017-07-06_32bit-NR Version 56.0a1 (2017-07-06) (32-bit) Build ID 20170706060058 Expected result (because Bug 1372375 did not regress the 32-bit build). 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. Actual result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. This is as expected. All is OK with the 32-bit build. ------ Nightly-Test-Start-2017-07-06_64bit-NR Version 56.0a1 (2017-07-06) (64-bit) Build ID 20170706060058 Expected result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe If Bug 1378442 is in this build, then like the 32-bit, close normally. If Bug 1378442 is not in this build, then 'Windows crash' on close [like the Version 56.0a1 (2017-07-05) (64-bit) - above]. Actual result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. Like the 32-bit build, when the 'UAC is waiting for me to click "Cancel" (or "Run")', the number of 'Nightly Firefox.exe' drops to 2. After I click "Cancel" both close. This is good news. It looks like Bug 1378442 is in this build, and 'Nightly 64-bit for Windows' is 'back to normal'. ====== Nightly-Test-Start-2017-07-07_32bit-NR Version 56.0a1 (2017-07-07) (32-bit) Build ID 20170707030206 Expected result (because Bug 1372375 did not regress the 32-bit build). 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. Actual result 5 x firefox.exe on 1st run <== saw 6 on all other 1st runs (so far). 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. This is almost as expected. All is OK with the 32-bit build. ------ Nightly-Test-Start-2017-07-07_64bit-NR Version 56.0a1 (2017-07-07) (64-bit) Build ID 20170707030206 Expected result (because Bug 1378442 is in this build, then like the 32-bit, close normally. We saw this in Version 56.0a1 (2017-07-06) (64-bit) but, as I have already downloaded the ZIP versions, I may as well report the test results). 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Actual result 6 x firefox.exe on 1st run, 4 x firefox.exe on 2nd and 3rd runs. All runs 'ask to send data' - pingsender.exe Nightly closes normally each time. ****** So, I think my tests verify that Bug 1378442 fixed the issues caused by Bug 1372375. I believe, if you install a Windows 64-bit build of Nightly [to be on the safe side Version 56.0a1 (2017-07-07) (64-bit) - or newer], then you will no longer have the 'Windows crash' on closing Nightly. I also hope that these newer versions will be able to Update normally. I have not tested this. I hope to update my 'real Nightly' later this week. DJ-Leith

Mark 56 won't fix as we are 1 week from RC.

Thank you DJ-Leith for the extensive testing. Based on these results in comment 33... > Nightly-Test-Start-2017-07-05_64bit-NR > So, as expected, Bug 1378442 is not in this build > and 'Nightly is still broken'. > Nightly-Test-Start-2017-07-06_64bit-NR > This is good news. > It looks like Bug 1378442 is in this build, > and 'Nightly 64-bit for Windows' is 'back to normal'. > Nightly-Test-Start-2017-07-07_64bit-NR > Nightly closes normally each time. > So, I think my tests verify that Bug 1378442 fixed the > issues caused by Bug 1372375. ...I think we can close out this bug. I'm setting the various tracking fields based on Bug 1378442 having landed in 56.