Closed Bug 1004813 Opened 11 years ago Closed 10 years ago

Investigate Windows 8 machines that are still out of action

Categories

(Infrastructure & Operations :: RelOps: General, task)

x86_64
Linux
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: q)

References

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/483] [time=10:00])

I don't know if all are related to the graphics situation. Please debug each case with Q. t-w864-ix-018 t-w864-ix-020 t-w864-ix-021 t-w864-ix-026 t-w864-ix-063 t-w864-ix-070 Video of non-working fakemoni.vbs: https://bugzilla.mozilla.org/attachment.cgi?id=8415450
No longer depends on: 977615
No longer blocks: 974684
No longer blocks: t-w864-ix-070
No longer blocks: t-w864-ix-083
No longer blocks: t-w864-ix-026
Or maybe if we tell Q this bug exists, he'll debug it on his own, since it appears nobody is going to "debug each case with" him.
Assignee: nobody → q
Blocks: 1027101
Q, any estimation of effort needed here, fixing time, etc?
Flags: needinfo?(q)
Q: ping - we could really use the extra capacity provided by these 6 nodes. When will you or Mark be able to start investigating?
Since we're tracking the individual slaves in the Buildduty Queue, I'm handing this bug over to the relops component, and additionally needinfo to amy since its been almost two weeks since my own needinfo, and almost a month since the initial self-assigned nature of the bug.
Component: Buildduty → RelOps
Flags: needinfo?(arich)
Product: Release Engineering → Infrastructure & Operations
QA Contact: bugspam.Callek → arich
Version: unspecified → other
I'll add this to the list of things to work on in the following week.
Flags: needinfo?(arich)
t-w864-ix-003.wintest.releng.scl3.mozilla.com should be added to this list.
Please re-enable t-w864-ix-003 and t-w864-ix-021
Flags: needinfo?(q)
what was the cause of the issue, what was the remedy, is this a systemic issue, random issue, or hardware issue? Is there any progress/plan/etc with the other machines, remedies to prevent more falling into this state, etc.? (p.s. sorry about other bug mini-rant went too fast and saw that one was resolved, and thought I was looking at this one)
Flags: needinfo?(q)
Flags: needinfo?(arich)
Please re-enable t-w864-ix-042
Commenting in each individual machine bugs attached here. The only one with a misconfiguration was 021.
Flags: needinfo?(q)
Flags: needinfo?(arich)
coop: so have you guys been adding nodes back into the pool as they're being fixed?
Flags: needinfo?(coop)
(In reply to Amy Rich [:arich] [:arr] from comment #11) > coop: so have you guys been adding nodes back into the pool as they're being > fixed? No, or at least, not until now. I've re-enabled the slaves from comment #7 and comment #9.
Flags: needinfo?(coop)
Adding t-w864-ix-001 to this list since it had issues in bug 1036627 (no graphics card shows up).
Whiteboard: [time=10:00]
any clue what the status is here? Last comments were from july... and we have (at least) 5 machines out of commission due to this, in one of our largest pained platforms
Flags: needinfo?(q)
Flags: needinfo?(arich)
coop asked that we drop priority in favor of other bugs.
Flags: needinfo?(q)
Flags: needinfo?(arich)
Have we successfully reimaged and put other Win8 slaves back into production since? If so, fine, focus on buying more instead, but I have the feeling that this bug is actually either "we can't reimage Win8 slaves and get working graphics" or "it's pretty nearly totally random whether or not a reimaged Win8 slave will have working graphics and our bus number for figuring out whether or not one does and fixing it if it does not is exactly 1" which doesn't bode especially well for a pile of new ones.
The problem is that these *specific* slaves have a variety of different and hard to diagnose problems.
Tough to tell for sure, but I think the answer to my question is that bug 1011673 was as close as we've come to a success since May, while we've added three more things to this since then.
Whiteboard: [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/457] [time=10:00]
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/457] [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/460] [time=10:00]
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/460] [time=10:00] → [time=10:00]
Whiteboard: [time=10:00] → [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1622] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1622] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1623] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1623] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1628] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1628] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1629] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1629] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1630] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00]
All machines in slavealloc minus: t-w864-ix-002 bug 1079396 - loaned to jrmuizel t-w864-ix-003 bug 1080023 - loaned to markco t-w864-ix-144 Bug 1093488 - Change on-board video setting for moved win8 machines t-w864-ix-148 Bug 1093488 - Change on-board video setting for moved win8 machines t-w864-ix-156 Bug 1093488 - Change on-board video setting for moved win8 machines are back in after a hand check of each.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1630] [kanban:engops:https://kanbanize.com/ctrl_board/6/483] [time=10:00] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/483] [time=10:00]
You need to log in before you can comment on or make changes to this bug.