Closed
Bug 1122975
Opened 10 years ago
Closed 10 years ago
[tracking] problematic windows slaves - make check timeouts, LNK1318: Unexpected PDB error, mach build timeout
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jlund, Assigned: jlund)
References
Details
there are a few intermittents windows jobs that could have many root causes however there are a few common slaves between them all that are appearing more than others.
namely:
b-2008-ix-0003
b-2008-ix-0007
b-2008-ix-0012
b-2008-ix-0013
b-2008-ix-0014
releng + relops should investigate these slaves and rule out 1) if there is something individually different about them and then 2) if our windows pool as a whole isn't meeting requirements.
Comment 1•10 years ago
|
||
I rage-disabled 7 and 12/13/14 this morning, though conveniently for me I hadn't yet gotten around to filing an "investigate" bug for them :)
Assignee | ||
Comment 2•10 years ago
|
||
Updated•10 years ago
|
Blocks: b-2008-ix-0003
Comment 4•10 years ago
|
||
b-2008-ix-0003 offended me too, so I plucked it out as well.
Updated•10 years ago
|
Blocks: b-2008-ix-0001
Assignee | ||
Comment 5•10 years ago
|
||
bugs 1115490, 1110236, and 1093664 all hinted at RAM issues in their own unique way.
after poking the handful of bad offenders from this bug and comparing to a sample size of good slaves, I think I have a good idea of what's going on:
'known good slaves':
cltbld@B-2008-IX-0166 ~
cltbld@B-2008-IX-0106 ~
cltbld@B-2008-IX-0115 ~
Total Physical Memory: 8,183 MB
'all slaves from this bug':
cltbld@B-2008-IX-0014 ~
cltbld@B-2008-IX-0013 ~
cltbld@B-2008-IX-0001 ~
cltbld@B-2008-IX-0003 ~
cltbld@B-2008-IX-0007 ~
cltbld@B-2008-IX-0012 ~
Total Physical Memory: 4,087 MB
seems like not all our slaves have the same amount of RAM and 4gb is just not good enough. next steps are figuring out which slaves have 4gb and which have 8gb. I think bumping RAM of these B-2008-IX machines will be hard to get budget/time approval for, particularly with our Q1 efforts of moving win builders to the cloud. If there are only a few 4gb slaves, I would like to propose the idea of disabling the bad ones and waiting for results of aws efforts.
I'll follow up with getting numbers for which slaves only have 4gb
Updated•10 years ago
|
Blocks: b-2008-ix-0002
Updated•10 years ago
|
Blocks: b-2008-ix-0004
Assignee | ||
Updated•10 years ago
|
Assignee | ||
Comment 6•10 years ago
|
||
just went through the b-2008 machines. automating with windows + ssh wasn't behaving so it ended up taking me a while to hack a tmux script together.
good news is there isn't many machines (12) with only 4gb of RAM and sheriffs pretty much discovered them all for this bug already.
I've made sure that all the 4gb ones are disabled for now. I'll follow up with a bug to request a RAM bump for the 12 4gb ones.
ftr here is the full list (surprise surprise, all the 4gb were the first numbered slaves which makes me think they all came from a different pool together):
> cat windows_ram_4gb
B-2008-IX-0001 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0002 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0003 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0004 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0005 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0007 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0007 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0009 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0010 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0011 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0012 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0013 ~ Total Physical Memory: 4,087 MB
B-2008-IX-0014 ~ Total Physical Memory: 4,087 MB
> cat windows_ram_8gb
B-2008-IX-0008 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0015 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0016 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0017 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0065 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0066 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0067 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0068 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0069 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0070 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0071 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0072 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0073 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0074 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0075 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0075 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0076 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0077 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0078 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0079 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0080 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0082 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0083 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0084 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0085 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0086 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0087 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0088 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0090 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0091 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0092 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0093 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0094 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0095 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0096 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0097 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0098 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0099 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0100 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0101 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0102 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0103 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0104 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0105 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0106 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0107 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0108 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0109 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0110 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0111 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0112 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0113 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0114 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0115 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0116 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0117 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0118 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0119 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0120 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0121 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0122 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0123 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0124 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0125 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0126 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0127 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0128 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0129 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0130 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0131 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0132 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0133 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0134 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0135 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0136 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0137 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0138 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0139 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0140 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0141 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0142 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0143 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0145 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0146 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0147 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0148 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0149 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0150 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0152 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0153 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0154 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0155 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0156 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0157 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0158 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0161 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0162 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0163 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0164 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0165 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0166 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0167 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0168 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0169 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0170 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0171 ~ Total Physical Memory: 8,183 MB
B-2008-IX-0172 ~ Total Physical Memory: 8,183 MB
Assignee: nobody → jlund
Status: NEW → ASSIGNED
Updated•10 years ago
|
Blocks: b-2008-ix-0051
Assignee | ||
Comment 8•10 years ago
|
||
All dep slaves have 8gb now. It is getting late in my day. I'll enable them in the morning so I can keep an eye on them.
Assignee | ||
Comment 9•10 years ago
|
||
there were some slaves that didn't end up getting RAM upgrades. They should all have 8gb now though. Going to resolve this tracker as I think the thing that linked all the blocking bugs was around memory constraints.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Product: Release Engineering → Infrastructure & Operations
Updated•5 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•