Closed Bug 643171 Opened 14 years ago Closed 14 years ago

moz2-linux64-slaveNN loads spiking when running fuzzer

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

References

Details

(Whiteboard: [buildslaves])

Attachments

(1 file)

ps-from-slave04.txt
(deleted), text/plain
Details 
14:10 < nagios> [14] moz2-linux64-slave11.build:avg load is CRITICAL: CRITICAL - load average: 688.19, 682.12, 686.14
15:31 < nagios> moz2-linux64-slave11.build:avg load is OK: OK - load average: 0.02, 0.21, 0.13

15:09 < nagios> [20] moz2-linux64-slave04.build:avg load is CRITICAL: CRITICAL - load average: 423.87, 540.26, 443.24

I logged into slave11 around 14:38, but by then its 1-minute average was back down to a normal level.  Trying to login to slave04 now hangs.
Whiteboard: [buildslaves]
Attached file ps-from-slave04.txt (deleted) — 

  

  



    
    

    
  
It looks like it's not processes per se, but threads.  From 'info threads' on a firefox as run from fuzzer:

  332 Thread 1084229952 (LWP 26549)  0x000000391deca0d9 in syscall () from /lib64/libc.so.6
  331 Thread 1094719808 (LWP 26550)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  330 Thread 1105209664 (LWP 26551)  0x000000391ea0a567 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  329 Thread 1115699520 (LWP 26552)  0x000000391dec4896 in *__GI___poll (fds=0x42802960, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:87
  328 Thread 1126189376 (LWP 26553)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  327 Thread 1136679232 (LWP 26554)  0x000000391ea0a567 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  326 Thread 1147169088 (LWP 26555)  0x000000391ea0a567 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  325 Thread 1157658944 (LWP 26556)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  324 Thread 1168148800 (LWP 26557)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  323 Thread 1178638656 (LWP 26558)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  322 Thread 1189128512 (LWP 26559)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  321 Thread 1199618368 (LWP 26560)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  320 Thread 1210108224 (LWP 26561)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  319 Thread 1220598080 (LWP 26562)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  318 Thread 1231087936 (LWP 26563)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  317 Thread 1241577792 (LWP 26564)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  316 Thread 1252067648 (LWP 26566)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  315 Thread 1262557504 (LWP 26568)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  314 Thread 1273047360 (LWP 26569)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  313 Thread 1283537216 (LWP 26590)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  312 Thread 1294027072 (LWP 26591)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  311 Thread 1304516928 (LWP 26592)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  310 Thread 1315006784 (LWP 26593)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  309 Thread 1325496640 (LWP 26594)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  308 Thread 1335986496 (LWP 26595)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  307 Thread 1346476352 (LWP 26596)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  306 Thread 1356966208 (LWP 26597)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  305 Thread 1367456064 (LWP 26598)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  304 Thread 1377945920 (LWP 26599)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  303 Thread 1388435776 (LWP 26600)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  302 Thread 1398925632 (LWP 26601)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  301 Thread 1409415488 (LWP 26602)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  300 Thread 1419905344 (LWP 26603)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  299 Thread 1430395200 (LWP 26604)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  298 Thread 1440885056 (LWP 26605)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  297 Thread 1451374912 (LWP 26606)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  296 Thread 1461864768 (LWP 26607)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  295 Thread 1472354624 (LWP 26608)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  294 Thread 1482844480 (LWP 26609)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  293 Thread 1493334336 (LWP 26610)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  292 Thread 1503824192 (LWP 26611)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  291 Thread 1514314048 (LWP 26612)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  290 Thread 1524803904 (LWP 26613)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  289 Thread 1535293760 (LWP 26614)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  288 Thread 1545783616 (LWP 26615)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  287 Thread 1556273472 (LWP 26616)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  286 Thread 1566763328 (LWP 26617)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  285 Thread 1577253184 (LWP 26618)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  284 Thread 1587743040 (LWP 26619)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  283 Thread 1598232896 (LWP 26620)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  282 Thread 1608722752 (LWP 26621)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  281 Thread 1619212608 (LWP 26622)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  280 Thread 1629702464 (LWP 26623)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  279 Thread 1640192320 (LWP 26624)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  278 Thread 1650682176 (LWP 26625)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  277 Thread 1661172032 (LWP 26626)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  276 Thread 1671661888 (LWP 26627)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  275 Thread 1682151744 (LWP 26628)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  274 Thread 1692641600 (LWP 26629)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  273 Thread 1703131456 (LWP 26630)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  272 Thread 1713621312 (LWP 26631)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  271 Thread 1724111168 (LWP 26632)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  270 Thread 1734601024 (LWP 26633)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  269 Thread 1745090880 (LWP 26634)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  268 Thread 1755580736 (LWP 26635)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  267 Thread 1766070592 (LWP 26636)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  266 Thread 1776560448 (LWP 26637)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  265 Thread 1787050304 (LWP 26638)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  264 Thread 1797540160 (LWP 26639)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  263 Thread 1808030016 (LWP 26640)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  262 Thread 1818519872 (LWP 26641)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  261 Thread 1829009728 (LWP 26642)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  260 Thread 1839499584 (LWP 26643)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  259 Thread 1849989440 (LWP 26644)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  258 Thread 1860479296 (LWP 26645)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  257 Thread 1870969152 (LWP 26646)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  256 Thread 1881459008 (LWP 26647)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  255 Thread 1891948864 (LWP 26648)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  254 Thread 1902438720 (LWP 26649)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  253 Thread 1912928576 (LWP 26650)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  252 Thread 1923418432 (LWP 26651)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  251 Thread 1933908288 (LWP 26652)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  250 Thread 1944398144 (LWP 26653)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  249 Thread 1954888000 (LWP 26654)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  248 Thread 1965377856 (LWP 26655)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  247 Thread 1975867712 (LWP 26656)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  246 Thread 1986357568 (LWP 26657)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  245 Thread 1996847424 (LWP 26658)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  244 Thread 2007337280 (LWP 26659)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  243 Thread 2017827136 (LWP 26660)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  242 Thread 2028316992 (LWP 26661)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  241 Thread 2038806848 (LWP 26662)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  240 Thread 2049296704 (LWP 26663)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  239 Thread 2059786560 (LWP 26664)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  238 Thread 2070276416 (LWP 26665)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  237 Thread 2080766272 (LWP 26666)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  236 Thread 2091256128 (LWP 26667)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  235 Thread 2101745984 (LWP 26668)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  234 Thread 2112235840 (LWP 26669)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  233 Thread 2122725696 (LWP 26670)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  232 Thread 2133215552 (LWP 26671)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  231 Thread 2143705408 (LWP 26672)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  230 Thread 46912815302976 (LWP 26673)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  229 Thread 46912825792832 (LWP 26674)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  228 Thread 46912836282688 (LWP 26675)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  227 Thread 46912846772544 (LWP 26676)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  226 Thread 46912857262400 (LWP 26677)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  225 Thread 46912867752256 (LWP 26678)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  224 Thread 46912878242112 (LWP 26679)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  223 Thread 46912888731968 (LWP 26680)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  222 Thread 46912899221824 (LWP 26681)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  221 Thread 46912909711680 (LWP 26682)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  220 Thread 46912920201536 (LWP 26683)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  219 Thread 46912930691392 (LWP 26684)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  218 Thread 46912941181248 (LWP 26685)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  217 Thread 46912951671104 (LWP 26686)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  216 Thread 46912962160960 (LWP 26687)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  215 Thread 46912972650816 (LWP 26688)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  214 Thread 46912983140672 (LWP 26689)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  213 Thread 46912993630528 (LWP 26690)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  212 Thread 46913004120384 (LWP 26691)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  211 Thread 46913014610240 (LWP 26692)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  210 Thread 46913025100096 (LWP 26693)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  209 Thread 46913035589952 (LWP 26694)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  208 Thread 46913046079808 (LWP 26695)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  207 Thread 46913056569664 (LWP 26696)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  206 Thread 46913067059520 (LWP 26697)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  205 Thread 46913077549376 (LWP 26698)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  204 Thread 46913088039232 (LWP 26699)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  203 Thread 46913098529088 (LWP 26700)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  202 Thread 46913109018944 (LWP 26701)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  201 Thread 46913119508800 (LWP 26702)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  200 Thread 46913129998656 (LWP 26703)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  199 Thread 46913140488512 (LWP 26704)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  198 Thread 46913150978368 (LWP 26705)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  197 Thread 46913161468224 (LWP 26706)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  196 Thread 46913171958080 (LWP 26707)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  195 Thread 46913182447936 (LWP 26708)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  194 Thread 46913192937792 (LWP 26709)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  193 Thread 46913203427648 (LWP 26710)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  192 Thread 46913213917504 (LWP 26711)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  191 Thread 46913224407360 (LWP 26712)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  190 Thread 46913234897216 (LWP 26713)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  189 Thread 46913245387072 (LWP 26714)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  188 Thread 46913255876928 (LWP 26715)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  187 Thread 46913266366784 (LWP 26716)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  186 Thread 46913276856640 (LWP 26717)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  185 Thread 46913287346496 (LWP 26718)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  184 Thread 46913297836352 (LWP 26719)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  183 Thread 46913308326208 (LWP 26720)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  182 Thread 46913318816064 (LWP 26721)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  181 Thread 46913329305920 (LWP 26722)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  180 Thread 46913339795776 (LWP 26723)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  179 Thread 46913350285632 (LWP 26724)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  178 Thread 46913360775488 (LWP 26725)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  177 Thread 46913371265344 (LWP 26726)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  176 Thread 46913381755200 (LWP 26727)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  175 Thread 46913392245056 (LWP 26728)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  174 Thread 46913402734912 (LWP 26729)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  173 Thread 46913413224768 (LWP 26730)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  172 Thread 46913423714624 (LWP 26731)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  171 Thread 46913434204480 (LWP 26732)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  170 Thread 46913444694336 (LWP 26733)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  169 Thread 46913455184192 (LWP 26734)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  168 Thread 46913465674048 (LWP 26735)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  167 Thread 46913476163904 (LWP 26736)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  166 Thread 46913486653760 (LWP 26737)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  165 Thread 46913497143616 (LWP 26738)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  164 Thread 46913507633472 (LWP 26739)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  163 Thread 46913518123328 (LWP 26740)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  162 Thread 46913528613184 (LWP 26741)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  161 Thread 46913539103040 (LWP 26742)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  160 Thread 46913549592896 (LWP 26743)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  159 Thread 46913560082752 (LWP 26744)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  158 Thread 46913570572608 (LWP 26745)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  157 Thread 46913581062464 (LWP 26746)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  156 Thread 46913591552320 (LWP 26747)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  155 Thread 46913602042176 (LWP 26748)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  154 Thread 46913612532032 (LWP 26749)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  153 Thread 46913623021888 (LWP 26750)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  152 Thread 46913633511744 (LWP 26751)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  151 Thread 46913644001600 (LWP 26752)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  150 Thread 46913654491456 (LWP 26753)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  149 Thread 46913664981312 (LWP 26754)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  148 Thread 46913675471168 (LWP 26755)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  147 Thread 46913685961024 (LWP 26756)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  146 Thread 46913696450880 (LWP 26757)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  145 Thread 46913706940736 (LWP 26758)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  144 Thread 46913717430592 (LWP 26759)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  143 Thread 46913727920448 (LWP 26760)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  142 Thread 46913738410304 (LWP 26761)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  141 Thread 46913748900160 (LWP 26762)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  140 Thread 46913759390016 (LWP 26763)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  139 Thread 46913769879872 (LWP 26764)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  138 Thread 46913780369728 (LWP 26765)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  137 Thread 46913790859584 (LWP 26766)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  136 Thread 46913801349440 (LWP 26767)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  135 Thread 46913811839296 (LWP 26768)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  134 Thread 46913822329152 (LWP 26769)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  133 Thread 46913832819008 (LWP 26770)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  132 Thread 46913843308864 (LWP 26771)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  131 Thread 46913853798720 (LWP 26772)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  130 Thread 46913864288576 (LWP 26773)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  129 Thread 46913874778432 (LWP 26774)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  128 Thread 46913885268288 (LWP 26775)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  127 Thread 46913895758144 (LWP 26776)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  126 Thread 46913906248000 (LWP 26777)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  125 Thread 46913916737856 (LWP 26778)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  124 Thread 46913927227712 (LWP 26779)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  123 Thread 46913937717568 (LWP 26780)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  122 Thread 46913948207424 (LWP 26781)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  121 Thread 46913958697280 (LWP 26782)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  120 Thread 46913969187136 (LWP 26783)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  119 Thread 46913979676992 (LWP 26784)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  118 Thread 46913990166848 (LWP 26785)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  117 Thread 46914000656704 (LWP 26786)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  116 Thread 46914011146560 (LWP 26787)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  115 Thread 46914021636416 (LWP 26788)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  114 Thread 46914032126272 (LWP 26789)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  113 Thread 46914042616128 (LWP 26790)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  112 Thread 46914053105984 (LWP 26791)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  111 Thread 46914063595840 (LWP 26792)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  110 Thread 46914074085696 (LWP 26793)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  109 Thread 46914084575552 (LWP 26794)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  108 Thread 46914095065408 (LWP 26795)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  107 Thread 46914105555264 (LWP 26796)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  106 Thread 46914116045120 (LWP 26797)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  105 Thread 46914126534976 (LWP 26798)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  104 Thread 46914137024832 (LWP 26799)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  103 Thread 46914147514688 (LWP 26800)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  102 Thread 46914158004544 (LWP 26801)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  101 Thread 46914168494400 (LWP 26802)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  100 Thread 46914178984256 (LWP 26803)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  99 Thread 46914189474112 (LWP 26804)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  98 Thread 46914199963968 (LWP 26805)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  97 Thread 46914210453824 (LWP 26806)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  96 Thread 46914220943680 (LWP 26807)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  95 Thread 46914231433536 (LWP 26808)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  94 Thread 46914241923392 (LWP 26809)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  93 Thread 46914252413248 (LWP 26810)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  92 Thread 46914262903104 (LWP 26811)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  91 Thread 46914273392960 (LWP 26812)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  90 Thread 46914283882816 (LWP 26813)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  89 Thread 46914294372672 (LWP 26814)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  88 Thread 46914304862528 (LWP 26815)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  87 Thread 46914315352384 (LWP 26816)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  86 Thread 46914325842240 (LWP 26817)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  85 Thread 46914336332096 (LWP 26818)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  84 Thread 46914346821952 (LWP 26819)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  83 Thread 46914357311808 (LWP 26820)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  82 Thread 46914367801664 (LWP 26821)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  81 Thread 46914378291520 (LWP 26822)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  80 Thread 46914388781376 (LWP 26823)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  79 Thread 46914399271232 (LWP 26824)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  78 Thread 46914409761088 (LWP 26825)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  77 Thread 46914420250944 (LWP 26826)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  76 Thread 46914430740800 (LWP 26827)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  75 Thread 46914441230656 (LWP 26828)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  74 Thread 46914451720512 (LWP 26829)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  73 Thread 46914462210368 (LWP 26830)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  72 Thread 46914472700224 (LWP 26831)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  71 Thread 46914483190080 (LWP 26832)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  70 Thread 46914493679936 (LWP 26833)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  69 Thread 46914504169792 (LWP 26834)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  68 Thread 46914514659648 (LWP 26835)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  67 Thread 46914525149504 (LWP 26836)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  66 Thread 46914535639360 (LWP 26837)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  65 Thread 46914546129216 (LWP 26838)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  64 Thread 46914556619072 (LWP 26839)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  63 Thread 46914567108928 (LWP 26840)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  62 Thread 46914577598784 (LWP 26841)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  61 Thread 46914588088640 (LWP 26842)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  60 Thread 46914598578496 (LWP 26843)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  59 Thread 46914609068352 (LWP 26844)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  58 Thread 46914619558208 (LWP 26845)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  57 Thread 46914630048064 (LWP 26846)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  56 Thread 46914640537920 (LWP 26847)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  55 Thread 46914651027776 (LWP 26848)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  54 Thread 46914661517632 (LWP 26849)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  53 Thread 46914672007488 (LWP 26850)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  52 Thread 46914682497344 (LWP 26851)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  51 Thread 46914692987200 (LWP 26852)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  50 Thread 46914703477056 (LWP 26853)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  49 Thread 46914713966912 (LWP 26854)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  48 Thread 46914724456768 (LWP 26855)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  47 Thread 46914734946624 (LWP 26856)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  46 Thread 46914745436480 (LWP 26857)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  45 Thread 46914766416192 (LWP 26858)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  44 Thread 46914776906048 (LWP 26859)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  43 Thread 46914787395904 (LWP 26860)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  42 Thread 46914755926336 (LWP 26861)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  41 Thread 46914797885760 (LWP 26862)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  40 Thread 46914808375616 (LWP 26863)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  39 Thread 46914818865472 (LWP 26864)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  38 Thread 46914829355328 (LWP 26865)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  37 Thread 46914839845184 (LWP 26866)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  36 Thread 46914850335040 (LWP 26867)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  35 Thread 46914860824896 (LWP 26868)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  34 Thread 46914871314752 (LWP 26869)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  33 Thread 46914881804608 (LWP 26870)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  32 Thread 46914892294464 (LWP 26871)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  31 Thread 46914902784320 (LWP 26872)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  30 Thread 46914913274176 (LWP 26873)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  29 Thread 46914923764032 (LWP 26874)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  28 Thread 46914934253888 (LWP 26875)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  27 Thread 46914944743744 (LWP 26876)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  26 Thread 46914955233600 (LWP 26877)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  25 Thread 46914965723456 (LWP 26878)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  24 Thread 46914976213312 (LWP 26879)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  23 Thread 46914986703168 (LWP 26880)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22 Thread 46914997193024 (LWP 26881)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 Thread 46915007682880 (LWP 26882)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 Thread 46915018172736 (LWP 26883)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 46915028662592 (LWP 26884)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 46915039152448 (LWP 26885)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 Thread 46915049642304 (LWP 26886)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  16 Thread 46915060132160 (LWP 26887)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 Thread 46915070622016 (LWP 26888)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14 Thread 46915081111872 (LWP 26889)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13 Thread 46915091601728 (LWP 26890)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12 Thread 46915102091584 (LWP 26891)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11 Thread 46915112581440 (LWP 26892)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 Thread 46915123071296 (LWP 26893)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 Thread 46915133561152 (LWP 26894)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 Thread 46915144051008 (LWP 26895)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 46915154540864 (LWP 26896)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 46915165030720 (LWP 26897)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 46915175520576 (LWP 26898)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 Thread 46915186010432 (LWP 26899)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 46915196500288 (LWP 26900)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 Thread 46915206990144 (LWP 26901)  0x000000391ea0a376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  1 Thread 46912587122160 (LWP 26548)  0x00002aaaab202c1a in std::vector<int, std::allocator<int> >::~vector () from /builds/slave/fuzzer-linux64/build/dist/bin/libxul.so

Even for firefox, that's a *lot* of threads.  Presumably they're all on the same cond var, and that's getting notified a lot, so we're seeing a 250-way thundering herd?

  

  



    
    

    
  
We should copy

  

  



    
    

    
  
sorry, partial comment.

Jesse: any thoughts on what this could be, or where we should send the bug for further diagnosis?

  

  



    
    

    
  
I think creating lots of <audio> elements can cause lots of threads to be created.  That might be what's happening here.

Can you turn off the load-average nagios warning for machines that are running the DOM fuzzer?

  

  


Summary: moz2-linux64-slaveNN loads spiking → moz2-linux64-slaveNN loads spiking when running fuzzer

    
    

    
  
The warnings aren't the problem - the load average is!

This isn't simply 300+ threads - it's 300+ *runnable* threads, so there's massive CPU contention going on here.

It's impossible to login to the slaves while they're running these tests, which impairs our ability to monitor them.

  

  



    
    

    
  
A single process with a 300 threads can make the system unusable? Sounds like the scheduler isn't handling this case well!

  

  



    
    

    
  
Not really sure what we can do about this...

Does the machine recover after this?  Does it take an inordinately long time to finish the job?

  

  



    
    

    
  
Jesse: I disagree, and don't find this at all surprising, but I won't reiterate my arguments.

Catlee: yes, the machine recovers afterward.  I don't know whether it ends up taking too long or not.

At any rate, this is a new phenomenon, and it demonstrates that Firefox could, in some circumstances, lock a user's system up with a thread bomb.

  

  



    
    

    
  
Bug 592833 should reduce the thread usage of <audio> elements. Chris Pearce might know which type of <audio> thread is causing the most trouble. See also bug 628967.

  

  



    
    

    
  
(In reply to comment #5)
> Can you turn off the load-average nagios warning for machines that are running
> the DOM fuzzer?

No, we cant do that. Those machines are shared pool machines, and we *need* nagios monitoring on them all the rest of the time. 


Given the super-high load, this makes it difficult to kill jobs if a fuzzer goes out of control. 

Marking as blocked on bug#592833 which is reducing the number of audio threads, and we hope will improve the situation.


Nothing for RelEng to do here, but keeping open to track.

  

  


Depends on: 592833

    
    

    
  
(In reply to comment #11)
> Given the super-high load, this makes it difficult to kill jobs if a fuzzer
> goes out of control. 

It's worth noting that this happened this morning and catlee had to resort to heroic measures (closing file descriptors manually via debugger on the master) to work around it.

  

  



    
    

    
  
Currently on Linux there's 4 threads per (ogg) audio element. bug 592833 should drop that down to 3/element, and bug 623444 should remove 1 more per element. We're targeting Firefox 6 for those refactorings (which is why we want a project branch, a la bug 642056 to land these changes in...).

Even then, we'd still likely have at least 1, probably two threads per audio element, so we'd still be vulnerable to this if you created enough audio elements - you'd just need to create more.

There's a few things you can do to mitigate the load.
* Stop fuzzing audio... :(
* Set the src of audio elements to "" and run a GC after finishing a test (we do this in the mochitestests in content/media/tests). This will shutdown the threads, and ensure they're GC'd.
* Make all audio elements have a preload="none" attribute, until such a time as we do the dethreadification. This means we won't create decoders (and threads) unless the elements actually play.

(In reply to comment #2)
> It looks like it's not processes per se, but threads.  From 'info threads' on a
> firefox as run from fuzzer:
> 
>   332 Thread 1084229952 (LWP 26549)  0x000000391deca0d9 in syscall () from
> /lib64/libc.so.6
[...]
>   2 Thread 46915206990144 (LWP 26901)  0x000000391ea0a376 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
>   1 Thread 46912587122160 (LWP 26548)  0x00002aaaab202c1a in std::vector<int,
> std::allocator<int> >::~vector () from
> /builds/slave/fuzzer-linux64/build/dist/bin/libxul.so
> 
> Even for firefox, that's a *lot* of threads.  Presumably they're all on the
> same cond var, and that's getting notified a lot, so we're seeing a 250-way
> thundering herd?

Is there anyway to get thread stacks from as many of these threads as possible? IIRC the only condvar that's shared across _all_ media elements is the media cache lock. It would be good to know if we need finer grain locking/notification on the media cache.

  

  



    
    

    
  
I can grab a stack trace next time I see this failure.

  

  


Assignee: nobody → dustin

    
    

    
  
bug 624260 might make that interesting to spot. Turns out moz2-linux64-slave06 is in this state right now though.

  

  



    
    

    
  
Sadly, you have to already be logged in when the failure starts to catch it - at least in my experience.

  

  



    
    

    
  
That would be great, thanks!

  

  



    
    

    
  
I used the VI interface to get a console on this VM. There was a lot of oomkill output there, refreshed every second or so, and couldn't get a login prompt or any useful info.

Jesse, are any of the suggestions in comment #13 for modifying the fuzzer palatable ? Until we get bug 577154 resolved it hurts to lose machines out of a small pool.

  

  



    
    

    
  
Forgot to say that moz2-linux64-slave06 got reset.

  

  



    
    

    
  
I pushed a change to the fuzzer to make it generate <audio> and <video> elements less frequently. Let's see if that make these occurrences less frequent.

  

  



    
    

    
  
It completely eliminated the load spikes.  Thanks!

  

  


Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED

    
    

    
  
Interesting. I only expected it to reduce the occurrences by 96% ;)

  

  


Product: mozilla.org → Release Engineering

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: