I noticed that when looking at a series of Hf builds on inbound, that we were flapping between 7 and 8 hazards. When I looked at 2 adjacent ones, https://tbpl.mozilla.org/php/getParsedLog.php?id=31829962&tree=Mozilla-Inbound has 7 hazards and https://tbpl.mozilla.org/php/getParsedLog.php?id=31831051&tree=Mozilla-Inbound has 8. Diffing the two hazards files (7) http://ftp.mozilla.org//pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-br-haz/20131211141826/hazards.txt.gz (8) http://ftp.mozilla.org//pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-br-haz/20131211143432/hazards.txt.gz shows the added hazard to be: Function 'void mozilla::dom::workers::XMLHttpRequest::Send(JSObject*, mozilla::ErrorResult*)' has unrooted 'aBody' of type 'JSObject*' live across GC call 'JSContext* mozilla::dom::workers::WorkerPrivate::GetJSContext() const' at dom/workers/XMLHttpRequest.cpp:1971 dom/workers/XMLHttpRequest.cpp:1971: Call(1,2, cx := this*.mWorkerPrivate*.GetJSContext()) dom/workers/XMLHttpRequest.cpp:1973: Call(2,3, __temp_1 := __builtin_expect(null(aBody*),0)) GC Function: JSContext* mozilla::dom::workers::WorkerPrivate::GetJSContext() const void mozilla::dom::workers::WorkerPrivate::AssertIsOnWorkerThread() const FieldCall: nsIEventTarget.IsOnCurrentThread Note that this particular hazard is a false positive that will go away with bug 948753. I looked at the annotations.js and computeCallgraph.js files for these two revisions, and they are the same. (Neither has the bug 948753 patch, either.) The hazards are listed in a different order too, but that's expected -- the hazards are produced by multiple jobs running in parallel, with their final outputs concatenated together. I really should make those deterministic. There is no known way that processing order can affect the output. Perhaps there is an unknown way, though. :(

Hm. When looking at this, I am getting strange output for the contents of that method. In the runs that report the hazard, I see the call to WorkerPrivate::GetJSContext(), which it thinks can GC. (The above reason is actually bogus, and fixed, but there's another bogus reason hiding behind that one.) I kept a copy of a run that doesn't show the hazard. It doesn't think XMLHttpRequest::Send(JSObject*,...) calls anything but itself. Here's the xdbfind output: block: void mozilla::dom::workers::XMLHttpRequest::Send(JSObject*, mozilla::ErrorResult*) command: /home/sfink/src/MI-upstream/obj-analyzed/dom/bindings: c++ -o 'UnifiedBindings29.o' '-c' '-I../../dist/stl_wrappers' '-I../../dist/system_wrappers' '-include' '/home/sfink/src/MI-upstream/config/gcc_hidden.h' '-DOS_POSIX=1' '-DOS_LINUX=1' '-DMOZ_GLUE_IN_PROGRAM' '-DMOZILLA_INTERNAL_API' '-DIMPL_LIBXUL' '-DSTATIC_EXPORTABLE_JS_API' '-DNO_NSPR_10_SUPPORT' '-I/home/sfink/src/MI-upstream/dom/bindings' '-I.' '-I/home/sfink/src/MI-upstream/content/base/src' '-I/home/sfink/src/MI-upstream/content/canvas/src' '-I/home/sfink/src/MI-upstream/content/events/src' '-I/home/sfink/src/MI-upstream/content/html/content/src' '-I/home/sfink/src/MI-upstream/content/html/document/src' '-I/home/sfink/src/MI-upstream/content/media/webaudio' '-I/home/sfink/src/MI-upstream/content/media/webspeech/recognition' '-I/home/sfink/src/MI-upstream/content/svg/content/src' '-I/home/sfink/src/MI-upstream/content/xbl/src' '-I/home/sfink/src/MI-upstream/content/xml/content/src' '-I/home/sfink/src/MI-upstream/content/xslt/src/base' '-I/home/sfink/src/MI-upstream/content/xslt/src/xpath' '-I/home/sfink/src/MI-upstream/content/xul/content/src' '-I/home/sfink/src/MI-upstream/content/xul/document/src' '-I/home/sfink/src/MI-upstream/dom/base' '-I/home/sfink/src/MI-upstream/dom/battery' '-I/home/sfink/src/MI-upstream/dom/bluetooth' '-I/home/sfink/src/MI-upstream/dom/camera' '-I/home/sfink/src/MI-upstream/dom/file' '-I/home/sfink/src/MI-upstream/dom/indexedDB' '-I/home/sfink/src/MI-upstream/dom/src/geolocation' '-I/home/sfink/src/MI-upstream/dom/workers' '-I/home/sfink/src/MI-upstream/js/ipc' '-I/home/sfink/src/MI-upstream/js/xpconnect/src' '-I/home/sfink/src/MI-upstream/js/xpconnect/wrappers' '-I/home/sfink/src/MI-upstream/layout/style' '-I/home/sfink/src/MI-upstream/layout/xul/tree' '-I/home/sfink/src/MI-upstream/media/mtransport' '-I/home/sfink/src/MI-upstream/media/webrtc/signaling/src/common/time_profiling' '-I/home/sfink/src/MI-upstream/media/webrtc/signaling/src/peerconnection' '-I/home/sfink/src/MI-upstream/ipc/chromium/src' '-I/home/sfink/src/MI-upstream/ipc/glue' '-I/home/sfink/src/MI-upstream/obj-analyzed/ipc/ipdl/_ipdlheaders' '-I../../dist/include' '-I/home/sfink/src/MI-upstream/obj-analyzed/dist/include/nspr' '-I/home/sfink/src/MI-upstream/obj-analyzed/dist/include/nss' '-fPIC' '-DMOZILLA_CLIENT' '-include' '../../mozilla-config.h' '-MD' '-MP' '-MF' '.deps/UnifiedBindings29.o.pp' '-Wall' '-Wpointer-arith' '-Woverloaded-virtual' '-Werror=return-type' '-Wtype-limits' '-Wempty-body' '-Wsign-compare' '-Wno-invalid-offsetof' '-Wcast-align' '-fno-exceptions' '-fno-strict-aliasing' '-fno-rtti' '-fno-exceptions' '-fno-math-errno' '-std=gnu++0x' '-pthread' '-pipe' '-DDEBUG' '-D_DEBUG' '-DTRACING' '-g' '-Os' '-freorder-blocks' '-fno-omit-frame-pointer' '-Wno-uninitialized' '/home/sfink/src/MI-upstream/obj-analyzed/dom/bindings/UnifiedBindings29.cpp' begin: "/home/sfink/src/MI-upstream/obj-analyzed/dist/include/mozilla/dom/workers/bindings/XMLHttpRequest.h:170" end: "/home/sfink/src/MI-upstream/obj-analyzed/dist/include/mozilla/dom/workers/bindings/XMLHttpRequest.h:173" define: Send : (void,mozilla::dom::workers::XMLHttpRequest)(JSObject*,mozilla::ErrorResult*) define: this : mozilla::dom::workers::XMLHttpRequest* define: aBody : JSObject* define: aRv : mozilla::ErrorResult* pentry: 1 pexit: 2 point 1: "/home/sfink/src/MI-upstream/obj-analyzed/dist/include/mozilla/dom/workers/bindings/XMLHttpRequest.h:172" point 2: "/home/sfink/src/MI-upstream/obj-analyzed/dist/include/mozilla/dom/workers/bindings/XMLHttpRequest.h:173" Call(1,2, this*.Send(aBody*,aRv*)) =============================================================================== Here's the build_xgill.log output for that file: Command: /home/sfink/src/MI-upstream/obj-analyzed/dom/workers /bin/c++ -DXGILL_PLUGIN -fplugin=/home/sfink/src/sixgill/scripts/wrap_gcc/xgill.so -fplugin-arg-xgill-gcc=/bin/gcc -fplugin-arg-xgill-basedir=obj-analyzed -fplugin-arg-xgill-remote=127.0.0.1:49996 -fplugin-arg-xgill-log=/home/sfink/Analysis/browser/work/current.browser/build_xgill.log -o 'XMLHttpRequest.o' '-c' '-I../../dist/stl_wrappers' '-I../../dist/system_wrappers' '-include' '/home/sfink/src/MI-upstream/config/gcc_hidden.h' '-DMOZ_GLUE_IN_PROGRAM' '-DMOZILLA_INTERNAL_API' '-DIMPL_LIBXUL' '-DSTATIC_EXPORTABLE_JS_API' '-DNO_NSPR_10_SUPPORT' '-I/home/sfink/src/MI-upstream/dom/workers' '-I.' '-I/home/sfink/src/MI-upstream/dom/workers/../base' '-I/home/sfink/src/MI-upstream/dom/workers/../system' '-I/home/sfink/src/MI-upstream/content/base/src' '-I/home/sfink/src/MI-upstream/content/events/src' '-I/home/sfink/src/MI-upstream/xpcom/build' '-I../../dist/include' '-I/home/sfink/src/MI-upstream/obj-analyzed/dist/include/nspr' '-I/home/sfink/src/MI-upstream/obj-analyzed/dist/include/nss' '-fPIC' '-DMOZILLA_CLIENT' '-include' '../../mozilla-config.h' '-MD' '-MP' '-MF' '.deps/XMLHttpRequest.o.pp' '-Wall' '-Wpointer-arith' '-Woverloaded-virtual' '-Werror=return-type' '-Wtype-limits' '-Wempty-body' '-Wsign-compare' '-Wno-invalid-offsetof' '-Wcast-align' '-fno-exceptions' '-fno-strict-aliasing' '-fno-rtti' '-fno-exceptions' '-fno-math-errno' '-std=gnu++0x' '-pthread' '-pipe' '-DDEBUG' '-D_DEBUG' '-DTRACING' '-g' '-Os' '-freorder-blocks' '-fno-omit-frame-pointer' '/home/sfink/src/MI-upstream/dom/workers/XMLHttpRequest.cpp' ERROR: Conflicting types for lhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for rhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for __temp_2: nsDefaultStringComparator nsDefaultCStringComparator ERROR: Conflicting types for lhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for rhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for __temp_2: nsDefaultStringComparator nsDefaultCStringComparator ERROR: Conflicting types for lhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for rhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for __temp_2: nsDefaultStringComparator nsDefaultCStringComparator ERROR: Conflicting types for lhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for rhs: nsAString_internal* nsACString_internal* ERROR: Conflicting types for __temp_2: nsDefaultStringComparator nsDefaultCStringComparator ERROR: Conflicting types for __temp_1: uint8 class JS::MutableHandle<JS::Value> ==================================================================================== but I don't know if those errors are even within that method; it's a unified compile, so all kinds of crap are lumped together. I see neither nsString stuff nor MutableHandle<Value> stuff in that method.

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=haz&rev=47aac229cc2d shows a run with the hazard missing. When I run that same revision on my slave, it finds the hazard. I retriggered it 3 times on tbpl to see if it'll give inconsistent results there.

I'm not sure if it's clear in these bug comments so far. The problem is basically that if I go to https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=haz and scan through the results of the Hf jobs (click on each one, look at the summary results eg "TinderboxPrint: 8/15 hazards allowed, 86 unsafe refs"), then you'll see the counts fluctuating up and down when nothing relevant is being changed. This is Bad.

Ah, perfect. https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=haz&rev=47aac229cc2d has 4 runs, two with 8 hazards, two with 9. Bizarre.

Different symptoms but same underlying cause as bug 950176.