It seems that this is occurring because the function on the RHS of the instanceof has a non-Function prototype. From my logging:

Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:1128:1005                                                                                                 
 -> RHS function's prototype is not an object or is not Function                                                                                                                                                                              
Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:850:749                                                                                                   
 -> RHS function's prototype is not an object or is not Function                                                                                                                                                                              
Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:850:916                                                                                                   
 -> RHS function's prototype is not an object or is not Function                                                                                                                                                                              
Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:850:749                                                                                                   
 -> RHS function's prototype is not an object or is not Function                                                                                                                                                                              
Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:850:916                                                                                                   
 -> RHS function's prototype is not an object or is not Function                                                                                                                                                                              
Trying to attach instanceof IC stub: script https://docs.google.com/static/document/client/js/4246577276-client_js_prod_kix_core.js:850:916                                                                                                   
 -> RHS function's prototype is not an object or is not Function                                                                               
....

i.e., we're hitting the case at https://searchfox.org/mozilla-central/source/js/src/jit/CacheIR.cpp#4435 .

Pulling down that .js and staring at it a bit, there's an interesting pattern like:

function f(x) { ... x instanceof C ... }    // this is the IC with the attachment failures
function C() { ... }
C.prototype.property = ...

Which I think means that (given funProto is the prototype object for C) funProto->staticPrototype() will have changed from that of Function, if my understanding of "static prototypes" is correct?

Actually, I slightly misunderstood things (and, being very rusty in JS, had forgotten the difference between __proto__ and prototype). It turns out that the important bit in the JS above is actually:

C.__proto__ = D;

i.e., the prototype chain of the constructor is modified. (It's a bit tricky to see in the obfuscated JS; for the file above, line 850 column 916, the RHS of instanceof is ckb, which is passed to A as first arg; function A is defined on line 68 col 1 and calls Oxa with ckb as first arg; Oxa is assigned from Kxa (line 67), and Kxa(a,b) does a.__proto__=b on line 67 col 403.

So tl;dr: we need to handle a instanceof b where b.__proto__ != Function.__proto__ to allow the IC stub to attach.

Based on discussions with :djvj, it seems that this IC attachment logic is
overly conservative. We're seeing a case where the __proto__ of a constructor
function is reassigned, which causes all instanceof ICs to fail to attach. The
test case is like:

function C() { /* ... */ }
C.__proto__ = D;
var o = new C();
var result = o instanceof C;  // this IC fails to attach

It seems that all that is really necessary is that the RHS of the instanceof is
a callable (required by the spec), and that its .prototype property is
present directly on the object (as opposed to somewhere up the prototype
chain). The IC guards on the shape of the RHS, so the baked-in load of the slot
for .prototype will always be correct. So it seems that deleting this check
for C.__proto__ == Function.__proto__ is safe.

Some stats collected from this change: using the document at link and with a simple printf addition to the attach code, the baseline (prior to this change) has 102 successful attachments of InstanceOf IC stubs to scripts from docs.google.com, while with this change, it has 397 successful attachments. (Will attach the simple printf diff and the output log for reproducibility.)

Patch accepted and try runs (link) are green; I think this should be ready to go...

Pushed by malexandru@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4b4511cf734e
remove restriction on IC attachment for instanceof: allow RHS with a reassigned proto. r=djvj,jandem

(In reply to Chris Fallin [:cfallin] from comment #8)

Some stats collected from this change: using the document at link and with a simple printf addition to the attach code, the baseline (prior to this change) has 102 successful attachments of InstanceOf IC stubs to scripts from docs.google.com, while with this change, it has 397 successful attachments. (Will attach the simple printf diff and the output log for reproducibility.)

It would be interesting to look at the number of times total times the optimized stub is hit before and after this change. Attaches indicate the number of places where we're successfully optimization, but some additional insight into the number of times each of those sites hits would be worthwhile.

(In reply to Kannan Vijayan [:djvj] from comment #14)

It would be interesting to look at the number of times total times the optimized stub is hit before and after this change. Attaches indicate the number of places where we're successfully optimization, but some additional insight into the number of times each of those sites hits would be worthwhile.

Yes, agreed, dynamic counts would be interesting. As far as I understand it, the cache IR logs only record attachment events, not execution of ICs, since the execution is on the fastpath (although I could be wrong, as I found the cacheIR spew output to be lots of verbose JSON and thus measured with simple printfs instead); but I suppose there's a way to get this from the warmup counters? I'll dig a bit more...

For printf-style you can use writer.callPrintString("Foo") before the return-from-IC CacheIR op to track success cases.

https://hg.mozilla.org/mozilla-central/rev/7e12c8c5d1a3

(In reply to Chris Fallin [:cfallin] from comment #16)

(although I could be wrong, as I found the cacheIR spew output to be lots of verbose JSON and thus measured with simple printfs instead)

Have you seen the CacheIR analyzer?

(In reply to Iain Ireland [:iain] from comment #19)

(In reply to Chris Fallin [:cfallin] from comment #16)

(although I could be wrong, as I found the cacheIR spew output to be lots of verbose JSON and thus measured with simple printfs instead)

Have you seen the CacheIR analyzer?

Wow, that's great, thanks (and thanks Jan for the tip as well)! I'll follow up with dynamic stats soon.

OK, here are some stats for closure: on the same Google Docs test (but interacting by hand so non-comparable with earlier stats), I see 101235 successful IC invocations (not bailed out) with this bug's change, of which 93055 would not have attached prior to this change. Will attach patch used to measure and output log.