Closed Bug 712032 Opened 13 years ago Closed 13 years ago

Intermittent cluster of failures on a single build: test_XHR.html | wrong blob size, test_pluginstream_asfileonly.html | content doesn't match, test_xhr2.html | 'blob' responseType failed, test_pluginstream_asfileonly.html | content doesn't match

Categories

(Core :: DOM: Core & HTML, defect)

Product:
Core
Component:
DOM: Core & HTML
Version:
11 Branch
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox11 + fixed

People

(Reporter: mbrubeck, Assigned: michal)

References

Details

(Keywords: intermittent-failure, Whiteboard: [qa-]) 

This first occurred on a nightly Linux 32-bit PGO build.  The failures did not occur on the non-nightly build from the same changeset:

https://tbpl.mozilla.org/php/getParsedLog.php?id=8014179&tree=Firefox
Rev3 Fedora 12 mozilla-central pgo test mochitests-1/5 on 2011-12-18 04:05:10 PST for push a5e63e00db27

5292 INFO TEST-PASS | /tests/content/base/test/test_XHR.html | should not be a File
5293 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_XHR.html | wrong blob size - got 19, expected 11
5294 INFO TEST-PASS | /tests/content/base/test/test_XHR.html | responseXML should be null. - null should equal null

https://tbpl.mozilla.org/php/getParsedLog.php?id=8014023&tree=Firefox
Rev3 Fedora 12 mozilla-central pgo test mochitests-2/5 on 2011-12-18 04:05:09 PST for push a5e63e00db27

6326 INFO TEST-START | /tests/dom/plugins/test/test_pluginstream_asfileonly.html
NPP_Destroy
NPP_NewStream
NPP_StreamAsFile, file=/tmp/tmpZEdB70/Cache/2/EC/3372Dd01
NPP_DestroyStream
6327 INFO TEST-PASS | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | bad XMLHttpRequest status - 200 should equal 200
6328 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | content doesn't match - got x^ÿffffedÿffff95Mnÿffffdb@%c…ÿfffff7%1r%7ÿffffc0ÿfffff0ÿffff82vtSÿfffff4, expected Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.

Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.

Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, At accusam aliquyam diam diam dolore dolores duo eirmod eos erat, et nonumy sed tempor et et invidunt justo labore Stet clita ea et gubergren, kasd magna no rebum. sanctus sea sed takimata ut vero voluptua. est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut laboreet dolore magna aliquyam erat.

Consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
6329 INFO TEST-PASS | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | plugin reported error - pass should equal pass
6330 INFO TEST-END | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | finished in 196ms

https://tbpl.mozilla.org/php/getParsedLog.php?id=8014032&tree=Firefox
Rev3 Fedora 12 mozilla-central pgo test mochitests-3/5 on 2011-12-18 04:05:11 PST for push a5e63e00db27

9509 INFO TEST-START | /tests/dom/workers/test/test_xhr2.html
9510 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_xhr2.html | an unexpected uncaught JS exception reported through window.onerror - 'blob' responseType failed at http://mochi.test:8888/tests/dom/workers/test/xhr2_worker.js:44
9511 INFO TEST-END | /tests/dom/workers/test/test_xhr2.html | finished in 311ms

https://tbpl.mozilla.org/php/getParsedLog.php?id=8014306&tree=Firefox
Rev3 Fedora 12 mozilla-central pgo test mochitest-other on 2011-12-18 04:05:07 PST for push a5e63e00db27

721 INFO TEST-START | /tests/dom/plugins/test/test_pluginstream_asfileonly.html
NPP_Destroy
NPP_NewStream
NPP_StreamAsFile, file=/tmp/tmpi5BKwG/Cache/2/EC/3372Dd01
NPP_DestroyStream
722 INFO TEST-PASS | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | bad XMLHttpRequest status - 200 should equal 200
723 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | content doesn't match - got x^ÿffffedÿffff95Mnÿffffdb@%c…ÿfffff7%1r%7ÿffffc0ÿfffff0ÿffff82vtSÿfffff4, expected Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat.

Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi.

Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.

Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, At accusam aliquyam diam diam dolore dolores duo eirmod eos erat, et nonumy sed tempor et et invidunt justo labore Stet clita ea et gubergren, kasd magna no rebum. sanctus sea sed takimata ut vero voluptua. est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut laboreet dolore magna aliquyam erat.

Consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
724 INFO TEST-PASS | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | plugin reported error - pass should equal pass
725 INFO TEST-END | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | finished in 330ms
Is it reproducible on this build?
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #1)
> Is it reproducible on this build?

Good question; I'll try retriggering to find out.

The following set occurred on a push that for some reason got four sets of M1/M2/M3 tests on a single Linux debug build.  Each of the suites failed once out of its four runs.  The mochitest-other suite ran only once and did not have the failure from comment 0:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=326455c9793d

https://tbpl.mozilla.org/php/getParsedLog.php?id=8025578&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-1/5 on 2011-12-18 21:02:51 PST for push 326455c9793d
5294 ERROR TEST-UNEXPECTED-FAIL | /tests/content/base/test/test_XHR.html | wrong blob size - got 19, expected 11

https://tbpl.mozilla.org/php/getParsedLog.php?id=8025429&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-2/5 on 2011-12-18 21:02:50 PST for push 326455c9793d
6318 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/plugins/test/test_pluginstream_asfileonly.html | content doesn't match - got x^ÿffffedÿffff95Mnÿffffdb@%c…ÿfffff7%1r%7ÿffffc0ÿfffff0ÿffff82vtSÿfffff4, expected Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

https://tbpl.mozilla.org/php/getParsedLog.php?id=8025521&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-3/5 on 2011-12-18 21:02:50 PST for push 326455c9793d
9516 ERROR TEST-UNEXPECTED-FAIL | /tests/dom/workers/test/test_xhr2.html | an unexpected uncaught JS exception reported through window.onerror - 'blob' responseType failed at http://mochi.test:8888/tests/dom/workers/test/xhr2_worker.js:44
On this Linux debug build, every test suite ran twice, and M1/2/3/other each failed the second time:

https://tbpl.mozilla.org/php/getParsedLog.php?id=8021184&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-1/5 on 2011-12-18 15:10:36 PST for push 7b4007e3cafa

https://tbpl.mozilla.org/php/getParsedLog.php?id=8020994&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-2/5 on 2011-12-18 15:10:33 PST for push 7b4007e3cafa

https://tbpl.mozilla.org/php/getParsedLog.php?id=8021089&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitests-3/5 on 2011-12-18 15:10:02 PST for push 7b4007e3cafa

https://tbpl.mozilla.org/php/getParsedLog.php?id=8021758&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound debug test mochitest-other on 2011-12-18 15:10:03 PST for push 7b4007e3cafa
And on this Linux PGO build, M1/2/3/other each ran once and failed:

https://tbpl.mozilla.org/php/getParsedLog.php?id=8033238&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound pgo test mochitests-1/5 on 2011-12-19 07:31:16 PST for push 2172873119db

https://tbpl.mozilla.org/php/getParsedLog.php?id=8032984&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound pgo test mochitests-2/5 on 2011-12-19 07:31:32 PST for push 2172873119db

https://tbpl.mozilla.org/php/getParsedLog.php?id=8033071&tree=Mozilla-Inbound
Rev3 Fedora 12 mozilla-inbound pgo test mochitests-3/5 on 2011-12-19 07:31:21 PST for push 2172873119db

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-pgo/1324305602/mozilla-inbound_fedora_test_pgo-mochitest-other-build141.txt.gz
builder: mozilla-inbound_fedora_test_pgo-mochitest-other
slave: talos-r3-fed-057
starttime: 1324308683.11
results: warnings (1)
buildid: 20111219064002
builduid: fd6f7d9ad52c42189104869f464fe1d0
revision: 2172873119db
comment 4 and comment 6 were from this Linux opt (not PGO) build where M1/2/3/other each ran once and failed: https://tbpl.mozilla.org/?tree=UX&rev=02fe687f8f52
In the cases where I retriggered, the retriggered tests were green.

The bug also doesn't seem associated with any particular slave(s).
Update: On this push, the retriggers failed too:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=2172873119db

Anyone want to download that build and do any manual testing?  I've archived it at
http://people.mozilla.com/~mbrubeck/firefox-11.0a1.en-US.linux-i686.712032.tar.bz2
> The following set occurred on a push that for some reason got four sets of
> M1/M2/M3 tests on a single Linux debug build.

The reason being that "is it bad builds?" was my first question, and that was the first build I found where there didn't seem to be any funkiness about multiple builds on a push or any possible confusion about whether the retriggers would run on the same exact bits. Not reproducible on a given build, but the same set of failures happens in multiple suites on a give push? Touching a flaky network is the usual reason for that failure mode.
Related with bug 648429?
It seems that cacheAsFile may not be honored if the response is already cached.
(In reply to Masatoshi Kimura [:emk] from comment #38)
> It seems that cacheAsFile may not be honored if the response is already
> cached.
Bug 661115?
If this is a symptom of bug #661115 I would expect it to be permanent, not orange.

Note that #bug 648429 do not compress requests with STORE_ON_DISK_AS_FILE.

(In reply to Matt Brubeck (:mbrubeck) from comment #3)
> and M1/2/3/other each
> failed the second time:

Do the test clear the cache initially? If something is left in the cache from a previous run of the same test or some other test, the compression introduced in #bug 648429 may cause confusion. I.e. if the first request is marked STORE_ON_DISK_AS_FILE the cache-entry will *not* be compressed. If a subsequent request loads the same resource without STORE_ON_DISK_AS_FILE, it may find the cached (uncompressed) entry but expect it to be smaller.
That set was talos-r3-fed-073, talos-r3-fed-073, talos-r3-fed-069 and talos-r3-fed-065.
And those two were talos-r3-fed-004 and talos-r3-fed-050
 talos-r3-fed-061
And those last two are the first time I've hit it while retriggering on a job that had failed the first time, on talos-r3-fed-046 and talos-r3-fed-054, neither of which were involved in the first round of failure there.
I started getting suspicious about the way that all of that explosion of retriggerable failure on inbound was on builds done by linux-ix-slave17, while in between builds done by linux-any-other-slave weren't failing, so catlee clobbered it. Of course it hasn't done a single build since, having taken up a life of building Android, where nobody even notices if none of your builds work right.
And yes, now that you mention it, those failures *were* on the clobber build by linux-ix-slave17.
That set, however, were on a build by linux-ix-slave33, and yesterday's debug were linux-ix-slave35.
Josh, this is new and disturbing, it appears to indicate an error in the networking or cache system. Can you find the right person to investigate and fix this?

That the plugin "asfileonly" streams use the on-disk cache file and pin it, so that's why I expect something funky is going on with the cache.
Hmm, cache, you say? Looking like compressed data, you say? Be interesting to see whether this still happens in the post bug 715198 world.
Depends on: 715198
If http://hg.mozilla.org/mozilla-central/pushloghtml?startdate=2011-12-16&enddate=2011-12-18 is the best regression range we can get, the most obvious regression candidate is bug 648429.
Blocks: 648429
tracking-firefox11: --- → ?
Current theory is that this is cache-related - Michal, can you take a look?
Assignee: nobody → michal.novotny
Cache compression (bug 648429) was disabled on m-i and m-c yesterday: see bug 715198.
tracking-firefox11: ? → ---
Appears to be fixed on trunk, but still needs to be tracked for Aurora.
Status: NEW → RESOLVED
Closed: 13 years ago
status-firefox11: --- → affected
tracking-firefox11: --- → +
Resolution: --- → FIXED
Depends on: 723100
This appears to have been fixed on mozilla-aurora when bug 715198 was uplifted (thanks bsmedberg).
status-firefox11: affected → fixed
Whiteboard: [orange] → [orange][qa-]
Whiteboard: [orange][qa-] → [qa-]
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.