When mochitest/reftest/crashtest/jsreftest/jetpack/gaia-ui-test/pretty much every single other framework has a test failure, it does a "TinderboxPrint: suitename: 500000/1/0" to say that 500K tests passed, 1 failed, 0 were todo. When gaia-integration has a test failure, it does a "TinderboxPrint: T-FAIL" which is what every single other test suite does when it has some catastrophic failure like a 330 second without output timeout that caused it to stop running the suite without finishing. So we have eleventy-seven suites where you click on the orange letter on tbpl, glance down at the tinderboxprint box, and if you see T-FAIL you retrigger, and exactly one suite where if you see T-FAIL you carefully read through the failure summary, maybe open the log to check where the "37 passing, 1 failing, 41 pending" is logged in a way that nobody will ever see, and then decide whether or not you need to retrigger. Or far more likely, you waste a ton of MoCo's money by just retriggering every single run that fails a single test, no matter how common that is.

The mozharness script should be parsing the output and writing the TinderboxPrint line.

(In reply to Jonathan Griffin (:jgriffin) from comment #1) > The mozharness script should be parsing the output and writing the > TinderboxPrint line. Is that happening? Are we good to close this?

The output is getting parsed, I think, but either the output doesn't include the magic format that gets picked up by the parser (most likely), or we're not handling the output properly in the mozharness script. I'll try to find some time to take a look at this.

So, the parser is expecting output in our standard format, which is: passed: xx failed: yy todo: zz Gaia-integration produces output like: 01:54:47 INFO - 36 passing 01:54:47 INFO - 2 failing 01:54:47 INFO - 40 pending We can make the parser look for output in that format.

Hmm, it's probably easier to get gaia-integration to output the summary in the expected format. What repo are those strings defined in?

Ah, this comes from https://github.com/mozilla-b2g/mocha-tbpl-reporter. I'll post a PR.

Comment on attachment 8357512 [details] Link to Github pull-request: https://github.com/mozilla-b2g/mocha-tbpl-reporter/pull/3 Thanks for the pr jgriffin! Comments on GH... Reset flag when you're ready for me to land.

https://github.com/mozilla-b2g/mocha-tbpl-reporter/commit/e9ca97012f87a17b4295a52e3d9d47177895add2 Gareth, do we need to update packson.json in gaia to pick this up, or will it happen magically?

Magically! It looks as if the gaia package will take anything with version 0.0.x.

> npm publish: npm http PUT https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin npm http 409 https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin npm http GET https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin npm http 200 https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin npm http PUT https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin/-rev/2-f806a7278d3fa54aef7ac68426258143 npm http 201 https://registry.npmjs.org/-/user/org.couchdb.user:jonallengriffin/-rev/2-f806a7278d3fa54aef7ac68426258143 jgriffin@:~/mocha-tbpl-reporter$ npm publish npm http PUT https://registry.npmjs.org/mocha-tbpl-reporter npm http 409 https://registry.npmjs.org/mocha-tbpl-reporter npm http GET https://registry.npmjs.org/mocha-tbpl-reporter npm http 200 https://registry.npmjs.org/mocha-tbpl-reporter npm http PUT https://registry.npmjs.org/mocha-tbpl-reporter/0.0.5/-tag/latest npm http 201 https://registry.npmjs.org/mocha-tbpl-reporter/0.0.5/-tag/latest npm http GET https://registry.npmjs.org/mocha-tbpl-reporter npm http 200 https://registry.npmjs.org/mocha-tbpl-reporter npm http PUT https://registry.npmjs.org/mocha-tbpl-reporter/-/mocha-tbpl-reporter-0.0.5.tgz/-rev/11-76fd44bd3b293f453a361900d4da7f81 npm http 201 https://registry.npmjs.org/mocha-tbpl-reporter/-/mocha-tbpl-reporter-0.0.5.tgz/-rev/11-76fd44bd3b293f453a361900d4da7f81

Do we need to update package.json in order for the npm mirror to pick up the new version, which will then get picked up by buildbot? https://github.com/mozilla-b2g/gaia/blob/master/package.json

Nope. But I do see the old version is still here https://tbpl.mozilla.org/php/getParsedLog.php?id=32931117&tree=Mozilla-Inbound even though the npm-mirror has picked up the new version http://npm-mirror.pub.build.mozilla.org/mocha-tbpl-reporter/. Perhaps npm is caching the old version on the build machines... I will add a step to the gaia-integration build to clean the npm ccache just in case.