Closed Bug 1763489 Opened 3 years ago Closed 3 years ago

add a `./mach test-info failure-report` to get more information about failures

Categories

(Testing :: General, task)

Default
task

Tracking

(firefox101 fixed)

RESOLVED FIXED
101 Branch
Tracking Status
firefox101 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

one issue to resolve with single tracking bugs is that we will have some bugs that have different failure modes and it is useful when investigating an intermittent to know those failure modes.

here is an example of running the command:

./mach test-info failure-info --bugid 1760085
21 errors with:
12:39:29  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/antitracking/test/xpcshell/test_purge_trackers.js | xpcshell return code: 0
12:39:29  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/antitracking/test/xpcshell/test_purge_trackers.js |  - cookie remains while storage access permission exists. - false == true

  macosx1015-64-qr/debug-xpcshell-spi-nw-fis: 1
  macosx1015-64-qr/opt-xpcshell-fis: 1
  macosx1015-64-qr/opt-xpcshell-spi-nw: 3
  macosx1015-64-shippable-qr/opt-xpcshell-spi-nw: 3
  macosx1015-64-qr/opt-xpcshell: 3
  windows7-32-qr/opt-xpcshell: 2
  windows7-32-shippable-qr/opt-xpcshell: 1
  linux1804-64-tsan-qr/opt-xpcshell: 1
  macosx1015-64-qr/debug-xpcshell: 4
  windows7-32-shippable-qr/opt-xpcshell-spi-nw: 1
  macosx1015-64-shippable-qr/opt-xpcshell: 1

you can see that in this case we have many configs but one failure.

here is an example of multiple failure types:

./mach test-info failure-info --bugid 1760126
2 errors with:
20:47:04  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | xpcshell return code: 1
20:52:23  WARNING -  TEST-UNEXPECTED-FAIL | Received SIGINT (control-C), so stopped run. (Use --keep-going to keep running tests after killing one with SIGINT)

  windows10-32-2004-qr/debug-xpcshell-fis: 1
  windows10-32-2004-qr/debug-xpcshell: 1
25 errors with:
16:00:51  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | xpcshell return code: 0
16:00:51  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | early_hints_multiple - [early_hints_multiple : 91] 0 == 2

  linux1804-64-tsan-qr/opt-xpcshell-fis: 2
  linux1804-64-tsan-qr/opt-xpcshell: 14
  linux1804-64-qr/debug-xpcshell: 5
  linux1804-64-asan-qr/opt-xpcshell: 2
  linux1804-64-qr/opt-xpcshell: 1
  linux1804-64-shippable-qr/opt-xpcshell: 1
1 errors with:
23:09:16  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | xpcshell return code: 0
23:09:27  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | setup - [setup : 1] "" == "h3-29"
23:14:35  WARNING -  TEST-UNEXPECTED-FAIL | Received SIGINT (control-C), so stopped run. (Use --keep-going to keep running tests after killing one with SIGINT)

  windows10-32-2004-qr/debug-xpcshell-spi-nw: 1
2 errors with:
21:50:43  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | xpcshell return code: 0
21:56:00  WARNING -  TEST-UNEXPECTED-FAIL | Received SIGINT (control-C), so stopped run. (Use --keep-going to keep running tests after killing one with SIGINT)

  windows10-32-2004-qr/debug-xpcshell-spi-nw: 2
7 errors with:
16:38:58  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | xpcshell return code: 0
16:38:58  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_early_hint_listener.js | early_hints - [early_hints : 70] 0 == 1

  linux1804-64-qr/debug-xpcshell: 4
  linux1804-64-qr/opt-xpcshell: 2
  linux1804-64-shippable-qr/opt-xpcshell: 1

and an example of a simple case:

/mach test-info failure-info --bugid 1760132
82 errors with:
13:50:57  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_large_post.js | xpcshell return code: 0
13:50:57  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_large_post.js | test_large_post2 - [test_large_post2 : 48] 8388608 == "851968"
13:50:57  WARNING -  TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_http3_large_post.js | test_large_post2 - [test_large_post2 : 65] 1 == 2

  macosx1100-64-shippable-qr/opt-xpcshell-spi-nw-fis: 2
  macosx1100-64-shippable-qr/opt-xpcshell-spi-nw: 80
Assignee: nobody → jmaher
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/db425d9271ed add failure report to test-info. r=gbrown
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 101 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: