Reporting API has been enabled on nightly for several years. And it is out dated that it requires work to catch up the latest spec https://bugzilla.mozilla.org/show_bug.cgi?id=1620573
It makes sense that we disable the feature on nightly. Enable it again when we have sorted out the release plan.

Agreed. And it probably shouldn't live under DOM: Security. I can take point to disable it myself.

This disables the outdated, incorrect implementation of the
Reporting API. The current implementation was only enabled
on Nightly builds, but given its current state it does not
even make sense there.

Backed out for causing mochitest failures on test_serviceworker_interfaces.html

Failure line: TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | false: Report should be defined on self

Push with failures

Failure log

Backout link

[task 2022-06-22T14:15:58.534Z] 14:15:58     INFO - TEST-PASS | dom/serviceworkers/test/test_serviceworker_interfaces.html | true: NetworkInformation should  NOT be defined on self 
[task 2022-06-22T14:15:58.534Z] 14:15:58     INFO - Buffered messages finished
[task 2022-06-22T14:15:58.535Z] 14:15:58     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | false: Report should  be defined on self 
[task 2022-06-22T14:15:58.535Z] 14:15:58     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:417:16
[task 2022-06-22T14:15:58.535Z] 14:15:58     INFO -     setupSW/window.onmessage@dom/serviceworkers/test/test_serviceworker_interfaces.html:31:11
[task 2022-06-22T14:15:58.536Z] 14:15:58     INFO - Not taking screenshot here: see the one that was previously logged
[task 2022-06-22T14:15:58.537Z] 14:15:58     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | false: ReportBody should  be defined on self 
[task 2022-06-22T14:15:58.537Z] 14:15:58     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:417:16
[task 2022-06-22T14:15:58.537Z] 14:15:58     INFO -     setupSW/window.onmessage@dom/serviceworkers/test/test_serviceworker_interfaces.html:31:11
[task 2022-06-22T14:15:58.537Z] 14:15:58     INFO - Not taking screenshot here: see the one that was previously logged
[task 2022-06-22T14:15:58.538Z] 14:15:58     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | false: ReportingObserver should  be defined on self 
[task 2022-06-22T14:15:58.538Z] 14:15:58     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:417:16
[task 2022-06-22T14:15:58.538Z] 14:15:58     INFO -     setupSW/window.onmessage@dom/serviceworkers/test/test_serviceworker_interfaces.html:31:11
[task 2022-06-22T14:15:58.539Z] 14:15:58     INFO - Not taking screenshot here: see the one that was previously logged
[task 2022-06-22T14:15:58.540Z] 14:15:58     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_serviceworker_interfaces.html | 3 === 0: The following interface(s) are not enumerated: Report, ReportBody, ReportingObserver 
[task 2022-06-22T14:15:58.540Z] 14:15:58     INFO -     SimpleTest.ok@SimpleTest/SimpleTest.js:417:16
[task 2022-06-22T14:15:58.540Z] 14:15:58     INFO -     setupSW/window.onmessage@dom/serviceworkers/test/test_serviceworker_interfaces.html:31:11
[task 2022-06-22T14:15:58.540Z] 14:15:58     INFO - TEST-PASS | dom/serviceworkers/test/test_serviceworker_interfaces.html | true: Bogus pref annotation for Module 

Since the backout was made there was another failure that was confirmed to have come from you push.

Failure line: TEST-UNEXPECTED-TIMEOUT | /html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html | Multiple globals for Worker constructor: COEP reports - Test timed out

Push with failures

Failure log

Backout link

[task 2022-06-22T14:20:02.715Z] 14:20:02     INFO - TEST-START | /html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html
[task 2022-06-22T14:20:02.724Z] 14:20:02     INFO - Setting pref browser.tabs.remote.useCrossOriginEmbedderPolicy to true
[task 2022-06-22T14:20:02.894Z] 14:20:02     INFO - PID 2240 | JavaScript error: https://web-platform.test:8443/html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html, line 15: TypeError: global.ReportingObserver is not a constructor
[task 2022-06-22T14:20:12.931Z] 14:20:12     INFO - 
[task 2022-06-22T14:20:12.931Z] 14:20:12     INFO - TEST-UNEXPECTED-TIMEOUT | /html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html | Multiple globals for Worker constructor: COEP reports - Test timed out
[task 2022-06-22T14:20:12.931Z] 14:20:12     INFO - TEST-INFO | expected FAIL
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - TEST-UNEXPECTED-ERROR | /html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html | Unhandled rejection: global.ReportingObserver is not a constructor
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - observeReports@https://web-platform.test:8443/html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html:15:20
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - onload<@https://web-platform.test:8443/html/cross-origin-embedder-policy/multi-globals/workers-coep-report.https.html:30:7
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - Test.prototype.step@https://web-platform.test:8443/resources/testharness.js:2590:25
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - Test.prototype.step_func/<@https://web-platform.test:8443/resources/testharness.js:2637:35
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - TEST-INFO took 10217ms
[task 2022-06-22T14:20:12.933Z] 14:20:12     INFO - Restarting browser for new test group
[task 2022-06-22T14:20:12.934Z] 14:20:12     INFO - PID 2240 | 1655907612933	Marionette	INFO	Stopped listening on port 55047
[task 2022-06-22T14:20:13.014Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15ec04400 - BlockShutdown: CanSend.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3607
[task 2022-06-22T14:20:13.015Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15ec04400 - ShutDownProcess: Sent shutdown message.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1808
[task 2022-06-22T14:20:13.016Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=152712800 - BlockShutdown: CanSend.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3607
[task 2022-06-22T14:20:13.016Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=152712800 - ShutDownProcess: Sent shutdown message.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1808
[task 2022-06-22T14:20:13.017Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15f495400 - BlockShutdown: CanSend.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3607
[task 2022-06-22T14:20:13.018Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15f495400 - ShutDownProcess: Sent shutdown message.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1808
[task 2022-06-22T14:20:13.019Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15ec04400 - ShutDownProcess: Closing channel.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1843
[task 2022-06-22T14:20:13.020Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15ec04400 - RemoveShutdownBlockers: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3709
[task 2022-06-22T14:20:13.021Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=152712800 - ShutDownProcess: Closing channel.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1843
[task 2022-06-22T14:20:13.022Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=152712800 - RemoveShutdownBlockers: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3709
[task 2022-06-22T14:20:13.023Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15f495400 - ShutDownProcess: Closing channel.: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:1843
[task 2022-06-22T14:20:13.024Z] 14:20:13     INFO - PID 2240 | [Parent 2240, Main Thread] WARNING: ContentParent: id=15f495400 - RemoveShutdownBlockers: file /builds/worker/checkouts/gecko/dom/ipc/ContentParent.cpp:3709
[task 2022-06-22T14:20:13.450Z] 14:20:13     INFO - Browser exited with return code 0
[task 2022-06-22T14:20:13.453Z] 14:20:13     INFO - Closing logging queue
[task 2022-06-22T14:20:13.453Z] 14:20:13     INFO - queue closed
[task 2022-06-22T14:20:13.473Z] 14:20:13     INFO - Application command: /opt/worker/tasks/task_165590695434041/build/application/Firefox Nightly.app/Contents/MacOS/firefox --marionette about:blank -foreground -profile /var/folders/xk/gftf94q92cb5r37qj2w7wtrm000014/T/tmpbwwaseuk
[task 2022-06-22T14:20:13.479Z] 14:20:13     INFO - PID 2291 | 1655907602580	Marionette	INFO	Marionette enabled
[task 2022-06-22T14:20:13.480Z] 14:20:13     INFO - PID 2291 | 1655907602609	Marionette	INFO	Listening on port 55668
[task 2022-06-22T14:20:13.481Z] 14:20:13     INFO - PID 2291 | console.warn: SearchSettings: "get: No settings file exists, new profile?" (new NotFoundError("Could not open the file at /var/folders/xk/gftf94q92cb5r37qj2w7wtrm000014/T/tmpuc0jrn0x/search.json.mozlz4", (void 0)))
[task 2022-06-22T14:20:13.481Z] 14:20:13     INFO - Starting runner
[task 2022-06-22T14:20:13.758Z] 14:20:13     INFO - PID 2291 | 2022-06-22 14:20:13.757 firefox[2291:56838] Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
[task 2022-06-22T14:20:14.076Z] 14:20:14     INFO - TEST-START | /html/editing/editing-0/autocapitalization/autocapitalize.html

I had no idea this is used in so many independent tests, which is sad. In essence, this means we need to rely on our own tests for these features (COEP, mostly).

Reminds me of a similar case, where we did not support securitypolicyviolation events (for CSP) and various CSP-unrelated tests started expecting it. That was in bug 1037335 (and its dependency tree).

Forgot to mention, that I'm on it. I'm afraid we got no choice but to mark those WPT as failing.

I had no idea this is used in so many independent tests, which is sad. In essence, this means we need to rely on our own tests for these features (COEP, mostly).

We could just enable the pref for these tests.

Backed out changeset e883552f154d (bug 1775194) for causing multiple wpt failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/592435477115b08d6bbf7efcec5c86d7aea0262d

Push with failures

Failure log wpt #1

Failure log wpt #2

Failure log wpt #3

Backed out changeset c3682c83547e (bug 1775194) for causing wpt failures in /deprecation-reporting/idlharness.any.html

Backout link: https://hg.mozilla.org/integration/autoland/rev/b75452cff52ce1c17ddf010504e78ad3fc9e9bb7

Push with failures

Failure log

INFO - TEST-PASS | /deprecation-reporting/idlharness.any.html | idl_test validation 
[task 2022-06-28T14:00:27.649Z] 14:00:27     INFO - TEST-UNEXPECTED-FAIL | /deprecation-reporting/idlharness.any.html | DeprecationReportBody interface: existence and properties of interface object - assert_own_property: self does not have own property "DeprecationReportBody" expected property "DeprecationReportBody" missing
[task 2022-06-28T14:00:27.649Z] 14:00:27     INFO - IdlInterface.prototype.assert_interface_object_exists@http://web-platform.test:8000/resources/idlharness.js:1309:24

Surprising these did not come up in the previous test run. OK, let's make sure these tests also get the dom.reporting.enabled set to true.

Backed out changeset 068f148f46f2 (Bug 1775194) for causing wpt failures on document-write-reporting-tentative.html.
Backout link
Push with failures
Failure Log

It's confusing why these test errors did not show up on try and I keep getting new ones between backout and backout.

@Hsin-Yi: It seems that a lot of APIs are using those observers for testing completely unrelated tests (e.g., COEP). I believe we need to keep this feature in some sort of reasonable state to not lose test coverage for COEP and such.

(In reply to Frederik Braun [:freddy] from comment #17)

It's confusing why these test errors did not show up on try and I keep getting new ones between backout and backout.

@Hsin-Yi: It seems that a lot of APIs are using those observers for testing completely unrelated tests (e.g., COEP). I believe we need to keep this feature in some sort of reasonable state to not lose test coverage for COEP and such.

Thank you Freddy for giving it a try, and sorry that it turns out to be not as trivial as the first thought. I don't really know who has a broad view of this API and other features like COEP, document policy, etc. So, yup, I believe we should keep the status for Reporting API as is for another while, and review a plan for this.

Backed out for causing multiple reporting failures.

• backout: https://hg.mozilla.org/integration/autoland/rev/bc875fbe86d7b1256f1af309835ecb6359a2d2fc
• push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=1491786f84021b12026f8570cb54eb02f2dfa450&group_state=expanded
• push where tier 1 fails showed up: https://treeherder.mozilla.org/jobs?repo=autoland&revision=b8740c0dade37fbaa6f39b89727dbf9fdc69b72a&searchStr=wpt&group_state=expanded
• failure log:
  • TEST-UNEXPECTED-FAIL | /deprecation-reporting/idlharness.any.html | DeprecationReportBody interface: existence and properties of interface object - assert_own_property: self does not have own property "DeprecationReportBody" expected property "DeprecationReportBody" missing
  • TEST-UNEXPECTED-NOTRUN | /document-policy/reporting/document-write-reporting-tentative.html | Document-write Report Format - expected FAIL

There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:freddy, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.

https://treeherder.mozilla.org/jobs?repo=try&revision=910fbcedeefba87c7d02f2931d8949463e4ac3a2

https://hg.mozilla.org/mozilla-central/rev/45318a69aa49

Should dom.reporting.featurePolicy.enabled be disabled too? Found this from https://github.com/mdn/browser-compat-data/pull/18557.

Oh, commandeered by Tom actually.

Oh, we probably should probably disable it even in Nightly and only enable it for tests.