In mode 2.5, when a DoH lookup fails to resolve an address due to any reason other than:

1. NXDOMAIN returned from the DoH provider (TRRSkippedReason::TRR_NXDOMAIN)
2. We skipped TRR because previously resolved addresses could not be connected to (TRRSkippedReason::TRR_DISABLED_FLAG)
3. Confirmation state is CONFIRM_FAILED or CONFIRM_TRYING_FAILED (TRRSkippedReason::TRR_NOT_CONFIRMED)
   we should retry the DoH lookup and make sure we use a fresh connection to do so.

This patch exposes a flag that tells a TRR to set the LOAD_FRESH_CONNECTION
load flag when creating the channel for the TRR lookup. This when seen by
TRRServiceChannel will cause us to close any existing DoH connections before
connecting. After a failed TRR lookup, nsHostResolver will retry with this
flag set, if we are in strict fallback mode and the skip reason was not one
of NXDOMAIN, DISABLED_FLAG, and NOT_CONFIRMED.

TODO:

• Haven't really looked at ODoH at all yet
• How should confirmation change now that we're retrying?
• If we kill the DoH connection, other ongoing lookups will be killed too.
  Need to confirm that this will at least eventually lead to a retry, and
  that this is appropriate (or what we should do instead if not) and document
  how the skip reason telemetry will look when this happens.
• Tests
• ???

This adds a feature to moz-http2's doh path that helps test connection
cycling. We log the remote ports of DoH requests and expose an API to
fetch the log. A specific name will trigger us to send a delayed response
to help simulate network volatility. The log is then checked for correctness
in the test script.

TODO: add assertions for the log. this is easy, just waiting to figure out
one thing: at the moment when we cycle the connection for TRR, the retry
lookup queries as well as the confirmation query all use different connections.
I think this is because at the time of their dispatch there's no cached
connection so we end up using new connections for them... for subsequent
lookups we re-use the connection that was created for Confirmation.

This is evident by looking at the log of ports seen by the server when
we dump it at the end.

Depends on D129227

Reverts bug 1651672 / rev D86520. It's been a few releases,
we can remove this cleanup code.

Depends on D131467

Depends on D132107

Depends on D132109

Depends on D132109

Depends on D132112

Data review request for telemetry probes added in Part 4 (D132109)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=77715c49b83311cdfae5b18ac9f3cbffe100343e

Comment on attachment 9252437 [details]
data-review-Bug-1737198.md

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Nihanth Subramanya is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

Result: datareview+

Hey chutten, the previous telemetry impl was piggy backing some data onto an existing histogram. Now I've added a new histogram for it - same as the old skip reason histogram but will exist to isolate data that was collected when this strict mode feature is on.

The questions we're answering, and everything else, remain the same.

Comment on attachment 9253065 [details]
Data review request v2

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Nihanth Subramanya is responsible.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

Is the data collection request for default-on or default-off?

Default on for all channels.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does the data collection use a third-party collection tool?

No.

Result: datareview+

Backed out 8 changesets (Bug 1743122, Bug 1737198) for causing build bustages on nsHostRecord.cpp.
Backout link
Push with failures
Failure Log
Also xpcshell - X1 Failure Log

Backed out 8 changesets (Bug 1737198, Bug 1743122) for causing xpcshell failures in unit/test_trr.js
Backout link: https://hg.mozilla.org/integration/autoland/rev/dae06634f2bfbfd75cc653817122978c85c84319
Push with failures, failure log.
(Update): Also caused bustages in nsHostResolver.

Depends on D132115

Backed our as per dev request. CLOSED TREE

Backout link : https://hg.mozilla.org/integration/autoland/rev/a19bf299a00192a5c3e7e89e7b69ee3d91ab4def

Backed out 9 changesets (Bug 1743122, Bug 1737198) for causing assertion failure in TRRServiceChild.cpp CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=044390f37a8fdf9c30d035baf630ef2f62d9f094&selectedTaskRun=Rl5eG-U8Tz2Vak8eichzow.0

Log: https://treeherder.mozilla.org/logviewer?job_id=359958659&repo=autoland&lineNumber=2544
TV failures: https://treeherder.mozilla.org/logviewer?job_id=359958535&repo=autoland&lineNumber=1756
https://treeherder.mozilla.org/logviewer?job_id=359953062&repo=autoland&lineNumber=1843
Crash: https://treeherder.mozilla.org/logviewer?job_id=359958254&repo=autoland&lineNumber=4393

Backout: https://hg.mozilla.org/integration/autoland/rev/9fe35e8ae5e46fe5d57c68b180d34c0055dec818

This failed to land again, this time because I forgot to check that we're actually in the socket process before trying to mirror the confirmation state. We agreed to back out. In the meantime, here is a try push with a fix, let's make sure this actually passes everything before attempting to land again...

https://treeherder.mozilla.org/jobs?repo=try&revision=dfac5e97256df9ac57079950f4559f7c9ca68b2e

Try's green, landing this again. 🤞

https://hg.mozilla.org/mozilla-central/rev/bc5a65d8d866
https://hg.mozilla.org/mozilla-central/rev/c016a5438637
https://hg.mozilla.org/mozilla-central/rev/24db7713515a
https://hg.mozilla.org/mozilla-central/rev/49b9db2831ac
https://hg.mozilla.org/mozilla-central/rev/4123e58dbff2
https://hg.mozilla.org/mozilla-central/rev/6d1884416d1d
https://hg.mozilla.org/mozilla-central/rev/fdad127a1d21
https://hg.mozilla.org/mozilla-central/rev/b2126f5ed075