There are several problems here, in descending order of impact on the number of sub-tests we're failing here:

1. Some general bugs: nsDataChannel::OpenContentStream double-parses the mime type in data URLs, doesn't fully serialize the parsed mime type and just uses its essense, and fetch/XHR then add a second copy of the charset. (done in bug 1845006)
2. The spec expects ";base64" to only be allowed at the end of a data URIs' path, but we allow it in the middle as well for old webcompat reasons (bug 781693). There is a spec discussion about this which seems either unaware of, or unconcerned with, any webcompat issues (at least from a quick reading). (done in bug 1848903)
3. The spec apparently expects us to pre-parse the data URI as a standard URL first, such that in the case of data://test:test/,X it will fail as though the URL has an invalid port, without even trying to parse a mime type. I'm not quite sure how this follows from the spec text yet, but the Chromium bug above says that: "Anne pointed out that this should fail before data URL handling, in the URL parser, due to an invalid port"

Based on the spec text, Firefox should not just fix these issues for fetch/XHR, but data URIs in general, so it may be good to add mirror WPTs for XHR as well (as I don't see any, and our current implementation of XHR doesn't use the fetch driver directly as the spec text expects).