Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

Backed out changeset 7558c8821a07 (bug 1654103) for multiple failures. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=319317636&repo=autoland&lineNumber=1362

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&revision=7558c8821a074b6f7c1e7d9314976e6b66176e5c

Backout
https://hg.mozilla.org/integration/autoland/rev/2309e130ea8d75bd6fb4a5fc056e503fae7f886c

The fact that this had to be backed out for test bustage (effectively, harness bustage) doesn't give me much confidence in black as a reformatting tool.

Well, I wouldn't blame black here. The thing is that two individual strings have been updated, but the regex string hasn't been taken into that it will fail to parse the source file. Here the causing change:

-    return re.findall("__version__ = '([\d\.]+)'",
-                      read('marionette_driver', '__init__.py'), re.M)[0]
+    return re.findall(
+        "__version__ = '([\d\.]+)'", read("marionette_driver", "__init__.py"), re.M
+    )[0]

That will match __version__ = 'x.y.z', but note that this string has also been changed to double quotes.

Yeah, I don't think this is a proof that black isn't a good tool. We had similar issues with the clang-format move:
A bunch of scripts doing code parsing with some strong assumptions on the style used. Example bug 1499323

(In reply to Mike Hommey [:glandium] from comment #5)

The fact that this had to be backed out for test bustage (effectively, harness bustage) doesn't give me much confidence in black as a reformatting tool.

black checks that the reformatted code still produces a valid AST that is equivalent to the original.
Of course that won't catch instances where we parse Python with regexes.

Amazing. Okay, I'll make that one-line change then re-land.

Backed out for conflicts when backing out bug 1518999.

Backout link: https://hg.mozilla.org/integration/autoland/rev/9006d6f3cb29754037aa0b5ef4c9b2ae67006459

(In reply to Bogdan Tara[:bogdan_tara | bogdant] from comment #12)

Backed out for conflicts when backing out bug 1518999.

Next time, please backout the smaller patch. It will be easier to rebase bug 1518999 than this one :)

(In reply to Sylvestre Ledru [:Sylvestre] from comment #13)

(In reply to Bogdan Tara[:bogdan_tara | bogdant] from comment #12)

Backed out for conflicts when backing out bug 1518999.

Next time, please backout the smaller patch. It will be easier to rebase bug 1518999 than this one :)

The problem was backing out a patch that landed before this one. This bug should land when the tree is closed and known to be clean. That would also reduce the risk of race condition between when the patch is updated and something pythonic landing.

(In reply to Mike Hommey [:glandium] from comment #14)

(In reply to Sylvestre Ledru [:Sylvestre] from comment #13)

(In reply to Bogdan Tara[:bogdan_tara | bogdant] from comment #12)

Backed out for conflicts when backing out bug 1518999.

Next time, please backout the smaller patch. It will be easier to rebase bug 1518999 than this one :)

The problem was backing out a patch that landed before this one. This bug should land when the tree is closed and known to be clean. That would also reduce the risk of race condition between when the patch is updated and something pythonic landing.

I agree. I'll engage the sheriffs on this.

https://hg.mozilla.org/mozilla-central/rev/994ae8e4833c

Can we get an exclusion added to the HG blame/searchfox logic so this doesn't pollute every display? I think this would be like https://bugzilla.mozilla.org/show_bug.cgi?id=1633969.

(In reply to Nick Alexander :nalexander [he/him] from comment #18)

Can we get an exclusion added to the HG blame/searchfox logic so this doesn't pollute every display? I think this would be like https://bugzilla.mozilla.org/show_bug.cgi?id=1633969.

bug 1673690

ignore-this-changeset

Depends on D96609

Hey Ryan -- it's been suggested that we backport this reformat to ESR so that anyone who needs to backport a Python change won't encounter merge conflicts. Does that make sense to you? We figured you might have a better idea of whether the benefit would be worth it.

Given that we've got about another year of support for ESR78, it's probably not a bad idea if it's not too much trouble.