Workaround for Python BUG https://bugs.python.org/issue32745 - ctypes string pointer fields should accept embedded null characters;
Which affects our possibility to run mozprocess on versions of Python higher than 3.5.3

More details can be found also in previous Mozprocess BUG https://bugzilla.mozilla.org/show_bug.cgi?id=1428713#c49

Run tests command: ./mach python-test --python=3.6 testing/mozbase/mozrunner/tests/test_start.py

The error is happening at line: https://searchfox.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/winprocess.py#167

Stacktrace:

self = <mozprocess.winprocess.EnvironmentBlock object at 0x000001D54F83D470>
 1:24.03 env = {b'!EXITCODE': b'00000001', b'ALLUSERSPROFILE': b'C:\\ProgramData', b'APPDATA': b'C:\\Users\\AH\\AppData\\Roaming', b'COMMONPROGRAMFILES': b'C:\\Program Files\\Common Files', ...}
 1:24.05
 1:24.06     def __init__(self, env):
 1:24.06         if not env:
 1:24.08             self._as_parameter_ = None
 1:24.08         else:
 1:24.09             values = []
 1:24.09             fs_encoding = sys.getfilesystemencoding() or 'mbcs'
 1:24.11             for k, v in env.items():
 1:24.11                 if isinstance(k, bytes):
 1:24.12                     k = k.decode(fs_encoding, 'replace')
 1:24.12                 if isinstance(v, bytes):
 1:24.14                     v = v.decode(fs_encoding, 'replace')
 1:24.14                 values.append("{}={}".format(k, v))
 1:24.14             values.append("")
 1:24.16 >           self._as_parameter_ = LPCWSTR("\0".join(values))
 1:24.17 E           ValueError: embedded null character
 1:24.17
 1:24.19 testing\mozbase\mozprocess\mozprocess\winprocess.py:167: ValueError
 1:24.19
 1:24.19 During handling of the above exception, another exception occurred:
 1:24.20
 1:24.20 runner = <mozrunner.base.browser.GeckoRuntimeRunner object at 0x000001D54F6D1240>
 1:24.20
 1:24.22     def test_start_with_outputTimeout(runner):
 1:24.22         """Start the process and set a timeout"""
 1:24.23 >       runner.start(outputTimeout=0.1)
 1:24.23
 1:24.23 testing\mozbase\mozrunner\tests\test_start.py:45:
 1:24.27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 1:24.27 testing\mozbase\mozrunner\mozrunner\base\browser.py:85: in start
 1:24.27     BaseRunner.start(self, *args, **kwargs)
 1:24.27 testing\mozbase\mozrunner\mozrunner\base\runner.py:141: in start
 1:24.28     reraise(RunnerNotStartedError, "Failed to start the process: %s" % value, tb)
 1:24.28 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 1:24.28
 1:24.28 tp = <class 'mozrunner.errors.RunnerNotStartedError'>
 1:24.28 value = 'Failed to start the process: embedded null character'
 1:24.28 tb = <traceback object at 0x000001D54F6ECBC8>
 1:24.28
 1:24.28     def reraise(tp, value, tb=None):
 1:24.28         if value is None:
 1:24.28             value = tp()
 1:24.28 >       if value.__traceback__ is not tb:
 1:24.28 E       AttributeError: 'str' object has no attribute '__traceback__'

As a workaround I found a fix that others had here: https://github.com/felix-lang/fbuild/issues/35

Fix is: https://github.com/felix-lang/fbuild/commit/fa486b30d1de126b1c2a51939d6d88ed491c7684

Had another error after fixing the encoding:

0:55.61 result = 0, func = <WinFunctionType object at 0x0000016C7798BD48>
 0:55.61 args = (None, '"C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" --headless --wait-for-browser -profile C:\\Users\\AH\\AppData\\Local\\Temp\\tmpaorjhj2x.mozrunner', None, None, 1, 1540, ...)
 0:55.61
 0:55.61     def ErrCheckBool(result, func, args):
 0:55.61         """errcheck function for Windows functions that return a BOOL True
 0:55.61         on success"""
 0:55.61         if not result:
 0:55.61 >           raise WinError()
 0:55.61 E           OSError: [WinError 87] The parameter is incorrect.
 0:55.61
 0:55.61 testing\mozbase\mozprocess\mozprocess\winprocess.py:57: OSError
 0:55.61
 0:55.61 During handling of the above exception, another exception occurred:
 0:55.61
 0:55.61 runner = <mozrunner.base.browser.GeckoRuntimeRunner object at 0x0000016C77F67240>
 0:55.61
 0:55.61     def test_start_process(runner):
 0:55.61         """Start the process and test properties"""
 0:55.61         assert runner.process_handler is None
 0:55.65
 0:55.65 >       runner.start()
 0:55.65
 0:55.65 testing\mozbase\mozrunner\tests\test_start.py:14:
 0:55.65 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 0:55.65 testing\mozbase\mozrunner\mozrunner\base\browser.py:85: in start
 0:55.65     BaseRunner.start(self, *args, **kwargs)
 0:55.65 testing\mozbase\mozrunner\mozrunner\base\runner.py:141: in start
 0:55.65     reraise(RunnerNotStartedError, "Failed to start the process: %s" % value, tb)
 0:55.65 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 0:55.65
 0:55.65 tp = <class 'mozrunner.errors.RunnerNotStartedError'>
 0:55.65 value = 'Failed to start the process: [WinError 87] The parameter is incorrect.'
 0:55.65 tb = <traceback object at 0x0000016C77F62088>
 0:55.65
 0:55.65     def reraise(tp, value, tb=None):
 0:55.65         if value is None:
 0:55.65             value = tp()
 0:55.65 >       if value.__traceback__ is not tb:
 0:55.65 E       AttributeError: 'str' object has no attribute '__traceback__'

This was fixed with a similar issue for Windows Unicode handling: https://github.com/felix-lang/fbuild/issues/36#issuecomment-424884647

This has come up as we start to look at making web-platform-tests (WPT) work on Python 3 on Windows; we use mozprocess for various binaries in the wptrunner and we immediately hit this error on trying out Python 3 (3.8, in my case, though wpt intends to support 3.6+ at this point).

I see there was an attached patch for this bug that was under review - if that could be revived and finished it would be very welcome :)

Are you sure this is still an issue, after https://hg.mozilla.org/mozilla-central/rev/9ea7f7d1768681d657daf36134d7a7d681b2e879?

Ah, it is possibly not. We are currently on mozprocess 1.0.0 and have not yet upgraded to mozprocess 1.2.0 (https://github.com/web-platform-tests/wpt/pull/24578), but I just took the existence of this bug as a sign that the issue was not yet fixed.

Once we get 1.2.0 merged, I can re-test and re-ping if its still broken. Sorry for the noise.

I can confirm that this was my mistake; we can no longer reproduce the null-byte error in mozprocess 1.2.0. Sorry about that!

Yes, I believe this was fixed properly in bug 1585702.