User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0

Steps to reproduce:

Start Nightly, get a message that profiles are now separated.
Attempt to start a release Firefox (/usr/bin/firefox), get a new window in the Nightly instance.
Start /usr/bin/firefox --new-instance, get a new instance of release Firefox.

Thus far, all is good.

Close Nightly and attempt to start a new instance of it while Release is still running:
/home/l/lars/local/bin/firefox --new-instance

Actual results:

A message box pops up with the good old message: "Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system."

Expected results:

A new Nightly instance should launch.

Doesn't sound right, but we should re-check once bug 1518639 is fixed.

With bug 1518639 landed, this issue is fixed & verified. Basically, now it is possible to have release(or any other different install) opened and just open Nightly, even if it wasn't opened before to generate it's dedicated profile.

Also, the --new-instance parameter is not needed for this particular scenario. Just calling ~/Documents/release/firefox and ~/Documents/nightly/firefox will automatically trigger the two separate instances.
The only note I would add here, in light of using the --new-instance parameter is that you cannot run the same install with two profiles without using the profile manager((e.g. calling ~/Documents/nightly/firefox --new-instance and ~/Documents/nightly/firefox --new-instance), in which case you will still hit the reported error message, but that would be the expected result.
To correctly start two nightly instances with the same installation path would be to:
~/Documents/nightly/firefox -p (select first profile to use)
~/Documents/nightly/firefox -p (select second profile to use)