They don't need to actually work (although they might), they just need to be there in /Applications/.

I was able to install Xcode 7.3.1 on the worker #139 (and without switching the default /Applications/Xcode.app to link to them).

Installing 8.3.3 succeeded but shows some warnings:

[10:27:38]: Successfully verified Xcode installation at path '/Applications/Xcode-8.3.3.app' 🎧
dyld: Symbol not found: _OBJC_IVAR_$_NSFont._fFlags
  Referenced from: /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
  Expected in: /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
 in /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
dyld: Symbol not found: _OBJC_IVAR_$_NSFont._fFlags
  Referenced from: /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
  Expected in: /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
 in /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit

I'll check+test if I can install the matching xcode commandline tools also.

[relops@macmini-r8-139.test.releng.mdc1.mozilla.com ~]$ /Applications/Xcode-8.3.3.app/Contents//Developer/usr/bin/xcodebuild -version                                                         
dyld: Symbol not found: _OBJC_IVAR_$_NSFont._fFlags
  Referenced from: /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
  Expected in: /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
 in /Applications/Xcode-8.3.3.app/Contents/SharedFrameworks/DVTDocumentation.framework/Versions/A/../../../../SharedFrameworks/DVTKit.framework/Versions/A/DVTKit
Abort trap: 6
[relops@macmini-r8-139.test.releng.mdc1.mozilla.com ~]$ /Applications/Xcode-7.3.1.app/Contents//Developer/usr/bin/xcodebuild -version
Xcode 7.3.1
Build version 7D1014

As mentioned in comment #0, they don't need to work. We don't need the command line tools either.

For the failures in bug 1686307,
xcode-select showed the system was still configured to use Xcode 12.2 as default and that the commandline tools were for 12.2, but something was still causing 8.3.3 to be picked up first and cause failures.

I tried removing Xcode 8.3.3 and re-running a failed task (on macmini-r8-198) and it passed. I am running another task and will leave it un-quarantined to see if the problem re-appears on other tasks or if deleting 8.3.3 fixed the problem.

#198 continued working without problems (with 8.3.3 xcode deleted, and it did not have 7.3.1 xcode installed). I tested on #139 also, and verified it failed with only 7.3.1 xcode (no 8.3.3 xcode) with the same "Could not find a suitable make implementation.".

Can you try to pkgutil --expand the .pkgs instead of trying to install them?

I expanded the Xcode 7.3.1 and 8.3.3 packages under /Applications on the worker #139 and a re-trigger of a recent task passed on there. I'm setting it up on a second, 198, and will confirm a task passes on that also.

I've started copying the 7.3.1 and 8.3.3 files to the level-1 workers.
#138 has a full copy. I'll make sure it continues completing tasks without problems and then continue by copying to the others.

copying between the machines like (instead of doing the extraction from the dmgs):

tar -c /Applications/Xcode-8.3.3.app | ssh macmini-r8-138.test.releng.mdc1.mozilla.com 'sudo tar -xf - -C /'

:glandium the 7.3.1 and 8.3.3 xcode (extracted not installed) is on all of the releng-hardware gecko-1-b-osx-1015 and gecko-3-b-osx-1015 workers now.

There were no task failures yesterday on the workers with the copied Xcode 7.3.1 and 8.3.3, and so I copied the files to the other level 1 and 3 workers also.

Looks like it works. Please ensure that whatever happens in the future (adding new workers in those pools, migrating to new workers, etc.), we still have these available (likewise for the existing Xcode).

Closing per bug scrub.