I have observed wide performance disparities on MacOS X when building using the cross-platform scripts vs. using xcodebuild. The attachment shows the results of such a run. The executables under test were built as follows: mkdir objdir cd objdir python ../configure.py --enable-shell make xcodebuild ARCHS=i386 -project platform/mac/avmshell/avmshell.xcodeproj -configuration Release It is likely that the default compiler options in the cross-platform scripts are suboptimal.

Not only does xcodebuild seem to give better performance overall, it also introduces additional variability in the results. While investigating bug 565489, I compared the effect of a patch with the baseline using executables built both with the cross-platform scripts and xcodebuild. With executables built using xcodebuild, we get, for example, the following: asmicro/closedvar-write-1.as 3771 3742.0 4847 4876.8 29.7 30.3 ++ asmicro/closedvar-write-2.as 4422 4379.8 3633 3644.8 -17.5 -22.7 -- With executables built using configure/make, we get the these results: asmicro/closedvar-write-1.as 4789 4698.6 4938 4942.0 3.4 2.9 + asmicro/closedvar-write-2.as 3652 3643.4 3606 3610.6 -1.0 -3.7 - It is inconceivable that the disparity observed in the first case is actually due to the (first order effect of) the patch. I have previously observed significant sensitivity of benchmark results to incidental changes in code placement in the executables, which is quite likely what we are seeing here. Based on the observations so far, I would draw the tentative conclusion that builds for performance benchmarking should be made with the cross-platform infrastructure. Clearly, we need to figure out what is going on with xcodebuild -- both why we see better performance overall but also greater sensitivity to perturbations in the code.

Performance benchmark results for the following patch: https://bug565489.bugzilla.mozilla.org/attachment.cgi?id=455231 Results shown for builds with configure/make and with xcodebuild. MacOS X (10.6.3), xcode 3.2.1 Developer Information: Version: 3.2 (10M2003) Location: /Developer Applications: Xcode: 3.2.1 (1613) Interface Builder: 3.2.1 (740) Instruments: 2.0.1 (1096) Dashcode: 3.0 (328) SDKs: Mac OS X: 10.4: (8S2167) 10.5: (9J61) 10.6: (10M2003)

i've also seen wide discrepancies with the same exact source code, Makefiles, with two clones using the commandline build. (on a clock-locked linux machine) Something's up, and to isolate that possible effect, are the xcode and commandline builds above done from the same physical repository?

(In reply to comment #3) > i've also seen wide discrepancies with the same exact source code, Makefiles, > with two clones using the commandline build. (on a clock-locked linux machine) > Something's up, and to isolate that possible effect, are the xcode and > commandline builds above done from the same physical repository? Just to second Ed's remarks here: on a different project (Larceny), I would sometimes see (wildly) different behaviors from two checkouts of the repository but otherwise identical builds. Once when it was really important to track down a particular bug, I tracked it down to the length of the paths I was using to invoke the executable and to refer to its subsidiary components, like its heap image. Once I put the checkouts into analogous directory structures, I got identical behavior. So even if you are using the same repository, you may want to also ensure that the build object subdirectories holding the executables and such are located at analogous places in the directory substructure. (One might go overboard here, and start wondering if the hash codes of the paths matter...)

QE: Can you take a crack at isolating the issue(s) at play here? I would expect the same default compiler flags to be used in both build systems, even if this is not a big factor here.

May not need to investigate this if we adopt gyp and drop the xplat system.

Tamarin isn't maintained anymore. WONTFIX remaining bugs.