The following run: % python runtests.py --csv=foo --vmargs "-Dnogc" \ asmicro/alloc-1.as jsbench/SOR.as asmicro/alloc-2.as produces output to the screen that tells me that the tests fail with exit code: 1. That's good. It also generates the following CSV file: avm,testname,metric,metricunit,iteration,value avm:cyclone,asmicro/alloc-2.as,v8,,1,14 avm:cyclone,asmicro/alloc-1.as,v8,,1,29 The problem: the generated CSV file is missing the line corresponding to the failed runs. If I don't watch the output or check the return status for runtests.py, then the CSV file does not have a record of the failure (other than the missing line). Such output is difficult to compare to other CSV files that include the line for a successful run of that particular program. (Well, its not ever fun to compare CSV files -- my point is that when importing into a spreadsheet, you need to massage the data to make the number of rows match up.) I admit that it is questionable whether we want to generate a CSV file in this case at all. I think we should, because some of my runs in crazy modes like -Dnogc do break benchmarks that need a lot of memory, but I still might want to analyze the data for the other programs without having to do a fresh run that filters out the failing programs. But I could certainly cope if we simply decided to not generate a CSV file at all in this case. Anyway its a minor issue but I wanted to note it before I forgot about it. (I'm going to work around it in the mean time by manually filtering out the failures.)

Tamarin is a dead project now. Mass WONTFIX.

Tamarin isn't maintained anymore. WONTFIX remaining bugs.