When running the raptor performance tests, we need to issue a pass or fail that can be used by the Try infrastructure to determine success of a test. This should allow us to fail builds based on performance testing. At the present we plan on using a percentage-based failure mechanism, e.g. if any of the performance test show a 10% regression, then issue a failure for the entire suite. We will iterate on what a good beginning failure threshold is during development.

We should also talk to the infrastructure folks about being able to use a commit or PR flag to determine that a patch can land with failed performance tests.