Sometimes a push only contain changes to tests. Waiting for another build phase could be painful. I hope we can skip build phase, and reuse build result from another push directly. (Probably a short build phase for packaging tests is still needed, but I suppose that would be much quicker than building everything again)

I've done this for *only* one build and *only* for TaskCluster: https://hg.mozilla.org/try/rev/a69ca3a7a59e

Sometimes the only change is the try syntax for a different job set. It would be very helpful if it could reuse the previous build result in that case.

I believe we can build this on the TaskCluster side and not worry about Buildbot. If we only see tests being touched in the list of files we could have a task which only creates the test packages and triggers the test jobs. Re-packaging only the test zip bundles will take a bit of build work. I don't "make package-tests" (if that is the command) works without calling other build steps first. Another approach would be if test jobs used a Gecko checkout, thus, making repackaging tests unnecessary and we could make the test jobs use a different taskId to download the Firefox builds from. We will also need to solve an easy way to grabbing builds from another push. Maybe a web app. Maybe special try syntax (e.g. try: --use-build-from <hg_revision>".

We have bug 1234728 filed on running tests from the srcdir, FYI.

From bug 1268481 ################# It would be great to iterate on an existing build on try and only make test changes. I'm going to track this under the awesome try tracking bug, however, this is more easily implemented after we change to a checkout model for testing (bug 1234728). We can teach |mach try| wish revision to continue from and we can modify the TaskCluster graph to include Buildbot jobs (mozci can do this for us without .yml files). > On Tue, Apr 26, 2016 at 03:49:11PM +0200, Gabor Krizsanits wrote: >> As someone who was high on the list of try server usage for two >> weeks.... My problem was a test I tried to fix for both e10s and >> non-e10s, and it timed out _sometimes_ on _some_ platforms even >> depending on debug/release build. It was a whack-a-mole game by >> fiddling with the test and a complex patch. I did stop old builds but >> I did not run only the test in question but the rest of them as well >> because of the invasive nature of the patch the whole thing was >> sitting on. Probably I could have been smarter, BUT... >> >> What would have helped me a lot in this case and most cases when I >> rely on the try server is the ability to push a new changeset on top >> of my previous one, and tell the server to use the previous session >> instead of a full rebuild (if there is only a change in the tests >> that's even better, no rebuild at all) and then tell the server >> exactly which tests I want to re-run with those changes (as it can be >> an empty set this can be used to trigger additional tests for a >> previous push). This could all be done by an extensions to the try >> syntax like -continue [hash]. As an addition this follow up push would >> also kill the previous job. >> >> Maybe there is already such functionality available, just I'm not >> aware of it (I would be so happy if this were the case, and would feel >> bad for the machine hours I wasted...), if so please let me know. > Another approach would be to let the gecko decision task take care of it (instead of |mach try|).

This would require some changes to taskcluster/taskgraph/decision.py and the mach try command. We can probably have an extra try syntax like --use-builds-from <changeset> The decision task can define the test jobs to grab the builds from the right task from the other changeset. We would also need to change the behaviour of the build jobs to skip the build command and only focus on repackaing the tests zips.

xidorn: Just to be clear, you're specifically speaking of try, correct? Optimization: Maybe |mach try| when used with --reuse-build without a changeset will show the user the last N commits for a user and be able to select with an int as an answer (rather than having to copy/paste the changeset from somewhere else)

There's not a clear path forward here, so for now we're going to close this.

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #8) > xidorn: Just to be clear, you're specifically speaking of try, correct? Yeah, try. > Optimization: Maybe |mach try| when used with --reuse-build without a > changeset will show the user the last N commits for a user and be able to > select with an int as an answer (rather than having to copy/paste the > changeset from somewhere else) That is optional. I don't want that to block the functionality. I can write down the build I want to use manually. It might be an improvement if I can choose from commits I pushed. But that info should probably be stored and queried locally rather than querying the server, otherwise the time it takes for querying that could be more annoying than do that manually. (In reply to Jonathan Griffin (:jgriffin) from comment #9) > There's not a clear path forward here, so for now we're going to close this. Why?! I think armenzg already has some idea about how to do that, no? Having this would make debugging tests, especially debugging intermittent failures significantly faster, and also saves tons of infra resources.

To iterate successfully on test changes without creating a new build, we'd have to have the ability to run |make package-tests| without invoking the rest of the build. We don't currently have that ability, and making that work is likely a significant amount of work. Instead, we're focusing efforts on allowing tests to be run from a source checkout; once that's implemented, we'll no longer need to package tests, and then a sort of 'skip build' functionality will become more practical. Feel free to re-open this bug if you want to wait for the above; regardless it will be some time before we implement.

Let's reopen this bug. I can wait, and I've been waiting for over half year, so it doesn't matter that would take longer :) Probably you can add the "test from source checkout" bug as its dependency? (And I'm not fighting with intermittent failures recently, so it's not something urgent to me.)

The main reason of wontfixing this bug is because we're working instead on artifact builds for automation (bug 1284882). This makes builds very short and quickly gets to packaging, thus, making builds be done in a handle of minutes. Having the source check out would make fixing this bug easier, however, we discussed this yesterday in our meeting and we're not going to focus on this since we expect artifact builds to accomplish the same (reduce end to end time), thus, wontfixing this. If when artifacts builds on automation is live is not enough we can re-consider this bug. I would prefer WONTFIXing until then.

How can you get proper artifacts for a given build? Does that work even if the try push based on a revision which does have source code change (rather than only test change)?

I don't know. It's a question that chmanchester can probably answer.

This works for many source code changes that don't interact with compiled code. For more information see: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds

So if I change any code need to be compiled, and then push multiple try run, each of they would still need a build individually, and cannot reuse build from existing try build? That sounds very restricted. It might be enough for debugging frontend tests, but may not be very useful for platform tests.

IIUC the work to make sccache work for try will improve the non-frontend cases (bug 1286934).