[meta] Run tests from source checkouts in CI
Categories
(Testing :: General, enhancement, P3)
Tracking
(Not tracked)
People
(Reporter: gps, Unassigned)
References
(Depends on 2 open bugs, Blocks 2 open bugs)
Details
(Keywords: meta)
Updated•6 years ago
|
Updated•6 years ago
|
Comment 1•5 years ago
|
||
nalexander: fyi
Comment 2•5 years ago
|
||
(In reply to Bob Clary [:bc:] from comment #1)
nalexander: fyi
Thanks, Bob. Just for history's sake, I got source checkouts working for Raptor jobs, and found that on Windows just doing a run-task
with checkout: true
takes 9-10 minutes. (I don't think a sparse profile would help that much here, either.) What ever this "incremental" HG operation is/could be, it is absolutely essential, 'cuz just using HG naively is orders of magnitude slower than the mozharness.zip
stuff.
Comment 3•5 years ago
|
||
I think this bug is predicated on the checkouts being cached on the workers. If raptor is running on a worker type that doesn't already have a cached checkout, then the initial overhead will be large. As more workers claim these types of tasks, then we should theoretically start seeing faster setup times.
There are lots of variables at play here though. If the cache gets too large, taskcluster will just delete the older stuff. I'm also not sure what happens to a cache if an instance is killed. I don't believe anyone took the time to make sure that our caching strategy on Windows is working as expected.
Comment 4•5 years ago
|
||
also switching between esr60, beta, trunk, etc. takes a long time, for me locally on windows with a SSD, it takes ~5 minutes to switch from esr68 to trunk, imagine esr60 :)
good news for perf is that we only run perf on beta/trunk, that is not as extreme, for all other unittests, this is something we need to support.
Comment 5•5 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #3)
I think this bug is predicated on the checkouts being cached on the workers. If raptor is running on a worker type that doesn't already have a cached checkout, then the initial overhead will be large. As more workers claim these types of tasks, then we should theoretically start seeing faster setup times.
In fact, the one thing I needed to do was to put HG_STORE_PATH
somewhere that wasn't on a cache, since the Win 10 HW devices don't have the default cache location.
There are lots of variables at play here though. If the cache gets too large, taskcluster will just delete the older stuff. I'm also not sure what happens to a cache if an instance is killed. I don't believe anyone took the time to make sure that our caching strategy on Windows is working as expected.
Thanks for the context. I'm still concerned that just the hg checkout
I/O will dwarf any benefits we might get. I did wonder about using the .tar.gz
and/or .zip
HG endpoints to do a "super sparse" checkout, which would actually make sense for Raptor. (Server load and the complexity that is https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/test_archive.py are why it wouldn't make sense more generally. If I'm wrong about server load (in either direction), please feed that back to me!)
Updated•4 years ago
|
Updated•2 years ago
|
Description
•