Closed Bug 703559 Opened 13 years ago Closed 3 years ago

Remove old firefox builds from latest-mozilla-*/

Categories

(Release Engineering :: General, enhancement, P3)

Product:
Release Engineering
Component:
General
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: whimboo, Unassigned)

References

Details

(Whiteboard: [upload][cleanup] Do not cleanup up firefox/nightly/latest-mozilla-aurora until we ship a week after merge)

All latest folders on the FTP server still have builds listed from before the last version bump. Can you please remove those? http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/ http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/ http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-1.9.2/ Instead of having to request it each time when a version bump happens, can we do this automatically in the future?
Would be nice if we can get this fixed. It's breaking our download script for Mozmill tests, because we usually expect only a single latest build in this folder, and now we always retrieve e.g. the broken 3.6.24pre nightly build from Oct 29th instead of 3.6.25pre. Thanks.
We know this is a longstanding problem. To my knowledge, we've never figured out a way to automatically clean up these directories. Until we do, I suggest updating your scripts to look for the later version.
(In reply to Ben Hearsum [:bhearsum] from comment #2) > We know this is a longstanding problem. To my knowledge, we've never figured > out a way to automatically clean up these directories. Until we do, I > suggest updating your scripts to look for the later version. FWIW we could probably adapt the code we use for cleaning up beta/nightly latest* as I saw in use from Bug 703828 c#1 Namely: VERSION=`wget --quiet -O- http://hg.mozilla.org/releases/mozilla-aurora/raw-file/default/browser/config/version.txt` && nice -n 19 find /home/ftp/pub/firefox/tinderbox-builds/mozilla-aurora-l10n -type f -name "firefox*" ! -name "*$VERSION*" -mtime +0 -exec rm -rf {} \;
I'll deal with the current duplicates.
Assignee: nobody → nrthomas
Status: NEW → ASSIGNED
Priority: -- → P2
All the {firefox,xulrunner}/nightly/latest* have been cleaned up. Leaving open for a general fix. I still reckon post_upload.py can parse out the version and do this, but I haven't actually tried to write that code.
Assignee: nrthomas → nobody
Status: ASSIGNED → NEW
Priority: P2 → P3
Component: Release Engineering → Release Engineering: Automation
Priority: P3 → --
QA Contact: release → catlee
Whiteboard: [upload][cleanup]
Severity: normal → enhancement
Priority: -- → P3
Nick, that task would have to run another time. All the latest* folders contain old builds again due to the version bump. Thanks.
{firefox,xulrunner}/nightly/latest-* have been cleaned up again.
It's also important to note that this also needs to be fixed for ESR10 nightlies. With the version bumps we have a lot of old builds in there. Can someone please clean those up?
esr10 and elm have been cleaned up.
One more round. This is necessary again for all related branches affected by the merge.
However we solve this, we have to cope with Aurora around merge-time. mozilla.org [1] points to the old version until QA signs off on the merge, so normally there's a period from the pre-merge Monday until about Friday where the old builds have to persist. [1] http://www.mozilla.org/en-US/firefox/aurora/
Did another round of manual cleanup.
The 6 weeks cycle came across again. Can someone please purge those folders? Thanks.
Done for b2g, firefox, xulrunner.
I could've sworn we had a cronjob for this now...
No, and it's time again for getting rid of older builds after the merge.
IIRC, it happens automatically, just takes a few days.
Got any pointers to where that's happening ? AFAIK it's just me going "oh, it's a few days after a merge, time to go delete again".
Ok, I guess that's just mobile currently for some reason.
Hurray! It's deletion time again! :)
Done, now that Aurora is signed-off after the merge. Also removed a bunch of stale directories like latest-birch, latest-elm, latest-fx-team (bug 855708), latest-maple, and jsshell-win64-x86_64.zip which was discontinued on many branches. Also did * in mobile/nightly/ removed a lot of empty latest-* dirs, eg latest*-xul, latest-mozilla-central-linux-l10n/ * cleaned up xulrunner/nightly/latest*
Product: mozilla.org → Release Engineering
Once more we have to do this manually. YAY! Anyone? :)
Nick, old builds from the merge beginning of August are still around. Can we please get those removed? Thanks.
Flags: needinfo?(nthomas)
Done.
Flags: needinfo?(nthomas)
I've done the cleaning after last weeks merge.
Hurray, the 6 weeks are over again and another cleanup is necessary. Any chance we can get this automated?
I cleaned up the builds. We've tried to automate this in the past, and it's harder than it seems. If someone comes up with a script that can safely and accurately remove the correct builds, it would be easy to deploy.
Do you have any hints for new people how to get started? Should we make it a mentored bug?
I can't commit to mentoring on this...but what I'd suggest is: - Make a directory structure like latest-mozilla-central (with files of two versions) - Write a script that will remove the older versioned of each file when multiple versions are present (this is the tricky part, because different files come in at different times for different platforms). Bash is fine. Python is fine. - Script needs to be non-interactive, runnable through cron, support different products (firefox & thunderbird at least - hopefully we don't need product names hardcoded though). - Script should accept the directory to clean as an argument - no need to hardcode path names
See comment #12 for a special case for mozilla-aurora-latest/.
If we uploaded per-platform, we could upload and softlink a new directory every time, and we wouldn't have to worry about cleaning up old bits. This would change latest-mozilla-central to latest-mozilla-central-linux64, for example.
Whiteboard: [upload][cleanup] → [upload][cleanup] Do not cleanup up firefox/nightly/latest-mozilla-aurora until we ship a week after merge
It's time again, can someone please clean up? Thanks.
Hurray, surprisingly we hit this problem again. It's time for clearing up the folders. :)
Done, including a few latest-<branch> dirs which hadn't had new builds in > 1 month.
It's time again... Nick, can you take care of it again please?
Aki did this cycle, a parting gift.
OMG there was a release again! Please do a cleaning.
I cleaned up the firefox and mobile directories. Thunderbird seems like it was already done. I think that's everything...
Done again.
Done again.
Done again.
It's time to do a cleanup. Looks like we missed it for the last merge.
Cleanup done.
Can someone please clean-up again? Thanks.
Done, now that we have re-enabled Aurora updates.
It's time again for clean-up.
Done, except for aurora until we turn on updates there and so on.
It's time for another cleanup.
Nick, it would need another clobber.
Done. Gonna have to script this for S3, it's rather more painful.
It's time again for a clobber.
http://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ has a collection of 44.0a1 and 45.0a2 still from mid-December. (there should only be 46.0a1) http://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-aurora/ has a collection of 43.0a2, and 44.0a2 (there should only be 45.0a2) ditto -l10n directories for both and http://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-esr38/ has old builds, and is MISSING 38.5.0
It's time again to get old files removed. Nick, can you please do it? Thanks.
Its a good idea but two things... * How can we make it easier for people to find those links? Could we put a link into those folder which points to the builds like: https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-central.latest.firefox/gecko.v2.mozilla-central.latest.firefox ? * I haven't added taskcluster support to mozdownload yet given that all was working fine for archive.mozilla.org after we switched to S3 and no-one complained about more work to do. So I will put this on my list of things to do, but not sure if I can get to it very soon. For now I created: https://github.com/mozilla/mozdownload/issues/365
It's time again... actually we missed to cleanup after the last release. So we have 3 versions now in the latest folders.
Anyone who could do this again? We still have builds from the former train in our latest folders.
I'd already cleaned up this cycle, but some builds+nightlies were run on a staging buildbot master and publishing into production. Fixed up firefox/nightly/latest-mozilla-{central,aurora}/.
Component: General Automation → General
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.