https://treeherder.mozilla.org/logviewer.html#?job_id=5424148&repo=mozilla-central#L33682 not sure where it fails but if affect both android nightlys it seems

Callek: i will retrigger android nightlys in the hope that fix this, but in any case could you take a look, thanks!

04:17:24 INFO - 07:17:24 INFO - WARNING: Found 50 duplicated files taking 347909 bytes (219451 compressed) 04:17:24 INFO - 07:17:24 INFO - ERROR: The following duplicated files are not allowed: 04:17:24 INFO - 07:17:24 INFO - chrome/en-US/locale/en-US/browser/devicePrompt.properties 04:17:24 INFO - 07:17:24 INFO - chrome/pt-PT/locale/pt-PT/browser/devicePrompt.properties 04:17:24 INFO - 07:17:24 INFO - chrome/en-US/locale/branding/brand.properties ... Lots of entries like that, which partly worries me either in that we're packaging far more than <locale> *or* that the checker script is bad when used with l10n. I suspect the retrigger will not solve anything. :catlee, is there an easy fix in your mind, or should we backout Bug 1303184

Richard, are there plans to remove the duplicate files as part of multilocale builds somehow?

(In reply to Justin Wood (:Callek) from comment #2) > Lots of entries like that, which partly worries me either in that we're > packaging far more than <locale> *or* that the checker script is bad when > used with l10n. Admittedly when I first looked at this I didn't realize these were multi-locale builds. So it is indeed (and expectedly) packaging more than one locale in one go. I'm still waiting on results for other sets of l10n with regard to this.

Android builds are indeed multilocale. I don't know if that dupe checker is spotting actual duplicate file contents, rather than repeated path subtrees; if so, I'd be happy to see a bug -- with analysis of space savings when packaged -- flattening those down to en-US. (Obviously the checker would still have to not warn for peer locale dupes that don't dupe to the root locale…)

It's checking for actual duplicated content. Most of these are already present in en-US, but not all. From the log (https://treeherder.mozilla.org/logviewer.html#?job_id=5424148&repo=mozilla-central#L33514), it looks like there's a total of 347909 bytes (219451 compressed) that are duplicated.

(In reply to Richard Newman [:rnewman] from comment #6) > I don't know if that dupe checker is spotting actual duplicate file > contents, rather than repeated path subtrees; if so, I'd be happy to see a > bug -- with analysis of space savings when packaged -- flattening those down > to en-US. If you search for logs from a completed build that runs find-dupes.py, it prints said statistics. e.g. "04:11:59 INFO - WARNING: Found 34 duplicated files taking 87540 bytes (30893 compressed)" (this was from an Android Nightly from this week).

Comment on attachment 8805555 [details] Bug 1313640 - Add support for allowing duplicate files in multilocale builds. https://reviewboard.mozilla.org/r/89322/#review88484 ::: toolkit/mozapps/installer/find-dupes.py:285 (Diff revision 1) > > +def normalize_l10n_path(p): > + ''' > + Normalizes localized paths to en-US > + > + >>> normalize_osx_path('chrome/es-ES/locale/branding/brand.properties') I think this wants to be `normalize_osx_path` ::: toolkit/mozapps/installer/find-dupes.py:288 (Diff revision 1) > + m = re.search(r'chrome/(\S+)/locale', p) > + if m: > + locale = m.group(1) > + p = p.replace(locale, 'en-US') > + return p This can be written as: return re.sub(r'chrome/(\S+)/locale', 'chrome/en-US/locale', p) This is safer than what's written because it won't replace all occurrences of the locale with `en-US`: only the ones in the `chrome/.../locale` pattern.

Comment on attachment 8805555 [details] Bug 1313640 - Add support for allowing duplicate files in multilocale builds. https://reviewboard.mozilla.org/r/89322/#review88484 > This can be written as: > > return re.sub(r'chrome/(\S+)/locale', > 'chrome/en-US/locale', > p) > > This is safer than what's written because it won't replace all occurrences of the locale with `en-US`: only the ones in the `chrome/.../locale` pattern. That doesn't catch all occurrences of locale strings in paths. There are many of the pattern 'chrome/$AB_CD/locale/$AB_CD' as well.

I filed Bug 1313702 to investigate skipping/stripping some of these duplicates at build time.

I can't get mozreview to re-request review. gps, can you please look when you get a chance?

can we fix this asap since this is hitting nightly user and so crash-data etc :(

Just to clarify: we have no crash reports for the Oct 28, 29, 30 Fennec builds. This is bad :(

Comment on attachment 8805555 [details] Bug 1313640 - Add support for allowing duplicate files in multilocale builds. https://reviewboard.mozilla.org/r/89322/#review88906 I tested this out locally by making a dummy directory with the duplicate files that are failing the Android nightly builds and running find-dupes.py manually. With this fix applied, the error goes away.

Comment on attachment 8805555 [details] Bug 1313640 - Add support for allowing duplicate files in multilocale builds. https://reviewboard.mozilla.org/r/89322/#review88940

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/a352e8d4b5d9 Add support for allowing duplicate files in multilocale builds. r=gps,mshal

OK well this is ridiculous. we go days at a time with no Android nightly builds because of this. Staring this to this bug is NOT an acceptable way to deal with this. we need to get nightly builds created in a timely manner.

Also I have no idea what the certificate issue that has been starred as a duplicate of this issue has to do with it.