I recently updated to m-c tip and did a |./mach build && ./mach package| for Fennec. This was working fine previously, but now the |./mach package| step fails because it can't find dist/bin/features/webcompat@mozilla.org/chrome/content/data/ua_overrides.jsm

Full output: kats@kgupta-pc mozilla-fennec$ ./mach package 0:00.25 /usr/bin/make -C . -j4 -s -w package 0:00.26 make: Entering directory '/home/kats/zspace/mozilla-fennec/obj-android-debug' 0:00.86 make[1]: Entering directory '/home/kats/zspace/mozilla-fennec/obj-android-debug/mobile/android/installer' 0:01.16 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:71: Missing file(s): bin/libnssdbm3.so 0:01.17 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:104: Missing file(s): bin/blocklist.xml 0:01.17 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:109: Missing file(s): bin/components/components.manifest 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:200: Missing file(s): bin/components/nsINIProcessor.manifest 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:201: Missing file(s): bin/components/nsINIProcessor.js 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:216: Missing file(s): bin/components/SystemMessageInternal.js 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:217: Missing file(s): bin/components/SystemMessageManager.js 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:218: Missing file(s): bin/components/SystemMessageCache.js 0:01.21 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:219: Missing file(s): bin/components/SystemMessageManager.manifest 0:01.75 Warning: /home/kats/zspace/mozilla-fennec/mobile/android/installer/package-manifest.in:295: Missing file(s): bin/defaults/pref/channel-prefs.js 0:01.98 Traceback (most recent call last): 0:01.98 File "/home/kats/zspace/mozilla-fennec/toolkit/mozapps/installer/packager.py", line 340, in <module> 0:01.98 main() 0:01.98 File "/home/kats/zspace/mozilla-fennec/toolkit/mozapps/installer/packager.py", line 334, in main 0:01.98 copier.copy(args.destination) 0:01.98 File "/home/kats/zspace/mozilla-fennec/python/mozbuild/mozpack/copier.py", line 430, in copy 0:01.98 copy_results.append((destfile, f.copy(destfile, skip_if_older))) 0:01.98 File "/home/kats/zspace/mozilla-fennec/python/mozbuild/mozpack/copier.py", line 603, in copy 0:01.98 file.copy(deflater, skip_if_older) 0:01.98 File "/home/kats/zspace/mozilla-fennec/python/mozbuild/mozpack/files.py", line 194, in copy 0:01.98 shutil.copyfileobj(self.open(), dest) 0:01.98 File "/home/kats/zspace/mozilla-fennec/python/mozbuild/mozpack/files.py", line 222, in open 0:01.98 return open(self.path, 'rb') 0:01.98 IOError: [Errno 2] No such file or directory: '../../../dist/bin/features/webcompat@mozilla.org/chrome/content/data/ua_overrides.jsm' 0:01.99 /home/kats/zspace/mozilla-fennec/toolkit/mozapps/installer/packager.mk:22: recipe for target 'stage-package' failed 0:01.99 make[1]: *** [stage-package] Error 1 0:01.99 make[1]: Leaving directory '/home/kats/zspace/mozilla-fennec/obj-android-debug/mobile/android/installer' 0:01.99 /home/kats/zspace/mozilla-fennec/mobile/android/build.mk:11: recipe for target 'package' failed 0:01.99 make: *** [package] Error 2 0:01.99 make: Leaving directory '/home/kats/zspace/mozilla-fennec/obj-android-debug'

Possibly regression from bug 1451484?

Does it work after a clobber?

Yes, it does.

This is unfortunate. It'd be worth digging in to find out what the root cause is.

OK, I think I see the issue. The Android package manifest has a wildcard for `features/*`: https://dxr.mozilla.org/mozilla-central/rev/c291143e24019097d087f9307e59b49facaf90cb/mobile/android/installer/package-manifest.in#268 Without a clobber there will be broken symlinks to the previous structure before the patch in dist/bin/features/*, and they will get added to the list of things to package but then the copier will fail to copy them because the symlinks are broken. We'd probably need to track changes to the contents of install manifests in dep builds and remove outdated entries to fix this.

I've noticed similar failures in the past for what I think were XBL binding removals or something like that, but never got around to filing a bug for that.

I hit this again, for the removal of the progressmeter stuff in bug 1499947. Same deal where ./mach package on Fennec failed.

This is the output, in case it's helpful: $ ./mach package 0:00.20 /usr/bin/make -C . -j4 -s -w package 0:00.20 make: Entering directory '/home/kats/zspace/gecko-fennec/obj-android-debug' 0:00.23 make[1]: Entering directory '/home/kats/zspace/gecko-fennec/obj-android-debug/mobile/android/installer' 0:00.41 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:71: Missing file(s): bin/libnssdbm3.so 0:00.41 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:104: Missing file(s): bin/blocklist.xml 0:00.41 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:109: Missing file(s): bin/components/components.manifest 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:198: Missing file(s): bin/components/nsINIProcessor.manifest 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:199: Missing file(s): bin/components/nsINIProcessor.js 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:214: Missing file(s): bin/components/SystemMessageInternal.js 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:215: Missing file(s): bin/components/SystemMessageManager.js 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:216: Missing file(s): bin/components/SystemMessageCache.js 0:00.43 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:217: Missing file(s): bin/components/SystemMessageManager.manifest 0:00.53 Warning: /home/kats/zspace/gecko-fennec/mobile/android/installer/package-manifest.in:293: Missing file(s): bin/defaults/pref/channel-prefs.js 0:00.92 Traceback (most recent call last): 0:00.92 File "/home/kats/zspace/gecko-fennec/toolkit/mozapps/installer/packager.py", line 340, in <module> 0:00.92 main() 0:00.92 File "/home/kats/zspace/gecko-fennec/toolkit/mozapps/installer/packager.py", line 334, in main 0:00.92 copier.copy(args.destination) 0:00.92 File "/home/kats/zspace/gecko-fennec/python/mozbuild/mozpack/copier.py", line 430, in copy 0:00.92 copy_results.append((destfile, f.copy(destfile, skip_if_older))) 0:00.92 File "/home/kats/zspace/gecko-fennec/python/mozbuild/mozpack/copier.py", line 603, in copy 0:00.92 file.copy(deflater, skip_if_older) 0:00.92 File "/home/kats/zspace/gecko-fennec/python/mozbuild/mozpack/files.py", line 194, in copy 0:00.92 shutil.copyfileobj(self.open(), dest) 0:00.92 File "/home/kats/zspace/gecko-fennec/python/mozbuild/mozpack/files.py", line 222, in open 0:00.92 return open(self.path, 'rb') 0:00.92 IOError: [Errno 2] No such file or directory: '../../../dist/bin/chrome/toolkit/content/global/elements/progressmeter.js' 0:00.94 /home/kats/zspace/gecko-fennec/toolkit/mozapps/installer/packager.mk:22: recipe for target 'stage-package' failed 0:00.94 make[1]: *** [stage-package] Error 1 0:00.94 make[1]: Leaving directory '/home/kats/zspace/gecko-fennec/obj-android-debug/mobile/android/installer' 0:00.94 /home/kats/zspace/gecko-fennec/mobile/android/build.mk:11: recipe for target 'package' failed 0:00.94 make: *** [package] Error 2 0:00.94 make: Leaving directory '/home/kats/zspace/gecko-fennec/obj-android-debug'

... and again. Does anybody have a workaround for this that I can apply when I hit it, so I don't have do a full clobber again?

in this case it couldn't find dist/bin/chrome/en-US/local/en-US/pippki/certManager.dtd so I just deleted the dist/bin/chrome folder and ran "./mach build && ./mach package" and that seemed to work. So I guess that's a workaround. Hopefully it didn't produce a garbage APK.

(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #11) > ... and again. Does anybody have a workaround for this that I can apply when > I hit it, so I don't have do a full clobber again? You can be pretty confident that `rm -rf dist/bin && ./mach build && ./mach package` will be fine. I think it used to be true that only `./mach package` populated `dist`, but that's no longer the case (especially after https://bugzilla.mozilla.org/show_bug.cgi?id=1255485 and similar tickets).