MozillaBuild 2.2.0pre4 on Windows 8.1 64 bit Executing any hg command seems to let. Hadn't used previous 2.2.0pre versions.

Okay, also happens with MozillaBuild 2.1.0, might just have been more prominent due to the big backout.

Add --profile to any command and pastebin the output.

Or upload the output. Or paste inline.

Hit this memory issue with inbound on 127e74cb33c3 when I tried to back out https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=3160ce8f87590a3a611267595cb478479dfdb2f3 : $ hg qbackout -r 3160ce8f8759:c4827087a333 --profile checking for uncommitted changes backing out 3160ce8f8759 applying patch from stdin backing out aa209325161f applying patch from stdin backing out d48bc6eca9c3 applying patch from stdin backing out 945141b1f8da applying patch from stdin backing out 48da132353c7 applying patch from stdin backing out 80443106b263 applying patch from stdin backing out 795bcfd6223d applying patch from stdin backing out 30c253e762c6 applying patch from stdin backing out 95412432bf10 applying patch from stdin backing out 38e3e48c8dd0 applying patch from stdin backing out 72cf70065470 applying patch from stdin CallCount Recursive Total(s) Inline(s) module:lineno(function) 124586 0 10.1973 10.1973 <mercurial.osutil.listdir> 1697957 0 7.8454 5.5950 ntpath:174(split) 165042 0 4.7737 4.7737 <zlib.decompress> 1697519 0 25.9577 4.2976 mercurial.windows:327(statfiles) 23 0 31.3874 1.9461 mercurial.dirstate:1093(status) 34 0 29.4656 1.5931 mercurial.dirstate:931(walk) 1701593 0 2.1625 1.5565 ntpath:96(splitdrive) 290 0 1.4594 1.4594 <method 'sort' of 'list' objects> 3400865 0 1.0169 1.0169 <mercurial.parsers.asciiupper> 3400865 0 2.7770 1.0041 mercurial.windows:223(normcase) 1704070 0 0.9919 0.9919 <method 'replace' of 'str' objects> 21 0 0.9038 0.8926 <method 'difference_update' of 'set' objects> 640 0 0.8033 0.8033 <method 'update' of '_hashlib.HASH' objects> 21 0 1.7584 0.7667 mercurial.manifest:212(filesnotin) 24 0 0.7609 0.7609 <mercurial.parsers.parse_dirstate> 3400865 0 1.7729 0.7560 mercurial.encoding:343(upper) 13308 0 0.6490 0.6490 <method 'read' of 'file' objects> 3398796 0 0.6096 0.6096 <method 'get' of 'dict' objects> 24 0 0.5658 0.5658 <mercurial.parsers.pack_dirstate> 235 0 0.5269 0.5269 <mercurial.mpatch.patches> 54 0 0.5394 0.4189 mercurial.dirstate:691(write) 9 0 1.3326 0.4185 mercurial.store:403(_load) 1697670 0 0.3992 0.3992 mercurial.dirstate:230(_join) 74 0 0.3799 0.3798 mercurial.manifest:176(__init__) 2522 0 0.2960 0.2960 mercurial.revlog:353(rev) 13901 0 0.2619 0.2619 <method 'write' of 'file' objects> 54 0 0.2535 0.2535 <method 'splitlines' of 'str' objects> 85 0 7.1321 0.2517 mercurial.manifest:962(read) 501 0 0.2469 0.2469 <nt.rename> 236 0 5.3348 0.2354 mercurial.revlog:1101(_chunks) abort: out of memory

(In reply to Sebastian H. [:aryx][:archaeopteryx] from comment #4) > Hit this memory issue with inbound on 127e74cb33c3 when I tried to back out > https://treeherder.mozilla.org/#/jobs?repo=mozilla- > inbound&revision=3160ce8f87590a3a611267595cb478479dfdb2f3 : > > $ hg qbackout -r 3160ce8f8759:c4827087a333 --profile "::" is the operator you want here as ":" may select revisions you didn't want to select. See `hg help revset` for the differences. I'm kinda surprised we got an out of memory error after only ~30s. Can you try `hg oops` (instead of `hg qbackout`) to try to remove MQ from the equation? Also, please paste the output of `hg qbackout --traceback <args>` to get a stack trace of the thing allocating the memory.

Thank you for the information. An OOM also occurs with |hg oops|. >$ hg qbackout --traceback -r c4827087a333::3160ce8f8759 >checking for uncommitted changes >backing out 3160ce8f8759 >applying patch from stdin >backing out aa209325161f >applying patch from stdin >backing out d48bc6eca9c3 >applying patch from stdin >backing out 945141b1f8da >applying patch from stdin >backing out 48da132353c7 >applying patch from stdin >backing out 80443106b263 >applying patch from stdin >backing out 795bcfd6223d >applying patch from stdin >backing out 30c253e762c6 >applying patch from stdin >backing out 95412432bf10 >applying patch from stdin >backing out 38e3e48c8dd0 >applying patch from stdin >backing out 72cf70065470 >applying patch from stdin >Traceback (most recent call last): > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 191, in _runcatch > return _dispatch(req) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 924, in _dispatch > cmdpats, cmdoptions) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 681, in runcommand > ret = _runcommand(ui, options, cmd, d) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\pager.py", line 143, in pagecmd > return orig(ui, options, cmd, cmdfunc) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\color.py", line 518, in colorcmd > return orig(ui_, opts, cmd, cmdfunc) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 1055, in _runcommand > return checkargs() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 1015, in checkargs > return cmdfunc() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 921, in <lambda> > d = lambda: util.checksignature(func)(ui, *args, **cmdoptions) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 3517, in mqcommand > return orig(ui, repo, *args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 126, in qbackout > **opts) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 252, in do_backout > commit_change(ui, repo, desc['name'], node=node, force_name=opts.get('name'), **new_opts) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 120, in commit_change > mq.new(ui, repo, patchname, **opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 2587, in new > q.new(repo, patch, *args, **opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 1205, in new > force=True, editor=editor) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 390, in newcommit > return repo.commit(*args, **kwargs) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 1042, in commit > return super(reviewboardrepo, self).commit(msg, *args[1:], **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 3420, in commit > editor, extra) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 103, in wrapper > return orig(repo.unfiltered(), *args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 1520, in commit > merge = len(wctx.parents()) > 1 > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 240, in parents > return self._parents > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 723, in __get__ > result = self.func(obj) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1394, in _parents > p = self._repo.dirstate.parents() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dirstate.py", line 324, in parents > return [self._validate(p) for p in self._pl] > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 526, in _dirstatevalidate > self.changelog.rev(node) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 69, in __get__ > return super(repofilecache, self).__get__(repo.unfiltered(), type) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\scmutil.py", line 1196, in __get__ > entry.obj = self.func(obj) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 505, in changelog > c = changelog.changelog(self.svfs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\changelog.py", line 141, in __init__ > revlog.revlog.__init__(self, opener, "00changelog.i") > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 271, in __init__ > indexdata = f.read() >MemoryError >abort: out of memory oops: >$ hg oops --traceback -r c4827087a333::3160ce8f8759 >checking for uncommitted changes >backing out 3160ce8f8759 >applying patch from stdin >created new head >backing out aa209325161f >applying patch from stdin >backing out d48bc6eca9c3 >applying patch from stdin >backing out 945141b1f8da >applying patch from stdin >backing out 48da132353c7 >applying patch from stdin >backing out 80443106b263 >applying patch from stdin >backing out 795bcfd6223d >applying patch from stdin >backing out 30c253e762c6 >applying patch from stdin >backing out 95412432bf10 >applying patch from stdin >backing out 38e3e48c8dd0 >applying patch from stdin >backing out 72cf70065470 >applying patch from stdin >backing out 8cffa26910a8 >applying patch from stdin >Traceback (most recent call last): > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 191, in _runcatch > return _dispatch(req) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 924, in _dispatch > cmdpats, cmdoptions) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 681, in runcommand > ret = _runcommand(ui, options, cmd, d) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\pager.py", line 143, in pagecmd > return orig(ui, options, cmd, cmdfunc) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\color.py", line 518, in colorcmd > return orig(ui_, opts, cmd, cmdfunc) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 1055, in _runcommand > return checkargs() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 1015, in checkargs > return cmdfunc() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dispatch.py", line 921, in <lambda> > d = lambda: util.checksignature(func)(ui, *args, **cmdoptions) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\extensions.py", line 195, in closure > return func(*(args + a), **kw) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 3517, in mqcommand > return orig(ui, repo, *args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 991, in check > return func(*args, **kwargs) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 312, in oops > **opts) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 252, in do_backout > commit_change(ui, repo, desc['name'], node=node, force_name=opts.get('name'), **new_opts) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext\qbackout\__init__.py", line 307, in commit_change > commands.commit(ui, repo, **opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\commands.py", line 1639, in commit > return _docommit(ui, repo, *pats, **opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\commands.py", line 1719, in _docommit > node = cmdutil.commit(ui, repo, commitfunc, pats, opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\cmdutil.py", line 2503, in commit > return commitfunc(ui, repo, message, matcher, opts) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\commands.py", line 1713, in commitfunc > extra=extra) > File "c:/Users/Eichelhaeher/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 1042, in commit > return super(reviewboardrepo, self).commit(msg, *args[1:], **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\hgext\mq.py", line 3420, in commit > editor, extra) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 103, in wrapper > return orig(repo.unfiltered(), *args, **kwargs) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 1526, in commit > status = self.status(match=match, clean=force) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\localrepo.py", line 1826, in status > listsubrepos) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 351, in status > listunknown) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1624, in _buildstatus > s.modified[:] = self._filtersuspectsymlink(s.modified) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1519, in _filtersuspectsymlink > if self.flags(f) == 'l': > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1321, in flags > return self._flagfunc(path) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 723, in __get__ > result = self.func(obj) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1209, in _flagfunc > return self._repo.dirstate.flagfunc(self._buildflagfunc) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\dirstate.py", line 249, in flagfunc > fallback = buildfallback() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 1183, in _buildflagfunc > man = parents[0].manifest() > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 180, in manifest > return self._manifest > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\util.py", line 723, in __get__ > result = self.func(obj) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\context.py", line 531, in _manifest > return self._repo.manifest.read(self._changeset[0]) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\manifest.py", line 977, in read > text = self.revision(node) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 1202, in revision > bins = self._chunks(chain, df=_df) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 1126, in _chunks > offset, data = self._chunkraw(revs[0], revs[-1], df=df) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 1088, in _chunkraw > return start, self._getchunk(start, length, df=df) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 1063, in _getchunk > return self._loadchunk(offset, length, df=df) > File "c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\revlog.py", line 1033, in _loadchunk > d = df.read(reallength) >MemoryError >abort: out of memory

I'm unable to reproduce the OOM. However, I did notice that the process memory would increase by ~200 MB for each changeset being backed out. It also seemed to GC every 2-3 changesets, as memory would increase in smaller steps then decrease in a giant step. Peak memory seemed to be maybe 800 MB. Is your system memory constrained? Do you not have swap configured? Even if Python/Mercurial consumed several hundred megabytes of memory, the OS should be able to page things for you and not fail the allocation. (This error is the rough equivalent of a malloc() failing.)

No, the system has 16 GB RAM + 16 GB page file and during |hg oops --traceback -r c4827087a333::3160ce8f8759| it stopped at 13.0 GB physical memory usage. Because it seems to fail every time during the same step, this shouldn't be a system issue. Is something hitting the 32 bit memory limit on Windows 8.1? Bug 1258837 comment 3 mentions the peak at 1.8 GB.

I wonder why my memory usage isn't close to that. This kinda/sorta looks like a gc not running problem. But I'm having a hard time coming up with reasons for several GB worth of allocations that aren't getting GCd. What changeset is your working copy on when you run this? (`hg summary` prints it) Also, please paste output from `hg debuginstall`. I'd *really* like to reproduce the severity of this.

Sorry, I think I express myself ambiguous: The whole system used 13.0 GB, the python.exe process only 1.8 GB. All the recent tests are with 127e74cb33c3 on inbound (see comment 4). $ hg summary parent: 301713:127e74cb33c3 Bug 1279930 - Stop using gfxContext device offsets on the temporary gfxContext in the GTK code's nsWindow::OnExposeEvent. r=mstange branch: default commit: 12 unknown (clean) update: 2331 new changesets (update) $ hg debuginstall checking encoding (cp1252)... checking Python executable (c:\Mozilla\Coding\Buildtools\MozillaBuild\python\python.exe) checking Python version (2.7.11) checking Python lib (c:\Mozilla\Coding\Buildtools\MozillaBuild\python\lib)... checking installed modules (c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial)... checking templates (c:\Mozilla\Coding\Buildtools\MozillaBuild\python\Lib\site-packages\mercurial\templates)... checking commit editor... checking username... no problems detected

This is a shot in the dark. I haven't looked into the underlying problem. My guess is a cycle somewhere. Review commit: https://reviewboard.mozilla.org/r/63264/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/63264/

Comment on attachment 8769292 [details] qbackout: run a garbage collection after processing each changeset (bug 1258838); https://reviewboard.mozilla.org/r/63264/#review60106 Thank you, this fixes the issue. The peak for private bytes of the python process is now ~400MB.

Pushed by gszorc@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/89bb283aec74 qbackout: run a garbage collection after processing each changeset ; r=aryx