Closed Bug 1041751 Opened 10 years ago Closed 10 years ago

Empty APZ test data buckets accumulate, using up memory, even when APZ test logging is pref'd off

Categories

(Core :: Graphics: Layers, defect, P1)

ARM
Gonk (Firefox OS)
defect

Tracking

()

RESOLVED FIXED
mozilla34
blocking-b2g 2.0+
Tracking Status
firefox32 --- wontfix
firefox33 --- wontfix
firefox34 --- fixed
b2g-v2.0 --- fixed
b2g-v2.1 --- fixed

People

(Reporter: tkundu, Assigned: botond)

References

Details

(Keywords: memory-footprint, perf, Whiteboard: [c=memory p= s=2014.08.01.t u=2.0] [caf priority: p1][MemShrink:P1][CR 691739])

Attachments

(3 files, 2 obsolete files)

Compare memory growth logs in timestamp 

1) 2014-07-21 07:59:08 ==> about-memory-34 with 
2) 2014-07-20 19:11:20 ==> about-memory-0

For (1) , I can see : 
b2g USS: 68.2 MB.

DMD log shows: 

Unreported: ~1,638 blocks in stack trace record 1 of 900
 ~6,704,334 bytes (~6,704,334 requested / ~0 slop)
 16.62% of the heap (16.62% cumulative);  30.72% of unreported (30.72% cumulative)
 Allocated at
   replace_malloc /local/mnt/workspace/tkundu/kk36d18/gecko/memory/replace/dmd/DMD.cpp:1245 (0xb6f24898 libdmd.so+0x3898)
   moz_xmalloc /local/mnt/workspace/tkundu/kk36d18/gecko/memory/mozalloc/mozalloc.cpp:53 (0xb652bdea libxul.so+0x1406dea)
   pair /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_pair.h:68 (0xb4bfc3fe libxul.so+0x4ea3fe)
   std::priv::_Rb_tree<unsigned int, std::less<unsigned int>, std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket>, std::priv::_Select1st<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> >, std::priv::_MapTraitsT<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> >, std::allocator<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> > >::_M_insert(std::priv::_Rb_tree_node_base*, std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> const&, std::priv::_Rb_tree_node_base*, std::priv::_Rb_tree_node_base*) /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_tree.c:374 (0xb4bfc464 libxul.so+0x4ea464)
   ~_Rb_tree /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_tree.h:414 (0xb4d8f19c libxul.so+0x67d19c)
   ~nsRefPtr /local/mnt/workspace/tkundu/kk36d18/out/target/product/msm8610/obj/objdir-gecko/dist/include/nsAutoPtr.h:869 (0xb4d9dcfc libxul.so+0x68bcfc)
   mozilla::layers::ClientLayerManager::BeginTransaction() /local/mnt/workspace/tkundu/kk36d18/gecko/gfx/layers/client/ClientLayerManager.cpp:178 (0xb4d9ce6a libxul.so+0x68ae6a)
   PresShell::Paint(nsView*, nsRegion const&, unsigned int) /local/mnt/workspace/tkundu/kk36d18/gecko/layout/base/nsPresShell.cpp:6120 (0xb540ec70 libxul.so+0xcfcc70)
   ....... (see in attached about-memory-34 DMD log file) 


We are seeing 61 FDs are opened with '/data/local/webapps/marketplace.firefox.com/application.zip' lsof logs for this timestamp and please also note total opened FDs at this timestamp is 392 for b2g process. 


and for (2), I can see :
b2g USS 50.0MB

DMD log shows: 

Unreported: ~47 blocks in stack trace record 3 of 696
 ~192,371 bytes (~192,371 requested / ~0 slop)
 0.79% of the heap (10.13% cumulative);  2.28% of unreported (29.31% cumulative)
 Allocated at
   replace_malloc /local/mnt/workspace/tkundu/kk36d18/gecko/memory/replace/dmd/DMD.cpp:1245 (0xb6f24898 libdmd.so+0x3898)
   moz_xmalloc /local/mnt/workspace/tkundu/kk36d18/gecko/memory/mozalloc/mozalloc.cpp:53 (0xb652bdea libxul.so+0x1406dea)
   pair /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_pair.h:68 (0xb4bfc3fe libxul.so+0x4ea3fe)
   std::priv::_Rb_tree<unsigned int, std::less<unsigned int>, std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket>, std::priv::_Select1st<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> >, std::priv::_MapTraitsT<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> >, std::allocator<std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> > >::_M_insert(std::priv::_Rb_tree_node_base*, std::pair<unsigned int const, mozilla::layers::APZTestData::Bucket> const&, std::priv::_Rb_tree_node_base*, std::priv::_Rb_tree_node_base*) /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_tree.c:374 (0xb4bfc464 libxul.so+0x4ea464)
   ~_Rb_tree /local/mnt/workspace/tkundu/kk36d18/gecko/build/stlport/stlport/stl/_tree.h:414 (0xb4d8f19c libxul.so+0x67d19c)
   ~nsRefPtr /local/mnt/workspace/tkundu/kk36d18/out/target/product/msm8610/obj/objdir-gecko/dist/include/nsAutoPtr.h:869 (0xb4d9dcfc libxul.so+0x68bcfc)
   mozilla::layers::ClientLayerManager::BeginTransaction() /local/mnt/workspace/tkundu/kk36d18/gecko/gfx/layers/client/ClientLayerManager.cpp:178 (0xb4d9ce6a libxul.so+0x68ae6a)
   PresShell::Paint(nsView*, nsRegion const&, unsigned int) /local/mnt/workspace/tkundu/kk36d18/gecko/layout/base/nsPresShell.cpp:6120 (0xb540ec70 libxul.so+0xcfcc70)
   .............. (see in attached about-memory-0 DMD log file) 

We are seeing ONLY ONE FD is opened with '/data/local/webapps/marketplace.firefox.com/application.zip' lsof logs for this timestamp and total FD is 314 in this timestamp.



For both (1) and (2) : 
b2g-procrank shows only b2g, homescreen, NUWA, preallocated and keyboard is running at this time and all memory reports are collected with --minimize option [1]. Important thing is that b2g is not releasing USS memory even with '--minimize' option.



b2g is hitting 80MB at some point and getting killed finally on 256MB device. 

Can we get a fix for above 'moz_xmalloc' allocations ? 
Are we leaking FD '/data/local/webapps/marketplace.firefox.com/application.zip' over time period ? 


[1] while true; do   adb shell date >> memory_logs_timestamp.txt && MOZ_IGNORE_NUWA_PROCESS=1 ./device/qcom/b2g_common/mozilla-b2g/tools/get_about_memory.py --minimize --gecko-objdir=./out/target/product/msm8610/obj/objdir-gecko/ --gonk-objdir=./out --product=msm8610 && sleep 1200 ; done


FULL logs are at : https://drive.google.com/file/d/0B1cSMS8_GuAEMFFPdmVZSGZzSjA/edit?usp=sharing
STR: 
1) Run camera, camcorder, SMS, call, airplane_on_off , bt_on_off , wifi_on_off for atleast 18 hours. 

I know STR will be very difficult to reproduce. But we can start with two debugging two issues (also mentioned in Comment 0) :

1) Can we get a fix for above 'moz_xmalloc' allocations ? 
2) Are we leaking FD '/data/local/webapps/marketplace.firefox.com/application.zip' over time period ?
blocking-b2g: --- → 2.0?
Whiteboard: [Memshrink] → [Memshrink][CR 691739]
Mike - Can someone from your team investigate this issue?
Component: General → Performance
Flags: needinfo?(mlee)
Keywords: footprint, perf
Whiteboard: [Memshrink][CR 691739] → [MemShrink][CR 691739]
24.24 MB (100.0%) -- explicit
├──12.88 MB (53.12%) ── heap-unclassified
├───6.52 MB (26.90%) ++ js-non-window
├───2.63 MB (10.83%) ++ images/content/raster
├───0.03 MB (00.12%) ++ window-objects
├───1.04 MB (04.30%) ++ heap-overhead
├───0.48 MB (01.97%) ++ xpconnect
├───0.35 MB (01.45%) ++ (11 tiny)
└───0.32 MB (01.31%) ++ dmd

Most of this in APZC code.  Why are we storing test data in a production build anyways?
Component: Performance → Graphics: Layers
Flags: needinfo?(mlee) → needinfo?(milan)
Product: Firefox OS → Core
Whiteboard: [MemShrink][CR 691739] → [caf priority: p1][MemShrink][CR 691739]
Isn't that related to what I see in bug 1027649
Maybe.  There's nothing in that bug that would help identify the cause.
Whiteboard: [caf priority: p1][MemShrink][CR 691739] → [c=memory p= s= u=] [caf priority: p1][MemShrink][CR 691739]
Botond, can you take a look at this bug?  I think the only way for APZTestData to be leaked is for the whole ClientLayerManager to be leaked, but I could be wrong.  I'm not sure if we do want APZTestData in the release build (if we do, the name for the class is less than optimal :), and if we don't, we should take care of that.  Either way, this could be pointing to a ClientLayerManager leak, and maybe we didn't see it before because it was much smaller?
As a quick remedy, can we disable the APZTestData in the release build to see if we get the memory leak under control?
Assignee: nobody → botond
Blocks: 961289
Flags: needinfo?(nical.bugzilla)
Flags: needinfo?(milan)
Flags: needinfo?(botond)
blocking-b2g: 2.0? → 2.0+
Botond, if this is a ClientLayerManager leak, can you assign to :nical instead?
Sotaro was mentioning that we should be in good shape (e.g., not leaking) ClientLayerManagers and such, so let's share all the information we have.
(In reply to Milan Sreckovic [:milan] from comment #8)
> Sotaro was mentioning that we should be in good shape (e.g., not leaking)
> ClientLayerManagers and such, so let's share all the information we have.

I confirmed about ClientLayerManagers destruction during Bug 1004191. On the STR in Bug 1004191 there was no ClientLayerManagers leak.
Flags: needinfo?(nical.bugzilla)
Botond thinks this is just the accumulation of the empty APZTestData containers (the actual data is behind the pref), so we'll put the container creation behind the same pref.
(In reply to Milan Sreckovic [:milan] from comment #10)
> Botond thinks this is just the accumulation of the empty APZTestData
> containers (the actual data is behind the pref), so we'll put the container
> creation behind the same pref.

To expand on that a bit: the APZ test logging code is compiled into release builds, but it's pref'd off. However, there's a deficiency in how be pref'd it off. The APZ test data consists of a bucket for each paint, and there are two places in the code that touch it: 1) at the beginning of each paint, we create a new empty bucket for that paint; 2) at various points in the paint, we write data into the current bucket. Only (2) is pref'd off, (1) is not. This means that over a long time of using the phone, the empty buckets for each paint accumulate.

It's straightforward to pref the creation of empty buckets off as well. Patch forthcoming.
Flags: needinfo?(botond)
Attachment #8460415 - Flags: review?(bugmail.mozilla)
Attachment #8460415 - Flags: review?(bugmail.mozilla) → review+
Summary: b2g process leaks USS memory → Empty APZ test data buckets accumulate, using up memory, even when APZ test logging is pref'd off
This seems simple enough to land without a Try push. Will land when the tree opens.
(In reply to Botond Ballo [:botond] from comment #13)
> This seems simple enough to land without a Try push. Will land when the tree
> opens.

I am trying it . I will update soon :)
Whiteboard: [c=memory p= s= u=] [caf priority: p1][MemShrink][CR 691739] → [c=memory p= s= u=] [caf priority: p1][MemShrink:P1][CR 691739]
Backed out in https://hg.mozilla.org/integration/mozilla-inbound/rev/b920a17bab70 for causing at least the Gip test failures in https://tbpl.mozilla.org/php/getParsedLog.php?id=44381024&tree=Mozilla-Inbound

There were also OSX tp failures starting around the same time, though it wasn't run on the push where this patch landed, so I'm not completely sure if they were caused by this bug: https://tbpl.mozilla.org/php/getParsedLog.php?id=44382853&tree=Mozilla-Inbound

Windows debug crashtests also started failing around the same time, but again, they didn't run on this push, so I'm not sure this patch caused them: https://tbpl.mozilla.org/php/getParsedLog.php?id=44382969&tree=Mozilla-Inbound



I'll retrigger jobs to bisect where they started failing, and will follow up with the results.
So, the windows crashtests do seem to have started with this as well: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=debug%20test%20crashtest&rev=c1ea5a8280d7


And the tp failures also started up on this landing.
Flags: needinfo?(botond)
For the record, the crashtest failures aren't constant, as they sometimes pass just fine.
keep in mind this has a lot of performance impact when we look to reland it, here is what we have:
+--------------------+--------------------------------------+---------+
| platform           | test                                 | percent |
+--------------------+--------------------------------------+---------+
| MacOSX 10.8        | Session Restore no Auto Restore Test | -11.8%  |
| MacOSX 10.8        | Ts, Paint                            | -35%    |
| MacOSX 10.8        | Paint                                | -60.4%  |
| MacOSX 10.8        | a11y Row Major MozAfterPaint         | -47.4%  |
| MacOSX 10.6 (rev4) | Paint                                | -29.3%  |
| MacOSX 10.6 (rev4) | a11y Row Major MozAfterPaint         | -22%    |
| MacOSX 10.6 (rev4) | Session Restore no Auto Restore Test | +9.88%  |
| MacOSX 10.6 (rev4) | Ts, Paint                            | -26.8%  |
| MacOSX 10.6 (rev4) | Session Restore Test                 | +4.38%  |
| MacOSX 10.6 (rev4) | TP5 Scroll                           | +3.72%  |
| MacOSX 10.8        | TP5 Scroll                           | +14.1%  |
| WINNT 6.1 (ix)     | Ts, Paint                            | -3.92%  |
+--------------------+--------------------------------------+---------+

so ts_paint, paint, a11y all saw regressions, but tp5o scroll saw an improvement.  Please look into the talos number prior to relanding to minimize the regressions!
wait there are more improvements than that list, in fact, we seem to have a lot of improvements in:
tart, cart, tscroll-asap, tresize!

And larger ones!

Here is the alerts from the backout (negative values are actually improvements from this patch, positive values here are regressions from this patch:
+--------------------+--------------------------------------+---------+
| platform           | test                                 | percent |
+--------------------+--------------------------------------+---------+
| MacOSX 10.6 (rev4) | Session Restore no Auto Restore Test | -10.7%  |
| MacOSX 10.6 (rev4) | Session Restore Test                 | -4.17%  |
| MacOSX 10.6 (rev4) | a11y Row Major MozAfterPaint         | +17.7%  |
| MacOSX 10.6 (rev4) | Ts, Paint                            | +21.3%  |
| MacOSX 10.6 (rev4) | Paint                                | +22.5%  |
| MacOSX 10.8        | TP5 Scroll                           | -16.3%  |
| MacOSX 10.6 (rev4) | TP5 Scroll                           | -3.57%  |
| MacOSX 10.6 (rev4) | Customization Animation Tests        | -50.3%  |
| MacOSX 10.6 (rev4) | tscroll-ASAP                         | -9.51%  |
| MacOSX 10.6 (rev4) | Tab Animation Test                   | -106%   |
| MacOSX 10.8        | a11y Row Major MozAfterPaint         | +30.6%  |
| MacOSX 10.6 (rev4) | TResize                              | +81.4%  |
| MacOSX 10.8        | Session Restore no Auto Restore Test | +9.08%  |
| MacOSX 10.8        | Paint                                | +36.7%  |
| MacOSX 10.8        | Customization Animation Tests        | -35%    |
| MacOSX 10.8        | tscroll-ASAP                         | -20.5%  |
| MacOSX 10.8        | Tab Animation Test                   | -119%   |
| MacOSX 10.8        | TResize                              | +84.7%  |
+--------------------+--------------------------------------+---------+
Comment on attachment 8460415 [details] [diff] [review]
Only create an APZ test data bucket for a paint if APZ test logging is enabled

Review of attachment 8460415 [details] [diff] [review]:
-----------------------------------------------------------------

Don't know the code, but this looks suspicious...

::: gfx/layers/client/ClientLayerManager.cpp
@@ +170,3 @@
>      ++mPaintSequenceNumber;
>      mApzTestData.StartNewPaint(mPaintSequenceNumber);
>    }

Did you want this instead?

if (!mIsRepeatTransaction) {
  ++mPaintSequenceNumber;
  if (gfxPrefs::APZTestLogginEnabled()) {
    mApzTestData.StartNewPaint(mPaintSequenceNumber);
  }
}
(In reply to Milan Sreckovic [:milan] from comment #21)
> Don't know the code, but this looks suspicious...
> 
> ::: gfx/layers/client/ClientLayerManager.cpp
> @@ +170,3 @@
> >      ++mPaintSequenceNumber;
> >      mApzTestData.StartNewPaint(mPaintSequenceNumber);
> >    }
> 
> Did you want this instead?
> 
> if (!mIsRepeatTransaction) {
>   ++mPaintSequenceNumber;
>   if (gfxPrefs::APZTestLogginEnabled()) {
>     mApzTestData.StartNewPaint(mPaintSequenceNumber);
>   }
> }

I don't believe so. The paint sequence number is only used by the APZ testing framework, and any code that relies on it is also pref'd off with the same pref.

I'll run the failing tests on Try with this change just to be absolutely sure, as I don't really have other ideas.
As I said, I don't know the code.  I just saw it being passed via IPC, but didn't track down what happens on the other side...
I'm also assuming there are no assertions in the debug, otherwise I'd check if gfxPrefs is getting initialized sooner than before.
(In reply to Milan Sreckovic [:milan] from comment #24)
> I'm also assuming there are no assertions in the debug, otherwise I'd check
> if gfxPrefs is getting initialized sooner than before.

Unfortunately, the Gip tests seem to only be run on Opt builds. I'll look into the possibility of running them locally on a debug build.
Comment 19 gives us some alert. We are waiting to see final patch for this memleak issue .
We're working on the final patch, but the only issue that would resolve is test specific - you should be able to test with the patch as is when doing normal running.
have you guys looked at the performance regressions introduced by this?
Haven't looked at performance regressions.  Don't believe there should be any once the code is correct; if after that we're still seeing some, we'll investigate.
Implementing Milan's suggestion from comment 21 appears to fix the failing tests, and not introduce performance regressions, as can be seen in this full (including Talos) Try push: https://tbpl.mozilla.org/?tree=Try&rev=1b60c98820db

While I don't understand why this change makes a difference, since this is a high-priority issue and we have a green Try run, I'm going to land this fix, and continue to investigate independently to understand the cause.
Flags: needinfo?(botond)
Attached patch Proper fix for test failures (obsolete) (deleted) — Splinter Review
(In reply to Botond Ballo [:botond] from comment #30)
> Implementing Milan's suggestion from comment 21 appears to fix the failing
> tests, and not introduce performance regressions, as can be seen in this
> full (including Talos) Try push:
> https://tbpl.mozilla.org/?tree=Try&rev=1b60c98820db
> 
> While I don't understand why this change makes a difference, since this is a
> high-priority issue and we have a green Try run, I'm going to land this fix,
> and continue to investigate independently to understand the cause.

Turns out I had done something very silly in my original bug 961289 patch: I accidentally switched the order of two arguments, 'bool aScheduleComposite' and 'uint32_t aPaintSequenceNumber', to one of the functions on the path from the client layer manager to the compositor.

As a result, we would schedule a composite whenever the sequence number was greater than zero, rather than whenever aScheduleComposite was true. This mostly worked out because the sequence number is greater than zero on every paint except the first, and aScheduleComposite is always true except in some scenarios involving plugins.

Now that I conditioned the incrementing of the sequence number on the pref being on, though, we stopped scheduling composites, and experienced various test failures as a result.

This patch puts the incrementing of the sequence number back under the pref, where it should be, and implements the proper fix for the test failures - fixing the order-of-arguments bug - instead.
Attachment #8462099 - Flags: review?(bugmail.mozilla)
Attachment #8460415 - Attachment description: bug1041751.patch → Only create an APZ test data bucket for a paint if APZ test logging is enabled
Attached patch Proper fix for test failures (deleted) — Splinter Review
Sorry, wrong patch.
Attachment #8462099 - Attachment is obsolete: true
Attachment #8462099 - Flags: review?(bugmail.mozilla)
Attachment #8462101 - Flags: review?(bugmail.mozilla)
(In reply to Botond Ballo [:botond] from comment #33)
> Created attachment 8462101 [details] [diff] [review]
> Proper fix for test failures
> 
> Sorry, wrong patch.

We want to ask our stability team to test this patch for memleak issue. It seems like we need to apply two patches in sequence for this
1)  Comment 12 and 
2)  Comment 33 .

Could you please confirm ?
Flags: needinfo?(botond)
patch from Comment 33 does not apply cleanly on FFOS v2.0. Could you please rebase for FFOS 2.0 and confirm us again ?
Comment on attachment 8462101 [details] [diff] [review]
Proper fix for test failures

Review of attachment 8462101 [details] [diff] [review]:
-----------------------------------------------------------------

Make sure to run it through try before landing. and s/tn/kats/ in the commit message.
Attachment #8462101 - Flags: review?(bugmail.mozilla) → review+
For convenience, posting a combined patch of Part 1 and Part 2.

This one applies to trunk. I will post a 2.0 one as well.
Flags: needinfo?(botond)
Attached patch Combined patch, applies to 2.0 (deleted) — Splinter Review
Here's the combined patch for 2.0.
Attachment #8460415 - Attachment is obsolete: true
Marking leave-open until the second patch lands on m-c as well.
Keywords: leave-open
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #36)
> Make sure to run it through try before landing. 

Try push: https://tbpl.mozilla.org/?tree=Try&rev=a540ac8ce9e4

> and s/tn/kats/ in the commit message.

Done.
(In reply to Botond Ballo [:botond] from comment #40)
> (In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #36)
> > and s/tn/kats/ in the commit message.
> 
> Done.

Looks like this didn't get done, oops :)
(In reply to Timothy Nikkel (:tn) from comment #42)
> (In reply to Botond Ballo [:botond] from comment #40)
> > (In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #36)
> > > and s/tn/kats/ in the commit message.
> > 
> > Done.
> 
> Looks like this didn't get done, oops :)

Oh, whoops! I amended the patch, but then pushed the old patch to inbound... sorry about that.
Closing now that both patches have landed.
Status: NEW → RESOLVED
Closed: 10 years ago
Keywords: leave-open
Resolution: --- → FIXED
Target Milestone: --- → 2.1 S1 (1aug)
Whiteboard: [c=memory p= s= u=] [caf priority: p1][MemShrink:P1][CR 691739] → [c=memory p= s=2014.08.01.t u=2.0] [caf priority: p1][MemShrink:P1][CR 691739]
No STR is present to create test case to address bug.
QA Whiteboard: [QAnalyst-Triage?]
Flags: needinfo?(ktucker)
QA Whiteboard: [QAnalyst-Triage?] → [QAnalyst-Triage+]
Flags: needinfo?(ktucker) → in-moztrap-
(In reply to Yeojin Chung [:YeojinC] from comment #47)
> No STR is present to create test case to address bug.

Right - it's a fairly simple STR, but it takes a long time.  Basically, just scroll the phone for 8 hours and watch the memory usage grow.  The bug was "losing" a small amount of memory, but it was doing it on refresh, so it added up.
We are unable to verify this issue because it is a backend problem.
QA Whiteboard: [QAnalyst-Triage+] → [QAnalyst-verify-][QAnalyst-Triage?]
Flags: needinfo?(ktucker)
QA Whiteboard: [QAnalyst-verify-][QAnalyst-Triage?] → [QAnalyst-verify-][QAnalyst-Triage+]
Flags: needinfo?(ktucker)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: