1. Title : When deleting 100 conversations in SMS app, SMS app takes 6.1sec to delete 2. Precondition :Restore 500 conversations 3. Tester's Action : Run SMS App >> Select edit mode icon >> Select 100 conversations >> Select delete icon >> Delete 4. Detailed Symptom : SMS app takes 6.1 seconds,however it should be under 1.5 seconds according to leo device spec 5. Expected : SMS app takes under 1.5 seconds 6. Reproducibility: Y 1) Frequency Rate : 100%

Gene, Is there any pending work here? Thanks!

We had ~1.3 secs on Unagi before. Maybe related to bug 880563, which introduced additional works during the process.

Hi Chuck, directly assign this to you if you don't mind. You must be familiar with this kind of issue. Thanks!

(In reply to Gene Lian [:gene] from comment #3) > Hi Chuck, directly assign this to you if you don't mind. You must be > familiar with this kind of issue. Thanks! I don't think so. We only have extra work when the send/retrieve transactions are still alive. I believe we won't often run into this case.

I've run marionette test case [1] to get the speed on emulator. It takes 957 ms to delete 100 SMS with patches in bug 880563, and 912 ms after removing |notifyObservers()| in |deleteMessage()| [2]. The speed are quite the same(just 5% longer), and as I can recall, the measured speed in bug 771458 is in the same range. [1] https://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/tests/marionette/test_massive_incoming_delete.js [2] https://mxr.mozilla.org/mozilla-central/source/dom/mobilemessage/src/gonk/MobileMessageDatabaseService.js#1541

Thanks Chuck for the verifications! I found something interesting: comment #0 says it is deleting 100 *conversations* not *messages*. Did we misunderstand that bit? Supposing each of the 100 conversions has at least one message and the Gaia end doesn't delete messages by using an array (when deleting conversions), then is it possible that's why it's so slow?

Hi Leo, Hoping to have double check: you're saying you're deleting 100 *conversations* instead of 100 *messages*. Right? If you're indeed deleting 100 *conversations*, how many messages does each conversation have at least?

(In reply to Gene Lian [:gene] from comment #6) > Supposing each of the 100 conversions has at least one message and the Gaia > end doesn't delete messages by using an array (when deleting conversions), > then is it possible that's why it's so slow? I think you're right. "Deleting conversations" deletes SMS one by one as a callback of |getMessages()| [1]. The speed of this bug mainly depends on Gaia's design. [1] https://mxr.mozilla.org/gaia/source/apps/sms/js/thread_list_ui.js#254

Hi Gene, Chuck, If you want I could take a look to this problem as well, Wdyt?

(In reply to Borja Salguero [:borjasalguero] from comment #9) > Hi Gene, Chuck, If you want I could take a look to this problem as well, > Wdyt? Hi Borja, I am not so familiar with Gaia part, so you can take it if you like to.

Put SMS to delete into an array and delete them once, instead of delete them one by one. Under test of deleting 100 conversations(threads), each conversation has 1 message. The speed before patch is about 7 seconds, and speed after patch is about 4 seconds. Comparing deleting 100 messages in one thread, and delete 100 threads with 1 message in each, deleting threads need to perform |GetMessages()| in Gaia to get message list in the thread to delete. So the later condition took more time than prior one. I think it's nearly impossible to delete 100 *conversations* within 1.5 seconds, because it already took ~1.3 seconds to delete 100 *messages*. Based on the test result in emulator, the |GetMessages()| part take more time then delete operation. Assume the speed improvement on device is as same as in emulator, I think deleting 100 conversations(1 messages each) in around 3.5 seconds might be our fastest speed.

Here mainly the bug is related that, for deleting a list of 100 conversations, with 20 messages each, we are requesting the IDs for every conversation (in this case 100x20 = 2000 message IDs), and we are deleting one by one. Currently as we have 'deleteMessages(Array)' we could reduce this to one call, so I will try to make a patch today and we could review it together. However it could be a great idea to 'delete' a thread directly, without requesting Gecko for getting the list of IDs, Wdyt about adding 'deleteThreads([Thread-IDs])? We could improve the performance a lot.

Gene, I would like to know your opinion about adding a 'deleteThreads' method given an array of threads, Wdyt?

I've stumbled upon this issue the other day while testing the x-heavy reference workload which is 2000 messages. I've tried deleting all the threads in one go and it took north of 10 minutes. I did not bother checking the actual duration as it was already too long for me to consider it acceptable.

Here more info. I've tested with: make reference-workload-light (around 400 SMS/MMS DB) I've added some logs when deleting the entire list of SMS/MMS: E/GeckoConsole( 8226): Content JS LOG at app://sms.gaiamobile.org/js/thread_list_ui.js:227 in checkDone: Time spent retrieving the list of messageIDs 10782 E/GeckoConsole( 8226): Content JS LOG at app://sms.gaiamobile.org/js/thread_list_ui.js:231 in deletionDone: Total time for deleting 400 messages is 15182 So we need around 10782ms for retrieving the whole list, and less than 5 seconds for removing the list of 400 messages calling directly to 'deleteMessage' given an array. I think that the best solution here should be adding a low-level deletion related with threads, Wdyt? Thanks!

This is the change in Gaia with some logs

I've just added the change in Gaia, but even with this modification we spend a lot of time trying to figure out the list of message-Ids that we have to remove. Waiting feedback from Gecko Team!

I've opened bug https://bugzilla.mozilla.org/show_bug.cgi?id=890921 for making the Gecko part, due to we need some work there for having this working as expected.

(In reply to Borja Salguero [:borjasalguero] from comment #15) > Here more info. I've tested with: > make reference-workload-light (around 400 SMS/MMS DB) > > I've added some logs when deleting the entire list of SMS/MMS: > E/GeckoConsole( 8226): Content JS LOG at > app://sms.gaiamobile.org/js/thread_list_ui.js:227 in checkDone: Time spent > retrieving the list of messageIDs 10782 Out of curiosity, have you tried measuring both the actual API call and the DOM querySelector request ?

(In reply to Borja Salguero [:borjasalguero] from comment #15) > Here more info. I've tested with: > make reference-workload-light (around 400 SMS/MMS DB) > > I've added some logs when deleting the entire list of SMS/MMS: > E/GeckoConsole( 8226): Content JS LOG at > app://sms.gaiamobile.org/js/thread_list_ui.js:227 in checkDone: Time spent > retrieving the list of messageIDs 10782 > E/GeckoConsole( 8226): Content JS LOG at > app://sms.gaiamobile.org/js/thread_list_ui.js:231 in deletionDone: Total > time for deleting 400 messages is 15182 > > So we need around 10782ms for retrieving the whole list, and less than 5 > seconds for removing the list of 400 messages calling directly to > 'deleteMessage' given an array. I think that the best solution here should > be adding a low-level deletion related with threads, Wdyt? Thanks! Borja, could you share your patch here ? Maybe something was overlooked ?

I am fine with adding API for deleting threads. But at the same time, I think we have to draw a line for what so called "acceptable" speed, or there will always be bugs complaining about speed if they test it with heavier load. For example, deleting 100 messages in single conversation takes 1.3 second is acceptable, 5.2 seconds for 400 messages might still be acceptable. But is it still acceptable of taking 26 seconds to delete 2000 messages?

Just like the Vicamo's comment bug 890921, comment #5. I don't really think we'd have significant improvement after providing .deleteThreads(), because the messages and the threads are actually maintained by two database stores separately. Even if we can provide .deleteThreads(), we sill need to delete the messages associated with those threads. This behaviour is similar to what we've done for .deleteMessage(), though.

(In reply to Julien Wajsberg [:julienw] from comment #21) > Borja, could you share your patch here ? Maybe something was overlooked ? The patch is an attachment :)

(In reply to Gene Lian [:gene] from comment #23) > Just like the Vicamo's comment bug 890921, comment #5. I don't really think > we'd have significant improvement after providing .deleteThreads(), because > the messages and the threads are actually maintained by two database stores > separately. It seems to me that it's not clear what's making message deletion slow so I would wholeheartedly suggest profiling this behavior before coming to any decision about what changes need to be made. Taking stabs in the dark won't do us much good. I'm willing to run a profile of the scenarios we've been discussing myself and I'll post it shortly.

(In reply to Gene Lian [:gene] from comment #23) > Just like the Vicamo's comment bug 890921, comment #5. I don't really think > we'd have significant improvement after providing .deleteThreads(), because > the messages and the threads are actually maintained by two database stores > separately. Even if we can provide .deleteThreads(), we sill need to delete > the messages associated with those threads. This behaviour is similar to > what we've done for .deleteMessage(), though. This could avoid the IPC cost still, and I suspect it can be high when we have a lot of data like here. However I agree we should profile with and without Borja's patch.

(In reply to Gene Lian [:gene] from comment #7) > Hi Leo, > > Hoping to have double check: you're saying you're deleting 100 > *conversations* instead of 100 *messages*. Right? > > If you're indeed deleting 100 *conversations*, how many messages does each > conversation have at least? Hi, Here we are deleting 100 conversations and each conversation is having only one message. Thanks

(In reply to Julien Wajsberg [:julienw] from comment #26) > This could avoid the IPC cost still, and I suspect it can be high when we > have a lot of data like here. But the current .deleteMessage(array) is already working in this way. That is, we only use one IPC request to delete the messages. In the back-end, after the messages are deleted and then their corresponding threads will also get updated. Later, using .getThreads() will then retrieve the updated threads (this goes by one IPC as well).

(In reply to Gene Lian [:gene] from comment #28) > (In reply to Julien Wajsberg [:julienw] from comment #26) > > This could avoid the IPC cost still, and I suspect it can be high when we > > have a lot of data like here. > > But the current .deleteMessage(array) is already working in this way. That > is, we only use one IPC request to delete the messages. In the back-end, > after the messages are deleted and then their corresponding threads will > also get updated. Later, using .getThreads() will then retrieve the updated > threads (this goes by one IPC as well). Gene, when we delete conversations, we first call getMessages with a filter to get the message ids for these conversations. When getting the getMessages result there are a lot of back and forth IPC using the cursor. That's what I was refering to in my comment.

BTW Gabriele is seeing this in bug 890921 comment 10 as well (but that's not the biggest issue). Borja, in Bug 890921 comment 10, Gabriele is saying we can try to remove all the threads in one go, instead of doing it once per thread. Also, I'd like to try this with your current patch calling deleteMessages only once. Your patch is not correct btw, because checkDone is called once per thread, and we would like to call deleteMessages once for all threads. In your patch, you call deleteMessages once per thread, without even emptying the messageIds array, so I think this is has quite a penalty. Could you propose another patch today, or maybe someone else can try this ?

(In reply to Julien Wajsberg [:julienw] from comment #29) > Gene, when we delete conversations, we first call getMessages with a filter > to get the message ids for these conversations. When getting the getMessages > result there are a lot of back and forth IPC using the cursor. That's what I > was refering to in my comment. Thanks for the clarification. Now I understand better. :) If so, indeed, the numbers of IPCs will dominate the performance here.

(In reply to Julien Wajsberg [:julienw] from comment #30) > Could you propose another patch today, or maybe someone else can try this ? For sure. This patch is not really a patch, is only a way for getting logs about timing with a small modification. I'll create a real patch with all the suggestions described in this thread and I'll ping you when I'll be done. Thanks!

(In reply to Julien Wajsberg [:julienw] from comment #30) > Borja, in Bug 890921 comment 10, Gabriele is saying we can try to remove all > the threads in one go, instead of doing it once per thread. I've tried even without removing anything in the DOM. Only Gecko requests. At the end of this comment the results > Also, I'd like to try this with your current patch calling deleteMessages > only once. Your patch is not correct btw, because checkDone is called once > per thread, and we would like to call deleteMessages once for all threads. > In your patch, you call deleteMessages once per thread, without even > emptying the messageIds array, so I think this is has quite a penalty. Currently we are calling 'deleteMessages' **only** once (when 'count == 0'), so the patch is working as expected. Despite of 'checkDone' is called per every thread, it's only for removing the element from the DOM. But even removing everything related with DOM, the results are: E/GeckoConsole( 2401): Content JS LOG at app://sms.gaiamobile.org/js/thread_list_ui.js:228 in checkDone: Time spent retrieving the list of messageIDs 10088 E/GeckoConsole( 2401): Content JS LOG at app://sms.gaiamobile.org/js/thread_list_ui.js:233 in deletionDone: Total time for deleting 400 messages is 13671 So despite of we have saved 2 seconds, the amount of time is huge IMHO. That's why we would need at least a filter for retrieving *all* messages given an *array* of thread IDs, or having a 'deleteThreads' which delegate all this stuff to Gecko. Julien I would like your opinion here, because I think we need at least the 'improved' filter... wdyt?

Gabriele, I've updated the code, so now we have well-differentiated steps: - At the beginning we remove all DOM Elements - Then we create the array with all messageIDs - After it we call deleteMessages with the array. With this approach Im saving 5 seconds more, so when deleting 400 messages my results are: - Total time for updating DOM 1080 - Time spent retrieving the list of messageIDs 5669 - Total time for deleting 400 messages is 3177 Total time 8846 I've been testing with "reference-workload-light". Could you profile this solution? Thanks!

Here's a profile of the workload using :borjasalguero latest PR, the testing environment is the same as in bug 890921 comment 10: http://people.mozilla.com/~bgirard/cleopatra/#report=e9e3b112afe7956f37a0783bb62d055dc5a84771 Now things look *much* better, the total measured time for deleting all threads is a little over 2s. We now have a single large reflow in the beginning and no more later which is saving a lot of time. We can still see the IPC ping-pong but it's contribution to the profile is still small; probably an effect of the app responding to IPC faster now that it's not busy reflowing the display. The largest contributor left to the profile is repainting (>40% of the time) but I fear that it's caused by the spinning animation we're displaying; alas this is a known problem and we can't really do anything about it in the app (see bug 827277). I will try to profile larger reference workloads to evaluate the impact but I expect it to be very good (the time spent reflowing did grow proportionally with the number of threads and made large workloads *extremely* slow). So I would recommend following this approach and closing bug 890921 as WONTFIX or INVALID since we don't need that particular functionality.

Julien, could you take a look to the proposal? Thanks!

To add another data-point, deleting all threads from the x-heavy workload using the latest PR now takes a bit over one minute, down from the 10+ minutes of the unmodified app. This impressive improvement (we just squashed what was taking ~90% of the time in this scenario) makes me want to stress a point I've already made but which is still critically important: before optimizing anything it is paramount to understand what needs to be made faster so profile, measure and profile again. Change the code only when you're sure you found the actual problem.

I noticed that, the deleting process are assuming every operation will success. If there's an |cursor.onerror| in the |getMessages()|, the DOMs are deleted, but messages in DB are not. Maybe we need a way to recovery these DOMs should this happens?

(In reply to Chuck Lee [:chucklee] from comment #38) > I noticed that, the deleting process are assuming every operation will > success. > If there's an |cursor.onerror| in the |getMessages()|, the DOMs are deleted, > but messages in DB are not. > Maybe we need a way to recovery these DOMs should this happens? I think the best way would be to invert the order in which we're doing operations, i.e. gather all messages from the threads we want to delete (using |cursor.onerror| to exclude the threads for which we hit errors), delete the messages and then remove the threads we're sure we have deleted from the DOM in one go.

Ready to review! Julien r?

Hi Borja, Tested with the existing patch 1.For deleting 100 list of conversations from existing 500 conversations, it is taking approximately 6.9sec 2.For deleting 100 list of conversations from existing 100 messages, it is taking approximately 2.3sec 3.For deleting 500 messages from existing 500 list, it is taking approximately 13.8sec Conclusion: There is an improvement in the 2nd and 3rd cases but the actual issue is to delete 100 messages from existing 500 list of messages,in this case it is taking 6.9sec.It should be 1.5sec. Please check once. Thanks

Hi Leo, Could you specify which Gecko BUILD you are using for this test? Because Gabriele's test (you can check https://bugzilla.mozilla.org/show_bug.cgi?id=890174#c35) are better than this. We are using latest B2G18 BUILD with latest Gaia master + this patch. Please confirm this in order to ensure that we are testing the same. On the other hand I would like to know how you are creating the DB of threads & messages, are you using our 'workload' profiles defined in our makefile? If it's not the case, could you provide exactly the script for loading the MessageDB for testing the same? Thanks

Furthermore, I would like to ensure that this patch is in your BUILD: https://bugzilla.mozilla.org/show_bug.cgi?id=880563 Could you confirm this?

Hi Borja, Please find the updates on this issue 1.The Gecko build which we are using is 499c1f8ea7ad0cdaa7086214278e2944b8a2ea33. 2.We are using AU149 version. 3.We are not using 'workload' profiles defined in makefile. 4.We have a indexDB which contains 500 messages.Everytime we are pusing to load all the messages. Thanks,

(In reply to Leo from comment #44) Hi Leo, We need to clarify some issues here because you are testing a quite old version of the code, that's why our results differs. Answers inline > 2.We are using AU149 version. This version is quite old. As I told you previously, for testing this properly you need (at least) to have in your BUILD the patch https://hg.mozilla.org/releases/mozilla-b2g18/rev/e3456aec01cf, which was landed 4th of July. Without this patch, we are not testing the same. Please update your BUILD with this Gecko patch. > 3.We are not using 'workload' profiles defined in makefile. > 4.We have a indexDB which contains 500 messages.Everytime we are pusing to > load all the messages. Im really concerned about the DB you are using for testing this issue, and the SRT for testing. The title of this bug is "When deleting 100 conversations...", and here you are talking about 'messages'. Please, Could you specify the DB structure you are using (and adding it as an attachment it would be a good idea IMHO)? In our tests, done by Gabriele, deleting hundreds of Threads (using our workloads available in the makefile), the performance test shows a huge improvement in all scenarios when deleting a set of Threads/Conversations. We need all this info in order to move forward with this. Thanks a lot.

Comment on attachment 772055 [details] Pull request reviewed on github mostly nits, hopefully will be finished today :)

Hi Borja, We have checked the Gecko patch which you have mentioned https://hg.mozilla.org/releases/mozilla-b2g18/rev/e3456aec01cf and this is available in our build We are deleting 100 conversations only(sorry by mistake i have updated as messages) here and the DB structure is SMSDB

Hi Borja, Please check the test attachments added.

Comment on attachment 772055 [details] Pull request removed the review flag. Borja, please ask review from someone else once the comments are fixed as I'm in PTO :)

I'd like to see all those units you deleted restored before reviewing this - but I can definitely take the review when you update the pull Borja.

I'll try to address all suggestions and I would like to get a new profiling taking into account the new DB attached by Leo for ensuring that this bug accomplish the minimum required by Leo. Thanks!

(In reply to Borja Salguero [:borjasalguero] from comment #51) > I'll try to address all suggestions and I would like to get a new profiling > taking into account the new DB attached by Leo for ensuring that this bug > accomplish the minimum required by Leo. Thanks! When everything's in place please send me a needinfo for the profile; I'm a bit swamped by requests ATM and I wouldn't want to lose track of this.

Comment on attachment 772055 [details] Pull request Steve, I've added all suggestions requested by Julien. As he is in PTO, could you take a look? Thanks!

Hi Borja, I just have a small issue in test case. I'm good with the rest of the changes. In the long term, maybe we need to allow delete api interface to accept filter as parameter, it could probably reduce the effort in accessing the db and simplify the thread deletion flow in gaia.

(reposted from github) @borjasalguero @steveck-chung As part of https://bugzilla.mozilla.org/show_bug.cgi?id=893838 I had to rewrite much of the delete operations and to be honest, I'd like to take over this issue entirely. I've written up a complete and comprehensive explanation of what needs to be addressed and why, and I like to see it through. Write up is here: https://bugzilla.mozilla.org/show_bug.cgi?id=880647 Ate the moment, the progress is all in 893838, because I needed the selectAll/deleteAll flags as part of the optimization. The changes are well documented, explaining each step of the delete operation optimizations. https://github.com/rwldrn/gaia/commit/8e2c4ef43ecccb14d2097adc36f1df164ec31cae This works incredibly well. The only remaining pieces: 1. Incoming messages should not be auto selected if selectAll flag is true 2. Delete cached nodes after manually selection. 3. Additional tests.

I forgot to mention: the work I'm doing in bug 893838 covers both threads and messages.

Hi Rick, I went through the bugs you pointed and I agree completely with your considerations (more on this later) but as Borja mentioned this is a leo+ and an urgent one too so we need to address this issue very shortly and your changes are extensive. Now back to the more general issue I fully support the approach you're describing in bug 880647 comment 2. In general one of the problems we have across all comm applications is a deep integration between DOM elements (i.e. UI) and the data/logic of the application. Now this is *bad* and decoupling the UI from the data model should be a priority but it implies a significant redesign of the application and I think we should all be aware of what's going on. so I would suggest CC'ing everybody involved in the SMS app in bug 880647 so everybody's aware of your new approach and we can discuss it further.

Hi Rick. I've seen your job in 'delete' operations, both in the ThreadList as in the MessageView, and it would be awesome to have this feature. However, this bug is leo+ and blocker for the release, and it's supposed to be landed 15th of July (actually we are a little bit late). So we should land this due to it's a requirement from leo, and we could take a look to the other bugs you mention and wait to be triaged, because one of them is 'leo?' and the other one is not even nominated to be a blocker. We could use both bugs as followups of this one, wdyt?

Partners requirements and schedules can't be always use as an excuse for going to quick and dirty solutions especially now that we're out of 1.0. I encourage you to go asap for a decoupled model, even if the bugs are not leo? or leo+.

Hi all, I've seen Rick's patch and it's really good for handling both threads and messages. Wayne will contact partner that whether we should focus on the performance issue in the stage.

(In reply to Fabrice Desré [:fabrice] from comment #59) > Partners requirements and schedules can't be always use as an excuse for > going to quick and dirty solutions especially now that we're out of 1.0. I > encourage you to go asap for a decoupled model, even if the bugs are not > leo? or leo+. I'm all for moving to a better architecture ASAP if we can get the leo+-pressure off this bug :-) Also, I'd like to profile Rick's patch to see how it affects a set of common activities (including deletions) that we know are currently performing poorly when large number of messages/threads are involved (besides having inconsistent results).

(In reply to Gabriele Svelto [:gsvelto] from comment #61) > (In reply to Fabrice Desré [:fabrice] from comment #59) > > Partners requirements and schedules can't be always use as an excuse for > > going to quick and dirty solutions especially now that we're out of 1.0. I > > encourage you to go asap for a decoupled model, even if the bugs are not > > leo? or leo+. > > I'm all for moving to a better architecture ASAP if we can get the > leo+-pressure off this bug :-) Also, I'd like to profile Rick's patch to see > how it affects a set of common activities (including deletions) that we know > are currently performing poorly when large number of messages/threads are > involved (besides having inconsistent results). Wish granted. Have discussed with partner today to go for a more complete solution and they agree that should be the case too. I'll leo- this bug and + the other so we know that should get done.

(In reply to Wayne Chang [:wchang] from comment #62) > Wish granted. > Have discussed with partner today to go for a more complete solution and > they agree that should be the case too. I'll leo- this bug and + the other > so we know that should get done. Thanks! If Borja is OK with it I think we should also close this bug as a duplicate of bug 893838.

(In reply to Gabriele Svelto [:gsvelto] from comment #63) > (In reply to Wayne Chang [:wchang] from comment #62) > > Wish granted. > > Have discussed with partner today to go for a more complete solution and > > they agree that should be the case too. I'll leo- this bug and + the other > > so we know that should get done. > > Thanks! If Borja is OK with it I think we should also close this bug as a > duplicate of bug 893838. For me it's ok. It would be great if you could help us profiling the new solution as well, for ensuring that we are reaching, at least, the results we achieved here. On the other hand, I would like to know which is going to be the 'roadmap' or the priority of this new solution, because probably we are working on parallel in different things, and it would be awesome to avoid duplicities here. I've seen a Gecko bug that could help us a lot for getting a better performance (check https://bugzilla.mozilla.org/show_bug.cgi?id=895734#c0), and also we have Rick's patch which should be profiled and could use some of the improvements that we discovered in this bug. So, should we create a patch for adding as well Gecko new functionality? Could we include all Gaia changes needed in Rick's patch? I would like to have a complete solution, having Gecko & Gaia improvements, instead of having different bugs & followups as we have decided that we can spend time on this. Thanks!

Comment on attachment 772055 [details] Pull request Clear the review before PM/partner have a final conclusion.

leo want to maintain the current status.

With partner's comment 66 I believe we're good to leave status quo and work on all changes required at once. i.e. we have more time to do things right on this here. (In reply to Borja Salguero [:borjasalguero] (PTO from 25th to 30th of July) from comment #64) > I would like to have a complete solution, > having Gecko & Gaia improvements, instead of having different bugs & > followups as we have decided that we can spend time on this. Thanks!

Therefore I'm duping against bug 893838 where Rick is working on a more complete solution.