In particular, create a IO Watcher which reads /proc/kmsg and looks for particular messages, like the OOM ones, which look like: <4>[10-24 02:36:18.671] [28: kswapd0]select 406 (Settings), adj 6, size 9205, to kill <4>[10-24 02:36:18.671] [28: kswapd0]send sigkill to 406 (Settings), adj 6, size 9205 and log them to logcat.

Hi Dave, After discussed with Thinker, he said that we can just call |cat /proc/kmsg | (while read var; do log "$var"; done)| in init.rc. Then we can see kmsg in logcat. What do you think?

That would work, although it would log alot more stuff than what I was thinking of. This could certainly be implemented as a script, although I'd be concerned about how much memory it would require.

Hello there, can I get this one assigned to me? Cheers!

Hi there, as an opener, here are a dozen questions of mine. I would have to thank you in advance for the answer :) 1) The discussion so far suggests that there are two ways of implementing this one. If it's going to be implemented in C++, am I supposed to code against the m-c repository? 2) Where should I plug in my code? Who will call it? Will it be something like a startup service? 3) How shall I test and debug the changes? Is the ARM emulator build appropriate for debugging this case? 4) Is IO Watcher some already defined class that I would make use of? 5) Does logcat refer to the logging system of Android? If so, is this bug specific to Android? 6) What does OOM mean?

It seems to me that we should measure the memory usage of the script and, if it's not good enough, we should write a little C/C++ program from scratch. Either way the program would go in B2G/gonk-misc. Genti, I don't suppose you have a B2G phone. You /might/ be able to use the emulator for this, although I've had a lot of trouble getting the emulator to work. IOW this might not be a good bug for you. https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_OS/Building_and_installing_Firefox_OS?redirectlocale=en-US&redirectslug=Firefox_OS%2FBuilding_and_installing_Firefox_OS

(In reply to Genti Tola [:d0kt0r1] from comment #4) > Hi there, as an opener, here are a dozen questions of mine. I would have to > thank you in advance for the answer :) > 1) The discussion so far suggests that there are two ways of implementing > this one. If it's going to be implemented in C++, am I supposed to code > against the m-c repository? We could do this as a completely standalone application, or we could do it as something builtin to gecko (m-c). I thiink it depends on the complexity and how much memory it consumes. > 2) Where should I plug in my code? Who will call it? Will it be something > like a startup service? If we do a standalone utility, then creating a subdirectory in gonk-misc (as jelbar suggests) seems like a reasonable place to me. > 3) How shall I test and debug the changes? Is the ARM emulator build > appropriate for debugging this case? You should be able to test using the emulator. > 4) Is IO Watcher some already defined class that I would make use of? You'd only need to use that if we put this into gecko. Starting out with a standalone utility seems to make sense (if nothing else, then as a prototype). With a standalone utility you can do regular blocking reads. > 5) Does logcat refer to the logging system of Android? If so, is this bug > specific to Android? Yes and Yes > 6) What does OOM mean? Out Of Memory

Ok, just managed to set up the ARM emulator. Now I am trying to find the right init.rc file so I can make the script start on startup. The problem is that an exhaustive search for init.rc yields many files and I do not know which is the correct one to modify. Afterwards I just plan to develop the following(based on StevenLee's Comment 1 ) : grep "pattern" /proc/kmsg | (while read var; do log "$var"; done) In here I need to know: 1. What kind of message patterns I would have to filter out. Maybe those that contain the word 'kill'? 2. Is 'log' a real command? If so, I suppose log command is making some changes to a file which logcat is able to read? Or the StevenLee's extract is just a pseudocode? Then I suppose I would write another script that dumps the memory footprint of my script so we could decide if it's OK to leave this as a script. What would be a an upper bound of memory consuption for the script? Thanks.

(In reply to Genti Tola [:d0kt0r1] from comment #7) > Ok, just managed to set up the ARM emulator. > > Now I am trying to find the right init.rc file so I can make the script > start on startup. The problem is that an exhaustive search for init.rc > yields many files and I do not know which is the correct one to modify. I'd probably use the init.b2g.rc file instead. I verified that on the emulator, if you edit B2G/gonk-misc/init.b2g.rc then this will show up in /init.b2g.rc when you run the emulator. > Afterwards I just plan to develop the following(based on StevenLee's > Comment 1 ) : > grep "pattern" /proc/kmsg | (while read var; do log "$var"; done) > In here I need to know: > 1. What kind of message patterns I would have to filter out. Maybe those > that contain the word 'kill'? This bug is mostly interested in the messages from the Android low memory killer: http://lxr.linux.no/#linux+v3.10.6/drivers/staging/android/lowmemorykiller.c#L147 http://lxr.linux.no/#linux+v3.10.6/drivers/staging/android/lowmemorykiller.c#L151 And these one (although additional debug needs to be enabled to see them: http://lxr.linux.no/#linux+v3.10.6/drivers/staging/android/lowmemorykiller.c#L96 http://lxr.linux.no/#linux+v3.10.6/drivers/staging/android/lowmemorykiller.c#L104 http://lxr.linux.no/#linux+v3.10.6/drivers/staging/android/lowmemorykiller.c#L159 > 2. Is 'log' a real command? If so, I suppose log command is making some > changes to a file which logcat is able to read? Or the StevenLee's extract > is just a pseudocode? log is a real command: adb shell log This is a test adb logcat -d and you'll see "This is a test" at the end of the logcat. > Then I suppose I would write another script that dumps the memory footprint > of my script so we could decide if it's OK to leave this as a script. What > would be a an upper bound of memory consuption for the script? I don't know. We're very tightly memory constrained, so the less memory we use the better. adb shell procrank or adb shell cat /proc/PID/status (replace PID with the PID of your process) will show us how much memory is being used.

(In reply to Dave Hylands [:dhylands] from comment #8) Thank you Dave for the details. Sorry for my late reply, I just found out that I had to put some status when not available. I am a little bit confused about the way I am suppposed to land the patch. I've been using Mercurial queues so far, but the B2G source is under GIT. I've seen this tutorial ove the web for GIT at Mozilla: http://bluishcoder.co.nz/2011/04/16/my-git-workflow-for-mozilla-development.html but I do not know if it applies for B2G. It's even more confusing that: https://wiki.mozilla.org/B2G/Hacking does have only a tutorial on Mercurial whilst the source is in GIT.

(In reply to Genti Tola [:d0kt0r1] from comment #9) > (In reply to Dave Hylands [:dhylands] from comment #8) > Thank you Dave for the details. Sorry for my late reply, I just found out > that I had to put some status when not available. > > I am a little bit confused about the way I am suppposed to land the patch. > I've been using Mercurial queues so far, but the B2G source is under GIT. > > I've seen this tutorial ove the web for GIT at Mozilla: > http://bluishcoder.co.nz/2011/04/16/my-git-workflow-for-mozilla-development. > html > but I do not know if it applies for B2G. > > It's even more confusing that: > https://wiki.mozilla.org/B2G/Hacking > does have only a tutorial on Mercurial whilst the source is in GIT. Sorry again, I guess I overlooked it. There is a part in this page that explains that I have to use git diff.

So for landing stuff that's in github (which is everything but gecko) you don't need to prepare patches. The normal github workflow goes something like this: https://github.com/sevntu-checkstyle/sevntu.checkstyle/wiki/Development-workflow-with-Git%3A-Fork,-Branching,-Commits,-and-Pull-Request So you're working in your own copy of the repository and you do a pull request. You should then request a review in the bug and include the URL to the pull request. If there are comments in the review then you should address them and update your branch (if you use "git push -f BRANCH_NAME" then it will auto-update your pull request (so you don't need to create a new one). Alternatively, you can prepase a patch. You need to commit your changes into your local tree and then I normally use: git format-patch -U8 HEAD~ This will create a patch between HEAD and HEAD~ and you can then attach the patch to the bug. I personally prefer this approach, because then I can use the bugzilla review stuff. Others prefer the github approach. I'd say use whichever method you prefer.

OK, here's another question: I wrote my solution using regular expressions. The problem that I am facing is that I cannot use any regex engine. I try to run grep, egrep, sed, awk, perl on the device by using "adb shell 'command'" but for any of those commands it replies that they are not found.

I don't think that we really need anything as complicated as regular expressions. I thing it would be perfectly acceptable to do searches like: if the line contains "select" and "to kill" log it. if the line contains "send sigkill" log it. If you're writing a shell script, then this could be written something like: if [ "${x/*select*to kill*/select to kill}" = "select to kill" ]; then it matches If you're using C then you can use strstr.

In my shell example, x was the name of the variable, you can replace it with something else more appropriate. I test this type of thing interactively by doing: x="blah blah select something to kill blah blah" if [ "${x/*select*to kill*/select to kill}" = "select to kill" ]; then echo yes; else echo no; fi

Ok I'm having issues with getting my script running. Here's what I did: 1) Modified gonk-misc/init.b2g.rc by adding: service OOMLogger /system/bin/oom-msg-logger.sh class main user root 2) The oom-msg-logger.sh is executable and contains the following code: #!/system/bin/sh log "OOM Message Logger Started" cat /proc/kmsg | while read line do if [[ ("${line/*select*to kill*/select to kill}" = "select to kill") || ("${line/*send sigkill to*/send sigkill to}" = "send sigkill to") || ("${line/*lowmem_shrink*, return*/lowmem_shrink , return}" = "lowmem_shrink , return") || ("${line/*lowmem_shrink*, ofree*/lowmem_shrink , ofree}" = "lowmem_shrink , ofree") ]]; then log $line; fi done 3) Modified gonk-misc/Android.mk to add the following(even though I have no idea what LOCAL_MODULE_CLASS means): include $(CLEAR_VARS) LOCAL_MODULE := oom-msg-logger.sh LOCAL_MODULE_TAGS := optional LOCAL_MODULE_CLASS := DATA LOCAL_SRC_FILES := oom-msg-logger.sh LOCAL_MODULE_PATH := $(TARGET_OUT_EXECUTABLES) include $(BUILD_PREBUILT) 4) Run the ./build.sh 5) Checked for the "OOM Message Logger Started" message using logcat, but no luck. I tried to use procrank to see if the process is running, but no luck. the same goes for ps aux command.

On many devices, we don't build the kernel/root file system, so even though you modified init.b2g.rc, if you do: adb shell cat /init.b2g.rc you may discover that your changes are not actually present on the phone. I'm going to guess that oom-msg-logger.sh isn't actually in /system/bin on your phone. I'm pretty sure that you need to edit gonk-misc/b2g.mk and add oom-msg-logger to PRODUCT_PACKAGES. I think you should also change LOCAL_MODULE := oom-msg-logger.sh to be LOCAL_MODULE := oom-msg-logger Whatever the module name is, it needs to be listed in the PRODUCT_PACKAGES to be included in the image for the phone. So you need to verify that all of the appropriate changes that you made actually propogated to the phone. Unfortunately, the init.b2g.rc change will require a kernel upgrade for the phone. As a temporary hack, you could try editing b2g.sh and have it launch oom-msg-logger.sh if it isn't running already. You should be able to do something like get_pid_by_name() { echo $(toolbox ps "$1" | (read header; read user pid rest; echo -n $pid)) } pid=$(get_pid_by_name "oom-msg-logger.sh") if [ -z "${pid}" ]; then echo "oom-msg-logger.sh isn't running" else echo "oom-msg-logger.sh is already runnging as ${pid}" fi

(In reply to Dave Hylands [:dhylands] from comment #16) > On many devices, we don't build the kernel/root file system, so even though > you modified init.b2g.rc, if you do: > > adb shell cat /init.b2g.rc > > you may discover that your changes are not actually present on the phone. > > I'm going to guess that oom-msg-logger.sh isn't actually in /system/bin on > your phone. I'm pretty sure that you need to edit gonk-misc/b2g.mk and add > oom-msg-logger to PRODUCT_PACKAGES. > > I think you should also change > > LOCAL_MODULE := oom-msg-logger.sh > > to be > > LOCAL_MODULE := oom-msg-logger > > Whatever the module name is, it needs to be listed in the PRODUCT_PACKAGES > to be included in the image for the phone. > > So you need to verify that all of the appropriate changes that you made > actually propogated to the phone. I made all these changes, then rebuilt it, but still |adb shell ls system/bin| does not show oom-msg-logger.sh to be present.

(In reply to Genti Tola [:d0kt0r1] from comment #17) > (In reply to Dave Hylands [:dhylands] from comment #16) > > On many devices, we don't build the kernel/root file system, so even though > > you modified init.b2g.rc, if you do: > > > > adb shell cat /init.b2g.rc > > > > you may discover that your changes are not actually present on the phone. > > > > I'm going to guess that oom-msg-logger.sh isn't actually in /system/bin on > > your phone. I'm pretty sure that you need to edit gonk-misc/b2g.mk and add > > oom-msg-logger to PRODUCT_PACKAGES. > > > > I think you should also change > > > > LOCAL_MODULE := oom-msg-logger.sh > > > > to be > > > > LOCAL_MODULE := oom-msg-logger > > > > Whatever the module name is, it needs to be listed in the PRODUCT_PACKAGES > > to be included in the image for the phone. > > > > So you need to verify that all of the appropriate changes that you made > > actually propogated to the phone. > I made all these changes, then rebuilt it, but still |adb shell ls > system/bin| does not show oom-msg-logger.sh to be present. OK, now I got my script running at startup. I did put a log statement at the beginning of the script to log that the "OOM Script is Started". When I run adb logcat -d | grep "OOM" I can see that the startup message is logged multiple times. Isn't the script supposed to be called only once?

If the script exits for any reason, then if its a service, then init will restart it automatically. If you want to attach a WIP (work-in-progress) patch to the bug, I can try it see what happens for me. What phone are you testing this on?

(In reply to Dave Hylands [:dhylands] from comment #19) > If the script exits for any reason, then if its a service, then init will > restart it automatically. > > If you want to attach a WIP (work-in-progress) patch to the bug, I can try > it see what happens for me. > > What phone are you testing this on? I do not have any phone. I am testing it on the ARM Emulator.

(In reply to Dave Hylands [:dhylands] from comment #16) > get_pid_by_name() { > echo $(toolbox ps "$1" | (read header; read user pid rest; echo -n $pid)) > } > > pid=$(get_pid_by_name "oom-msg-logger.sh") > if [ -z "${pid}" ]; then > echo "oom-msg-logger.sh isn't running" > else > echo "oom-msg-logger.sh is already runnging as ${pid}" > fi I attached this piece of code at b2g.sh, but I can't see any output(maybe I have to use some command through adb to see the emulator console output?). This is also present in the patch that I attached.

(In reply to Genti Tola [:d0kt0r1] from comment #22) > I attached this piece of code at b2g.sh, but I can't see any output(maybe I > have to use some command through adb to see the emulator console output?). > This is also present in the patch that I attached. You wouldn't seen any output, since stdout is redirected to /dev/null when b2g.sh is launch from android. To see the echo output, you'd need to do: adb shell stop b2g adb shell /system/bin/b2g.sh or you could change the echo statements to log statements so that they show up in logcat.

Comment on attachment 799411 [details] [diff] [review] Create a logger to put selected messages from kmsg into logcat I made a bunch of comments on github. Since it looks like your tree wasn't on the proper branch, I'm pretty sure you'll need to do a new pull request. So I'm going to minus this. If you need help with getting synced up, we can arrange a time to walk through the steps needed.

(In reply to Dave Hylands [:dhylands] from comment #24) > Comment on attachment 799411 [details] [diff] [review] > Create a logger to put selected messages from kmsg into logcat > If you need help with getting synced up, we can arrange a time to walk > through the steps needed. That's great. I have millions of questions to ask, but I'll try to gzip-tar them :) What's your availability time? I live in Albania(right now, due to daylight saving we're at UTC +2 CET)

Comment on attachment 801500 [details] [diff] [review] Implementation of OOM logger Looks good

(In reply to Dave Hylands [:dhylands] from comment #27) > Comment on attachment 801500 [details] [diff] [review] > Implementation of OOM logger > > Looks good Cool, shall I mark this one as resolved?

So, this still needs to be landed. Since this goes on github, you should create a pull request. Once the pull request gets merged, then this bug would get marked as resolved (by whoever does the merge). I also notice that there are 2 patches attached to this bug. The r-'d one should probably be marked as obsolete.

(In reply to Dave Hylands [:dhylands] from comment #27) > Comment on attachment 801500 [details] [diff] [review] > Implementation of OOM logger > > Looks good I suppose the last pull request of mine is OK right? I do not know how to render the previous obsolete without adding a new attachment. Is there any other way? Thanks.

To change the status of an existing attachment, click on Details, then click on (edit details) at the end of the title and the next page will have an obsolete checkbox that can be ticked. The existing pull request needs to be rebased, it can't be merged in its present form. Let me know when its rebased and I can go ahead and merge it.

Comment on attachment 799411 [details] [diff] [review] Create a logger to put selected messages from kmsg into logcat >https://github.com/mozilla-b2g/gonk-misc/pull/112

(In reply to Dave Hylands [:dhylands] from comment #31) > To change the status of an existing attachment, click on Details, then click > on (edit details) at the end of the title and the next page will have an > obsolete checkbox that can be ticked. > > The existing pull request needs to be rebased, it can't be merged in its > present form. > > Let me know when its rebased and I can go ahead and merge it. I just rebased it. Regards.