pdflush and syslogd 100% CPU usage

[geforce]

Hi guys, running the standard Ubuntu install on my Sheevaplug.

Almost all of the time one or the other of pdflush/syslogd are taking 100% CPU usage. 1 minute load average is always around 4-5.

Any idea why this might be?

Thanks,
Jon

[dattaway]

Any chance they are writing to a slow sd card?

[geforce]

That is entirely possible - is there any way to stop them or any other actions worth taking to stop this?

I have stopped syslogd but pdflush is still taking 100% CPU occasionally.

Thanks for the reply

[dattaway]

From my experience, the slow storage device might have been mounted synchronously, or some log files might have the sync file attribute set.  That means the writes are immediate.  Good for log files when debugging crashes, because the last events before impending doom are saved.  Bad for slow sdcards!

You can tune the kernel to flush every minute, hour, day, or never. Check to see how often:

cat /proc/sys/vm/dirty_writeback_centisecs

and if its 500, you can increase it to 50000 by "cat 50000 > /proc/sys/vm/dirty_writeback_centisecs"

and give your storage systems a rest.  But if the power ever goes out without a "sync" command, massive data is lost.

You can also check the mounted file systems for the any sync options by checking /etc/mtab:

cat /etc/mtab

You can check the rarely used, but sneaky file attributes with lsattr and set them from chattr...

from "man chattr"
"When  a  file with the `S' attribute set is modified, the changes are written synchronously on the disk; this is equivalent to the `sync' mount option applied to a subset of the files.

All kinds of ways to tune the system and make it fast or force data updates.

[geforce]

Code:

cat /proc/sys/vm/dirty_writeback_centisecs

returns 30000, which I assume is OK even for a slower SD card.

Here are the contents of /etc/mtab:

Code:

rootfs / rootfs rw 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
varrun /var/run tmpfs rw,nosuid,mode=0755 0 0
varlock /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
tmpfs /var/cache/apt tmpfs rw,noatime 0 0
/dev/mmcblk0p1 /mnt vfat rw 0 0

Anything obviously wrong here?

I can't see any log files on the SD card, here are it's contents:

Code:

drwxr-xr-x 11 root root   32768 Dec 31 01:32 .
drwxr-xr-x 22 root root       0 Jan  1  1970 ..
-rwxr-xr-x  1 root root   12292 Dec 17 20:00 .DS_Store
drwxr-xr-x  3 root root   32768 Dec 17 17:16 .TemporaryItems
-rwxr-xr-x  1 root root    4096 Dec 17 17:16 ._.TemporaryItems

Thanks for the help - still not sure what to do though!

Jon

[dattaway]

You can always kill syslogd.  Might be something that keeps logging in a loop.

killall -9 syslogd

[geforce]

I've killed syslogd so that's one way of solving that one.

At the moment pdflush is still takine 100% CPU very frequently - for example at the moment:

Code:

12:32:47 up 4 days, 21:48,  1 user,  load average: 21.50, 12.64, 7.08

Trying to kill this fails and may be a bad idea anyway. Any ideas what I can do about pdflush?

Thanks again for your help
Jon

[dattaway]

Try copying a large file and see if the load average gets unreasonable.

[geforce]

It does, but it does it randomly anyway. I left the plug doing absolutely nothing for about 12 hours and when I tried to SSH into it, getting to the bash prompt was very slow and when I eventually did get in, it showed pdflush taking 100% CPU.

[birdman]

Look into what is producing a load-average of > 20?

[geforce]

Well it's pdflush, but the question is what to do about it! Killing it doesn't work, and I'm not sure that killing it is a good idea anyway...

[fragfutter]

pdflush writes dirty memory pages to "disk". From this thread i would guess, that your SD Card is either very slow, defective or you are doing a lot of write transaction to the sdcard.

Try with a different card or from nand. Or read up a lot of documentation and start to use tmpfs in a few places, tune dirty_writebacks and other stuff.

[birdman]

Well it's pdflush, ...

If pdflush is producing a load average of > 20 that would mean you have > 20 of them running.  Do you?

[fragfutter]

no you need pdflush to be at 100% blocking paging requests. All other processes will be blocked because they wait for disc activity. The don't us CPU, but want to do something, so they produce load.

For example you can manage to get a NFS Fileserver to loads of 100+ but ssh login is still smooth (system disk seperate from filedisks). Load values need to be interpreted.

[geforce]

It did it again today, although it seems to be less frequent since I've changed the SD card to ext2 format.

The 1min load average went to ~33 and doing a "ps aux" afterwards showed that it was [pdflush] taking all the CPU.

During the time it was doing so, the plug was completely unresponsive. SSH login attempts timed out and existing sessions hung. The plug did respond to ping, but nothing else.