• Home
  • Help
  • Search
  • Login
  • Register
Pages: [1] 2
Author Topic: pdflush and syslogd 100% CPU usage  (Read 9554 times)
geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« on: January 01, 2010, 03:16:19 PM »

Hi guys, running the standard Ubuntu install on my Sheevaplug.

Almost all of the time one or the other of pdflush/syslogd are taking 100% CPU usage. 1 minute load average is always around 4-5.

Any idea why this might be?

Thanks,
Jon
Logged

dattaway
Jr. Member
**

Karma: 5
Posts: 91



View Profile WWW
« Reply #1 on: January 01, 2010, 03:35:09 PM »

Any chance they are writing to a slow sd card?
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #2 on: January 01, 2010, 03:48:16 PM »

That is entirely possible - is there any way to stop them or any other actions worth taking to stop this?

I have stopped syslogd but pdflush is still taking 100% CPU occasionally.

Thanks for the reply Smiley
Logged

dattaway
Jr. Member
**

Karma: 5
Posts: 91



View Profile WWW
« Reply #3 on: January 01, 2010, 05:20:50 PM »

From my experience, the slow storage device might have been mounted synchronously, or some log files might have the sync file attribute set.  That means the writes are immediate.  Good for log files when debugging crashes, because the last events before impending doom are saved.  Bad for slow sdcards!

You can tune the kernel to flush every minute, hour, day, or never. Check to see how often:

cat /proc/sys/vm/dirty_writeback_centisecs

and if its 500, you can increase it to 50000 by "cat 50000 > /proc/sys/vm/dirty_writeback_centisecs"

and give your storage systems a rest.  But if the power ever goes out without a "sync" command, massive data is lost.

You can also check the mounted file systems for the any sync options by checking /etc/mtab:

cat /etc/mtab

You can check the rarely used, but sneaky file attributes with lsattr and set them from chattr...

from "man chattr"
"When  a  file with the `S' attribute set is modified, the changes are written synchronously on the disk; this is equivalent to the `sync' mount option applied to a subset of the files.

All kinds of ways to tune the system and make it fast or force data updates.
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #4 on: January 02, 2010, 07:40:50 AM »

Code:
cat /proc/sys/vm/dirty_writeback_centisecs

returns 30000, which I assume is OK even for a slower SD card.

Here are the contents of /etc/mtab:

Code:
rootfs / rootfs rw 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
varrun /var/run tmpfs rw,nosuid,mode=0755 0 0
varlock /var/lock tmpfs rw,noexec,nosuid,nodev,mode=1777 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
tmpfs /var/cache/apt tmpfs rw,noatime 0 0
/dev/mmcblk0p1 /mnt vfat rw 0 0

Anything obviously wrong here?

I can't see any log files on the SD card, here are it's contents:

Code:
drwxr-xr-x 11 root root   32768 Dec 31 01:32 .
drwxr-xr-x 22 root root       0 Jan  1  1970 ..
-rwxr-xr-x  1 root root   12292 Dec 17 20:00 .DS_Store
drwxr-xr-x  3 root root   32768 Dec 17 17:16 .TemporaryItems
-rwxr-xr-x  1 root root    4096 Dec 17 17:16 ._.TemporaryItems

Thanks for the help - still not sure what to do though!

Jon
Logged

dattaway
Jr. Member
**

Karma: 5
Posts: 91



View Profile WWW
« Reply #5 on: January 02, 2010, 04:55:27 PM »

You can always kill syslogd.  Might be something that keeps logging in a loop.

killall -9 syslogd
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #6 on: January 04, 2010, 05:33:49 AM »

I've killed syslogd so that's one way of solving that one.

At the moment pdflush is still takine 100% CPU very frequently - for example at the moment:

Code:
12:32:47 up 4 days, 21:48,  1 user,  load average: 21.50, 12.64, 7.08

Trying to kill this fails and may be a bad idea anyway. Any ideas what I can do about pdflush?

Thanks again for your help Smiley
Jon
Logged

dattaway
Jr. Member
**

Karma: 5
Posts: 91



View Profile WWW
« Reply #7 on: January 04, 2010, 04:53:31 PM »

Try copying a large file and see if the load average gets unreasonable.
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #8 on: January 05, 2010, 02:01:43 PM »

It does, but it does it randomly anyway. I left the plug doing absolutely nothing for about 12 hours and when I tried to SSH into it, getting to the bash prompt was very slow and when I eventually did get in, it showed pdflush taking 100% CPU.
Logged

birdman
Sr. Member
****

Karma: 4
Posts: 440


View Profile WWW
« Reply #9 on: January 06, 2010, 01:43:39 PM »

Look into what is producing a load-average of > 20?
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #10 on: January 06, 2010, 03:55:37 PM »

Well it's pdflush, but the question is what to do about it! Killing it doesn't work, and I'm not sure that killing it is a good idea anyway...
Logged

fragfutter
Sr. Member
****

Karma: 12
Posts: 280


View Profile
« Reply #11 on: January 07, 2010, 01:12:35 AM »

pdflush writes dirty memory pages to "disk". From this thread i would guess, that your SD Card is either very slow, defective or you are doing a lot of write transaction to the sdcard.

Try with a different card or from nand. Or read up a lot of documentation and start to use tmpfs in a few places, tune dirty_writebacks and other stuff.
Logged

birdman
Sr. Member
****

Karma: 4
Posts: 440


View Profile WWW
« Reply #12 on: January 07, 2010, 02:01:27 PM »

Well it's pdflush, ...
If pdflush is producing a load average of > 20 that would mean you have > 20 of them running.  Do you?
Logged

fragfutter
Sr. Member
****

Karma: 12
Posts: 280


View Profile
« Reply #13 on: January 08, 2010, 02:15:25 AM »

no you need pdflush to be at 100% blocking paging requests. All other processes will be blocked because they wait for disc activity. The don't us CPU, but want to do something, so they produce load.

For example you can manage to get a NFS Fileserver to loads of 100+ but ssh login is still smooth (system disk seperate from filedisks). Load values need to be interpreted.
Logged

geforce
Newbie
*

Karma: 1
Posts: 12


View Profile
« Reply #14 on: January 14, 2010, 03:56:28 PM »

It did it again today, although it seems to be less frequent since I've changed the SD card to ext2 format.

The 1min load average went to ~33 and doing a "ps aux" afterwards showed that it was [pdflush] taking all the CPU.

During the time it was doing so, the plug was completely unresponsive. SSH login attempts timed out and existing sessions hung. The plug did respond to ping, but nothing else.
Logged

Pages: [1] 2
Print
Jump to: