Unusable files with Debian on Sheevaplug ! (eSata)

[theYinYeti]

Hello,

I have a strange problem I never had before. I don't usually use Debian, so I'm not sure if it comes from Debian or from the Sheevaplug.

Some files on my filesystem seem to be in bad shape, even though no fsck error seems to be reported in the log. There are these messages, though:

Code:

Dec 27 00:24:30 SERVER kernel: [   22.433383] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
Dec 27 00:24:30 SERVER kernel: [   22.442091] Scanning device for bad blocks
Dec 27 00:24:30 SERVER kernel: [   22.492202] Bad eraseblock 1239 at 0x000009ae0000
Dec 27 00:24:30 SERVER kernel: [   22.555544] Bad eraseblock 2820 at 0x000016080000
Dec 27 00:24:30 SERVER kernel: [   22.560304] Bad eraseblock 2821 at 0x0000160a0000
Dec 27 00:24:30 SERVER kernel: [   22.612275] Creating 3 MTD partitions on "orion_nand":
Dec 27 00:24:30 SERVER kernel: [   22.617441] 0x000000000000-0x000000100000 : "u-boot"
Dec 27 00:24:30 SERVER kernel: [   22.623006] 0x000000100000-0x000000500000 : "uImage"
Dec 27 00:24:30 SERVER kernel: [   22.628459] 0x000000500000-0x000020000000 : "root"

This is what happens:

Code:

SERVER:/var/log# free -m
-bash: /usr/bin/free: cannot execute binary file

Some more information:

Code:

SERVER:/var/log# cat /etc/debian_version
5.0.7
SERVER:/var/log# uname -a
Linux SERVER 2.6.32-5-kirkwood #1 Wed Oct 20 12:58:44 UTC 2010 armv5tel GNU/Linux
SERVER:/var/log# df -h
/dev/mmcblk0p2        1,7G  677M  951M  42% /
…

So:
1/ How can I tell Debian to check every file belonging to a .deb package, and then repair any broken file?
2/ How can I ensure such problem won't occur again?
3/ What is happening?!

Yves.

[marcus]

There's nothing abnormal about bad eraseblocks; they aren't related to the problems you are reporting.

[theYinYeti]

Here are all further messages from the kernel log (/var/log/kern.log) related to the SD card:

Code:

Dec 27 00:24:30 SERVER kernel: [   24.241099] mmc0: mvsdio driver initialized, using GPIO 47 for card detection
Dec 27 00:24:30 SERVER kernel: [   24.427266] mmc0: new SD card at address ae40
Dec 27 00:24:30 SERVER kernel: [   24.524911] mmcblk0: mmc0:ae40 SD02G 1.84 GiB
Dec 27 00:24:30 SERVER kernel: [   24.529545]  mmcblk0:
Dec 27 00:24:30 SERVER kernel: [   26.210260] EXT3-fs: INFO: recovery required on readonly filesystem.
Dec 27 00:24:30 SERVER kernel: [   26.216697] EXT3-fs: write access will be enabled during recovery.
Dec 27 00:24:30 SERVER kernel: [   33.357219] kjournald starting.  Commit interval 5 seconds
Dec 27 00:24:30 SERVER kernel: [   33.362783] EXT3-fs: recovery complete.
Dec 27 00:24:30 SERVER kernel: [   33.826782] EXT3-fs: mounted filesystem with ordered data mode.
Dec 27 00:24:30 SERVER kernel: [   39.575903] EXT3 FS on mmcblk0p2, internal journal

This all seems normal. The FS recovery is because I had forgotten to flash the kernel after upgrading it; next reboot was not clean (this is fine now). My problem, however, was already there before the upgrade.

Complete system mounts are:

Code:

/dev/mmcblk0p2 on / type ext3 (rw,relatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mmcblk0p1 on /boot type ext2 (rw,relatime)

Doesn't anyone have any idea on what could be wrong? On how I could diagnose, for a start? I mean: what files, apart from /usr/bin/free, are broken? Maybe dpkg could help? Is it a serious problem? Should I worry for my server…

Please help!

Yves.

[theYinYeti]

Well, I'm perfecting my Debian skills due to this… I searched Google and found about the “debsums” tool. The result was very interesting:

Code:

SERVER:/var/log# debsums | grep -v ' OK$' >/tmp/debsums.log
debsums: no md5sums for at
debsums: no md5sums for binutils
debsums: no md5sums for bzip2
debsums: no md5sums for ed
debsums: no md5sums for g++
debsums: no md5sums for initscripts
debsums: no md5sums for installation-report
debsums: no md5sums for libbz2-1.0
debsums: no md5sums for libdb4.5
debsums: no md5sums for libgdbm3
debsums: no md5sums for liblockfile1
debsums: no md5sums for mawk
debsums: no md5sums for module-init-tools
debsums: no md5sums for netbase
debsums: no md5sums for sysv-rc
debsums: no md5sums for sysvinit
debsums: no md5sums for sysvinit-utils
debsums: no md5sums for update-inetd

I don't know if I should worry about the missing md5sums…

Code:

SERVER:/var/log# cat /tmp/debsums.log
/usr/bin/pgrep                                                            FAILED
/usr/bin/watch                                                            FAILED
/usr/bin/skill                                                            FAILED
/usr/bin/pwdx                                                             FAILED
/usr/bin/top                                                              FAILED
/usr/bin/tload                                                            FAILED
/usr/bin/slabtop                                                          FAILED
/usr/bin/pmap                                                             FAILED
/usr/bin/vmstat                                                           FAILED
/usr/bin/free                                                             FAILED
/usr/bin/uptime                                                           FAILED
/sbin/sysctl                                                              FAILED
/bin/kill                                                                 FAILED
/bin/ps                                                                   FAILED

Googling for all those files' basenames together led me to one single Debian package: procps. Does this ring a bell for anyone? Is there a possibility this would come from a compromised server? Or is this simply a broken install of procps? Should the following command be enough to repair?

Code:

apt-get --reinstall install procps

When I try to execute it, here is the result:

Code:

SERVER:/var/log# LANG=C apt-get --reinstall install procps
…
(Reading database ... 34837 files and directories currently installed.)
Preparing to replace procps 1:3.2.7-11 (using .../procps_1%3a3.2.7-11_armel.deb) ...
Unpacking replacement procps ...
dpkg: error processing /var/cache/apt/archives/procps_1%3a3.2.7-11_armel.deb (--unpack):
 unable to make backup link of `./usr/bin/pgrep' before installing new version: Operation not permitted
dpkg-deb: subprocess paste killed by signal (Broken pipe)
dpkg: error while cleaning up:
 subprocess post-removal script returned error exit status 1
Processing triggers for menu ...
Processing triggers for man-db ...
Errors were encountered while processing:
 /var/cache/apt/archives/procps_1%3a3.2.7-11_armel.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Thanks for any advice.

Yves.

[theYinYeti]

For those having the same problem, googling a bit more gave me this link:
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/93320
then this one:
http://forum.ubuntuusers.de/topic/kein-upgrade-fuer-tzdata/#post-1154627
leading to this solution (depends on what actual attributes are set on the files):

Code:

SERVER:~# dpkg -L procps | grep bin/ | while read f; do chattr -ia "$f"; lsattr "$f"; done
------------------- /usr/bin/pgrep
------------------- /usr/bin/watch
------------------- /usr/bin/w.procps
------------------- /usr/bin/skill
------------------- /usr/bin/pwdx
------------------- /usr/bin/top
------------------- /usr/bin/tload
------------------- /usr/bin/slabtop
------------------- /usr/bin/pmap
------------------- /usr/bin/vmstat
------------------- /usr/bin/free
------------------- /usr/bin/uptime
------------------- /sbin/sysctl
------------------- /bin/kill
------------------- /bin/ps
SERVER:~# LANG=C apt-get --reinstall install procps
…
(Reading database ... 34847 files and directories currently installed.)
Preparing to replace procps 1:3.2.7-11 (using .../procps_1%3a3.2.7-11_armel.deb) ...
Unpacking replacement procps ...
Processing triggers for menu ...
Processing triggers for man-db ...
Setting up procps (1:3.2.7-11) ...
Setting kernel variables (/etc/sysctl.conf)...done.
SERVER:~# free -m
             total       used       free     shared    buffers     cached
Mem:           503        481         22          0        130        202
-/+ buffers/cache:        148        355
Swap:          249          1        248

The point seems to NOT have any attributes set on the upgraded files.
As I did not set the attributes myself, I guess this is a packaging bug for procps, affecting a particular upgrade…

Still, this does not explain where the problem came from in the first place… any idea? Compromised system? Bad luck? Flaky SD card?

Yves.

[radael]

First, thank you for continuing to post the results of your research.  It's tough when there is no answer.

In your introductory post it was not exactly clear how you encountered the problem.  Probably you had done nothing new or special, but it can be helpful to be clear that the system had been stable for a period of time, and without any change, simply began "misbehaving."

That said, this situation is very unusual.  I have no answers, only observations.

I doubt it was a system compromise, normally crackers want to use the system, not damage it.  And, if one is motivated to damage it, there are far better ways to do so than make procps temporarily unusable.

It could, hypothetically, be a problem with your SD card, except the fact that it only effected the file permissions of one entire package.  The conditions seem so unlikely that it is hardly worth considering.

Seems that the best answer is that this was a packaging problem.

If I were in the same situation, my next questions would be along the lines of:

-- What is status of the permissions for the other files in /usr/bin ?

-- Had you recently done any installation that might have been related to the problem?

-- When was the last time you used any of the commands from that package?

Doubt that this is much help, but again, thanks for sharing.

[theYinYeti]

Thank you very much for your logical and helpful observations.

I picked three random commands (a2p, whatis, xauth); each had no attributes set. This concurs to my previous conclusion/solution. Apart from that, nothing special; I don't usually access this server directly, only via its services.

I think I'll leave it at that, for now. I consider this matter solved (and a Debian packaging bug).

Yves.

[theYinYeti]

VERY IMPORTANT !!! EXIM4 SECURITY HOLE

I don't know if anyone is still reading this thread… anyway… the above problem may not at all be a packaging problem, but this ROOTKIT instead:
http://www.reddit.com/r/netsec/comments/en650/details_of_the_root_kit_that_go
http://lists.debian.org/debian-security/2010/12/msg00093.html

Indeed, this attack changes most procps files, and leaves some traces in some exim directories, which I found on my system. I was lucky that no harm was done to my system, thanks to the other protections I have. Still, be carefull and look for unusual content in /var/spool/exim4 for a start.

Yves.

[radael]

Wow!, Thank you for posting that follow-up.