Dead SD Card after 3 weeks : caused by Sheevaplug or something else?

[mundhra]

good to know you've made some progress. i haven't had much time to look at it, but i'm trying to figure out if the output from lsof will help.

hrm, after stopping exim4, rsyslog, and cron the mountro script worked. i enabled them one by one and... mountro still works. 

okay, i've eliminated exim4 and it's not just cron; after a reboot:

Code:

tardis:~# /etc/init.d/rsyslog restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.
tardis:~# /etc/init.d/cron restart
Restarting periodic command scheduler: cron.
tardis:~# mountro
tardis:~# mount
/dev/mmcblk0p2 on / type ext2 (ro,noatime,errors=remount-ro)

not quite sure how to use this knowledge, though. 

* here's the lsof output. i don't see any omissions, but the DEVICE is different (from 179,2 to 0,16) for the rsyslog /dev and /var files, as well as the cron /var files.

Code:

tardis:~# lsof -c rsyslog -c cron
COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
rsyslogd 949 root  cwd    DIR      179,2     4096          2 /
rsyslogd 949 root  rtd    DIR      179,2     4096          2 /
rsyslogd 949 root  txt    REG      179,2   264652     362974 /usr/sbin/rsyslogd
rsyslogd 949 root  mem    REG      179,2   117596     572347 /lib/ld-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    82124     361787 /usr/lib/libz.so.1.2.3.4
rsyslogd 949 root  mem    REG      179,2   120294     572332 /lib/libpthread-2.10.2.so
rsyslogd 949 root  mem    REG      179,2     9808     572356 /lib/libdl-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    26628     572350 /lib/librt-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    47204     572375 /lib/libgcc_s.so.1
rsyslogd 949 root  mem    REG      179,2  1201860     572339 /lib/libc-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    18652     378117 /usr/lib/rsyslog/lmnet.so
rsyslogd 949 root  mem    REG      179,2    42680     572345 /lib/libnss_files-2.10.2.so
rsyslogd 949 root  mem    REG      179,2     9136     378124 /usr/lib/rsyslog/imuxsock.so
rsyslogd 949 root  mem    REG      179,2    19844     378118 /usr/lib/rsyslog/imklog.so
rsyslogd 949 root  mem    REG      179,2    26480     572338 /lib/libnss_compat-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    75732     572351 /lib/libnsl-2.10.2.so
rsyslogd 949 root  mem    REG      179,2    38608     572344 /lib/libnss_nis-2.10.2.so
rsyslogd 949 root    0u  unix 0xdf8e7a00      0t0       1927 /dev/log
rsyslogd 949 root    3w   REG      179,2    28433     295242 /var/log/auth.log
rsyslogd 949 root    4w   REG      179,2   113068     295485 /var/log/syslog
rsyslogd 949 root    5w   REG      179,2     1878     295467 /var/log/daemon.log
rsyslogd 949 root    6w   REG      179,2   266051     295482 /var/log/kern.log
rsyslogd 949 root    7w   REG      179,2        0     294820 /var/log/lpr.log
rsyslogd 949 root    8w   REG      179,2        0     294931 /var/log/mail.log
rsyslogd 949 root    9w   REG      179,2     1128     295486 /var/log/user.log
rsyslogd 949 root   10w   REG      179,2        0     294968 /var/log/mail.info
rsyslogd 949 root   11w   REG      179,2        0     294975 /var/log/mail.warn
rsyslogd 949 root   12w   REG      179,2        0     295022 /var/log/mail.err
rsyslogd 949 root   13w   REG      179,2        0     310860 /var/log/news/news.crit
rsyslogd 949 root   14w   REG      179,2        0     310861 /var/log/news/news.err
rsyslogd 949 root   15w   REG      179,2        0     310862 /var/log/news/news.notice
rsyslogd 949 root   16w   REG      179,2    21539     295471 /var/log/debug
rsyslogd 949 root   17w   REG      179,2   248846     294396 /var/log/messages
rsyslogd 949 root   18u  FIFO       0,14      0t0       1915 /dev/xconsole
rsyslogd 949 root   19r   REG        0,3        0 4026532154 /proc/kmsg
cron     979 root  cwd    DIR      179,2     4096     294550 /var/spool/cron
cron     979 root  rtd    DIR      179,2     4096          2 /
cron     979 root  txt    REG      179,2    30844     362294 /usr/sbin/cron
cron     979 root  mem    REG      179,2   117596     572347 /lib/ld-2.10.2.so
cron     979 root  mem    REG      179,2    43048     572394 /lib/libpam.so.0.82.2
cron     979 root  mem    REG      179,2    96164     572379 /lib/libselinux.so.1
cron     979 root  mem    REG      179,2  1201860     572339 /lib/libc-2.10.2.so
cron     979 root  mem    REG      179,2     9808     572356 /lib/libdl-2.10.2.so
cron     979 root  mem    REG      179,2    34376     572346 /lib/libcrypt-2.10.2.so
cron     979 root  mem    REG      179,2  1330784     380242 /usr/lib/locale/locale-archive
cron     979 root  mem    REG      179,2    26480     572338 /lib/libnss_compat-2.10.2.so
cron     979 root  mem    REG      179,2    75732     572351 /lib/libnsl-2.10.2.so
cron     979 root  mem    REG      179,2    38608     572344 /lib/libnss_nis-2.10.2.so
cron     979 root  mem    REG      179,2    42680     572345 /lib/libnss_files-2.10.2.so
cron     979 root    0r   CHR        1,3      0t0        729 /dev/null
cron     979 root    1w   CHR        1,3      0t0        729 /dev/null
cron     979 root    2w   CHR        1,3      0t0        729 /dev/null
cron     979 root    3u   REG      179,2        4     278000 /var/run/crond.pid

tardis:~# /etc/init.d/rsyslog restart
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.
tardis:~# /etc/init.d/cron restart
Restarting periodic command scheduler: cron.

tardis:~# lsof -c rsyslog -c cron
COMMAND   PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
rsyslogd 1373 root  cwd    DIR      179,2     4096          2 /
rsyslogd 1373 root  rtd    DIR      179,2     4096          2 /
rsyslogd 1373 root  txt    REG      179,2   264652     362974 /usr/sbin/rsyslogd
rsyslogd 1373 root  mem    REG      179,2   117596     572347 /lib/ld-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    82124     361787 /usr/lib/libz.so.1.2.3.4
rsyslogd 1373 root  mem    REG      179,2   120294     572332 /lib/libpthread-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2     9808     572356 /lib/libdl-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    26628     572350 /lib/librt-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    47204     572375 /lib/libgcc_s.so.1
rsyslogd 1373 root  mem    REG      179,2  1201860     572339 /lib/libc-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    18652     378117 /usr/lib/rsyslog/lmnet.so
rsyslogd 1373 root  mem    REG      179,2    42680     572345 /lib/libnss_files-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2     9136     378124 /usr/lib/rsyslog/imuxsock.so
rsyslogd 1373 root  mem    REG      179,2    19844     378118 /usr/lib/rsyslog/imklog.so
rsyslogd 1373 root  mem    REG      179,2    26480     572338 /lib/libnss_compat-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    75732     572351 /lib/libnsl-2.10.2.so
rsyslogd 1373 root  mem    REG      179,2    38608     572344 /lib/libnss_nis-2.10.2.so
rsyslogd 1373 root    0u  unix 0xdf25a000      0t0       3413 /dev/log
rsyslogd 1373 root    3w   REG       0,16    28237       3180 /var/log/auth.log
rsyslogd 1373 root    4w   REG       0,16   113467       3182 /var/log/syslog
rsyslogd 1373 root    5w   REG       0,16     1878       3183 /var/log/daemon.log
rsyslogd 1373 root    6w   REG       0,16   266068       3184 /var/log/kern.log
rsyslogd 1373 root    7w   REG       0,16        0       3185 /var/log/lpr.log
rsyslogd 1373 root    8w   REG       0,16        0       3186 /var/log/mail.log
rsyslogd 1373 root    9w   REG       0,16     1128       3187 /var/log/user.log
rsyslogd 1373 root   10w   REG       0,16        0       3188 /var/log/mail.info
rsyslogd 1373 root   11w   REG       0,16        0       3189 /var/log/mail.warn
rsyslogd 1373 root   12w   REG       0,16        0       3190 /var/log/mail.err
rsyslogd 1373 root   13w   REG       0,16        0       3192 /var/log/news/news.crit
rsyslogd 1373 root   14w   REG       0,16        0       3193 /var/log/news/news.err
rsyslogd 1373 root   15w   REG       0,16        0       3194 /var/log/news/news.notice
rsyslogd 1373 root   16w   REG       0,16    21478       3195 /var/log/debug
rsyslogd 1373 root   17w   REG       0,16   249059       3196 /var/log/messages
rsyslogd 1373 root   18u  FIFO       0,14      0t0       1915 /dev/xconsole
rsyslogd 1373 root   19r   REG        0,3        0 4026532154 /proc/kmsg
cron     1400 root  cwd    DIR       0,16       60       3257 /var/spool/cron
cron     1400 root  rtd    DIR      179,2     4096          2 /
cron     1400 root  txt    REG      179,2    30844     362294 /usr/sbin/cron
cron     1400 root  mem    REG      179,2   117596     572347 /lib/ld-2.10.2.so
cron     1400 root  mem    REG      179,2    43048     572394 /lib/libpam.so.0.82.2
cron     1400 root  mem    REG      179,2    96164     572379 /lib/libselinux.so.1
cron     1400 root  mem    REG      179,2  1201860     572339 /lib/libc-2.10.2.so
cron     1400 root  mem    REG      179,2     9808     572356 /lib/libdl-2.10.2.so
cron     1400 root  mem    REG      179,2    34376     572346 /lib/libcrypt-2.10.2.so
cron     1400 root  mem    REG      179,2  1330784     380242 /usr/lib/locale/locale-archive
cron     1400 root  mem    REG      179,2    26480     572338 /lib/libnss_compat-2.10.2.so
cron     1400 root  mem    REG      179,2    75732     572351 /lib/libnsl-2.10.2.so
cron     1400 root  mem    REG      179,2    38608     572344 /lib/libnss_nis-2.10.2.so
cron     1400 root  mem    REG      179,2    42680     572345 /lib/libnss_files-2.10.2.so
cron     1400 root    0r   CHR        1,3      0t0        729 /dev/null
cron     1400 root    1w   CHR        1,3      0t0        729 /dev/null
cron     1400 root    2w   CHR        1,3      0t0        729 /dev/null
cron     1400 root    3u   REG       0,16        5       3222 /var/run/crond.pid

this is looking like rsyslog and cron are starting before flashybrid scripts and are still writing to memcard rather than ramdisk. so, boot priority, i guess?

[PlugPBX]

You may be right on boot priority or dependencies...might have to tinker here

Code:

root@PlugPBX:/etc/rc3.d# ls

README       S14nfs-common  S17cron   S17ssh         S18rmnologin
S01bootlogs  S16rsyslog     S17exim4  S18flashybrid  S18stop-bootlogd
S13portmap   S17atd         S17rsync  S18rc.local

[restamp]

Another data point:  I use a generic 16GB SDcard I purchased from MicroCenter.  It has been in continuous use on my Plug since last June, and is still running.  I've had two events in the last month that may signal it is beginning to fail -- Plug essentially dead except for "bad block" messages spewing on the console every couple of seconds -- but power cycling and/or reseating the card has brought it back to life.  I've done nothing special to minimize system writes to the card, except to make my root file system ext2 -- indeed, I even keep my home directory on it -- but I have moved the heavily IO intensive stuff off to a USB hard drive.

Just out of curiosity, how to you all measure your IO to the SDcard?  The best way I've found is to examine /proc/diskstats:

Code:

$ grep mmcblk0p1 /proc/diskstats
 179       1 mmcblk0p1 4574 227 156788 32200 1708 426 41344 1628880 0 131840 1661070

(possibly substituting another partition for mmcblk0p1 if you don't use your card's first partition)

Here's how to interpret the numbers:

Code:

Field 1 -- # of reads issued
Field 2 -- # of reads merged
Field 3 -- # of sectors read
Field 4 -- # of milliseconds spent reading
Field 5 -- # of writes completed
field 6 -- # of writes merged
Field 7 -- # of sectors written
Field 8 -- # of milliseconds spent writing
Field 9 -- # of I/Os currently in progress
Field 10 -- # of milliseconds spent doing I/Os
Field 11 -- weighted # of milliseconds spent doing I/Os

On my systems, I average a write every 45 seconds.  YMMV.

[PlugPBX]

I've had this occur on systems up for less then a month, with lots of slow writes, large write caches, no swap etc. SD card quality varies largely. I also think larger cards might have more 'spares' to write level errors perhaps? There is so much variability in SD cards, I want to focus on a method that ensures reliable long term operation on SD cards of any type/kind and flashybrid for debian SD card based Plugs seems to be idea (once we can master it lol).

Interesting tip on the /proc stuff.

I have the same issue with this system, I reboot and it all comes back up. I removed munin (big disk writer) and have not have issues again for a week now. Flashybrid's scripts are the way to go for sure though....

[mundhra]

i'm just in the initial stages of getting my sheevaplug operational, so quantifying IO isn't even on my radar. i'm thinking (hoping) that using flashybrid will make it a moot issue.

i'm using an 8GB transcend card. i figured i'd rather go with replaceable media as i wouldn't want to exhaust NAND writeability. i'm almost at the point where i'm going to start migrating services over from my main machine, but it's been sitting idle for a week now.

plugpbx: i didn't have time to make a post before leaving for work, but my rc.3 looks much different than yours. multiple S01, S02, and S03 entries. are you using lenny? i've installed squeeze.

coming from gentoo i could set up init dependencies, but it looks like dependencies aren't the default way debian does it. i found a couple links for me to try (one, two). more digging  .

[PlugPBX]

This is a default Squeeze install as well, bare bones system.

I changed the order so that Its invoked first via naming it S01 - all others secondary, still something is holding out somewhere.

Oh if only to find what, and we could nail this so close.... I've gotten PlugPBX now running all variability items including MySQL on a 192 meg ram disk (75 meg free) .... so close to having a root ro filesystem..

[mundhra]

for reference:

Code:

tardis:/etc/rc3.d# ls
README      S01rsyslog  S01sudo  S02dbus   S02rsync  S03flashybrid  S03rmnologin
S01bootlogs  S01samba S02cron  S02exim4  S02ssh    S03rc.local    S03stop-bootlogd

hrm, my /etc/inittab contains

Code:

# The default runlevel.
id:2:initdefault:

i guess debian defaults to runlevel 2 instead of 3?

Code:

tardis:~# runlevel
N 2

my /etc/rc2.d is the same as rc3.d.

i'm thinking about two possible approaches:

1) adding restart cron and rsyslog commands to /etc/rc.local. from that file:

Code:

# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.

2) renaming S03flashybrid to S01flashybrid and changing /etc/init.d/bootlogs to include flashybrid in the following line:

# Required-Start:    hostname $local_fs

the only problem is, i don't know what to call flashybrid here. in /etc/init.d/cron there's the line:

# Required-Start:    $remote_fs $syslog $time

but the /etc/init.d/rsyslog script only refers to itself as rsyslog, so i'm not sure where the variable designation to $syslog is being done.

this is making me feel like a linux newbie all over again.

victory!?

relevant snippet of /etc/rc.local:

Code:

# By default this script does nothing.

/etc/init.d/rsyslog restart
/etc/init.d/cron restart
mountro

exit 0

and bootup:

Code:

INIT: Entering runlevel: 2
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.
Starting MTA:eth0: link up, 100 Mb/s, full duplex, flow control disabled
 exim4.
NET: Registered protocol family 10
Starting OpenBSD Secure Shell server: sshd.
Setting up flashybrid system...done.
mount: / is busy
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.
Restarting periodic command scheduler: cron.

Debian GNU/Linux squeeze/sid tardis ttyS0

tardis login: root
Password:
Last login: Thu Feb 18 19:07:41 EST 2010 on ttyS0
Linux tardis 2.6.32.8 #1 PREEMPT Tue Feb 9 14:36:02 MST 2010 armv5tel

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
tardis:~# mount
/dev/mmcblk0p2 on / type ext2 (ro,noatime,errors=remount-ro)

lsof indicates that relevant files of those services are now pointed to device 0,16.

i've noticed, though, that my bash history isn't always being updated on reboot. if i exit, then login, then reboot, it saves the previous history. i'll have to do more fiddling.

* last login time does seem to be recorded correctly after reboots

* didn't flashybrid stuff say that doing an apt-get install would automatically mountrw, do the install, then mountro? not so, it seems. still, not necessarily a dealbreaker.

[PlugPBX]

It did claim to integrate with apt....

Worse case you'd have to issue a mountrw command,do your apt stuff and then close out with mountro

I think flashybrid has not been maintained in awhile perhaps... Still solid foundation to fixup

[mundhra]

It did claim to integrate with apt....

Worse case you'd have to issue a mountrw command,do your apt stuff and then close out with mountro

I think flashybrid has not been maintained in awhile perhaps... Still solid foundation to fixup

yeah, i don't think that's too inconvenient of a workaround for apt.

the extent of my scripting experience has mostly been perl, but i think i might mess around and see if there are any changes i'd like to make to it.

looking through the flashybrid script, it's looking for a VERBOSE param in /etc/default/flashybrid. i enabled that to see if there's anything else to be concerned about.

Code:

Starting OpenBSD Secure Shell server: sshd.
Syncrhonizing RAM - /etc.
Syncrhonizing RAM - /var/lib/dbus.
Syncrhonizing RAM - /var/lib/exim4.
Syncrhonizing RAM - /var/lib/logrotate.
Syncrhonizing RAM - /var/log.
Syncrhonizing RAM - /var/run.
Syncrhonizing RAM - /root.
Syncrhonizing RAM - /var/spool.
Syncrhonizing RAM - /var/mail.
mount: / is busy
Stopping enhanced syslogd: rsyslogd.
Starting enhanced syslogd: rsyslogd.
Restarting periodic command scheduler: cron.

Debian GNU/Linux squeeze/sid tardis ttyS0

tardis login: root
Password:
Last login: Thu Feb 18 20:28:07 EST 2010 from 192.168.1.222 on pts/0
Linux tardis 2.6.32.8 #1 PREEMPT Tue Feb 9 14:36:02 MST 2010 armv5tel

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
tardis:~# reboot
INIT: S   
Broadcast message from root@tardis (ttyS0) (Thu Feb 18 20:33:33 2010):

The system is going down for reboot NOW!
Stopping MTA: exim4_listener.
Syncing flashybrid system...
Synchronizing /etc.
Synchronizing /var/lib/dbus.
Synchronizing /var/lib/exim4.
Synchronizing /var/lib/logrotate.
Synchronizing /var/log.
Synchronizing /var/run.
Synchronizing /root.
Synchronizing /var/spool.
Synchronizing /var/mail.
done.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds....done.
Stopping enhanced syslogd: rsyslogd.
Saving the system clock.
Deconfiguring network interfaces...done.
Cleaning up ifupdown....
Unmounting temporary filesystems...umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /ram: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
failed.
Deactivating swap...done.
Unmounting local filesystems...done.
Will now restart.
Restarting system.

the presence of

Code:

Unmounting temporary filesystems...umount: /etc: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
umount: /ram: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
failed.

is a little troubling.

i remember reading about aufs (successor of unionfs) somewhere. i found this (1) and this (2) to consider.

[PlugPBX]

I think its worth looking at all the options, flashybrid needs some work and is risky at times. I'll have a look at these two as well...

I would think a large number of Plug geeks would like to see how this plays out, as you've stated, I'd rather replace (and easily backup) SD cards then the NAND on a plug.
 
Had a look at both. flashybrid does closest what i'd like (easy to make changes and apply them hot to a system, but otherwise its readonly mostly)... The other stuff might have to get turned onto its head to work the same way...

[cjm]

Looks very intersting, indeed, although it's the other way round for me as I use a spinning HD for root and just want to use the SD card to reduce spin-ups for log writes. And if my SD card dies, I haven't lost anything I value.

The mount tricks used by flash-hybrid seem to be more or less the same that I use. My fstab looks like this:

Code:

proc            /proc                   proc    defaults        0       0
/dev/sda3       /                       ext3    relatime,errors=remount-ro 0       1
/dev/sda2       none                    swap    sw              0       0
/dev/mmcblk0p1  /boot                   ext2    relatime,noexec,nodev,nosuid        0       2
/dev/mmcblk0p2  /var/log                ext2    relatime,noexec,nodev,nosuid        0       2
/dev/mmcblk0p3  /var/.mmc               ext2    relatime,noexec,nodev,nosuid        0       2

# Overlay some volatile directories in /var with MMC mounts; I'm not using
# links because MMC cards could be removed, faulty due to wear, ... and I
# want the whole thing to fall back to the hard disk during the next reboot
# in case /var/.mmc did not get mounted.
/var/.mmc/milter-greylist       /var/milter-greylist    none    bind    0 0
/var/.mmc/run/sendmail          /var/run/sendmail       none    bind    0 0
/var/.mmc/run/samba             /var/run/samba          none    bind    0 0
/var/.mmc/cache/samba           /var/cache/samba        none    bind    0 0

The trick is to overlay a directory using the "bind" option. Since I'm doing this in fstab, it happens at the beginning of the boot process right after the root filesystem has been mounted read-write and before starting tons of daemons. That might be one of the problems with flash-hybrid mentioned above -- it might come too late and some files might already be open in the underlying root directory. Other than that, if I would mount a tmpfs over /var/log and other directories, the result would be more or less the same, minus copying things back and forth of course.

As you can see, I'm mounting /var/log directly into a partition while having another partition, /var/.mmc, to overlay the remaining directories with frequent write access. Again, I'm doing this the other way round, pushing frequent writes to the SD card to prevent HD spin-ups

BTW, if you want to remount root with different options, add the mount option "remount". Like so:

Code:

mount -o remount,ro /

[mundhra]

thanks for the post, cjm. it's always nice to know someone is doing what you've been considering. i've been thinking about using fstab as a solution because it happens before everything else.

i agree with your assessment of flashybrid regarding startup. i shouldn't have to restart cron and rsyslog in rc.local in order to remount root ro; they should start up logging in the correct place (ramdisk).

i still think flashybrid is useful as far as syncing on shutdown/reboot. i was also considering a daily cron job to sync in case of power loss. log retention isn't necessarily critical for me, but it's nice to have some continuity in order to track down any issues that may pop up.

[PlugPBX]

Indeed good point cjm.

Maybe I'll tinker and throw some mount points from /etc/mtab into my /etc/fstab, disable flashybrid and reboot - see if that does the trick.

Could possibly alter flashybrid to store to fstab as an option, or just roll my own exact needs directly.

Update:

So I got this working with PlugPBX itself. Another warning. I noticed that dhclient somtimes can setup with the timing of PlugPBX and lock root with its lease file too. To be safe using fstab and just dumping what flashybrid from /etc/mtab might be the easiest way to have something reliable and still retain its sync scripts if you wish to use them (just wrappers for rsync really)

Going to keep playing around, I might just purpose build my own scripts and rely on fstab myself long term. Still a neat trick and great way to go more embedded with the SheevaPlug and yet flip back to read/write at teh flick of a switch when needed.

I posted the thread here if anyone is curious on the details of this dirty dirty 'hack test' hehe

http://forums.plugpbx.org/index.php?topic=32.0

[PlugPBX]

On a debian System do a 'man rcS'

Have a lookie see at these entries in man page

       RAMRUN Make /var/run/ available as a ram file system (tmpfs).  Will also disable cleaning of /var/run/  during  boot.   Set  to  'yes'  to
              enable,  to 'no' to disable.  The size of the tmpfs can be controlled using TMPFS_SIZE and RUN_SIZE in /etc/default/tmpfs.  Because
              of this, packages can not expect directories in /var/run to exist after boot.  Packages expecting this are buggy  and  need  to  be
              fixed.

       RAMLOCK
              Make  /var/lock/  available  as  a ram file system (tmpfs).  Will also disable cleaning of /var/lock/ during boot.  Set to 'yes' to
              enable, to 'no' to disable.  The size of the tmpfs can be controlled using TMPFS_SIZE and LOCK_SIZE in /etc/default/tmpfs.  Because
              of  this,  packages  can not expect directories in /var/lock to exist after boot.  Packages expecting this are buggy and need to be
              fixed.

Hmm that might do it

If not one could perhaps run flashybrid via /etc/init.d/rcS script (prior to the rcd.x scripts)

Will have to try this!  This might be what gets flashybrid working issue free

[mundhra]

the default daily cronjob involving dpkg is trying to write to /var/backups, so i added this to my ramstore.

i found this link for setting up tmpfs fstab entries similar to how cjm has them.

since flashybrid creates tmpfs mounts and does the syncing, the options we have :

1) stay with flashybrid and use the rc.local restart workaround
2) put tmpfs entries in fstab and a) comment out flashybrid parts that create the tmpfs mounts or b) write your own scripts to sync (i think the link above actually has a script that does this)