Last time I introduced pivotroot scripts for OpenWRT. This time I’ll discuss my improvements.
The standard OpenWRT pivotroot is pretty basic. It needs to know where your USB disk will show up, and it assumes that the filesystem is in good shape. But I have OpenWRT systems with multiple USB ports, plus I’d like to be able to use USB hubs. And my OpenWRT systems tend to run until the power goes off – so I want to make sure the filesystem is clean before I boot off it.
In order to make things safer, I choose to run an ext3 filesystem. This is harder on the flash, because of the journal writes, but safer recovery for unclean shutdowns – which are the norm. A corrupted filesystem only gets worse, though, so I want to run fsck before I use the filesystem. (That means you’ll need to install e2fsprogs in the flash-based root filesystem.)
Since my USB sticks can move around, I search through all the available disk partitions to find one that looks like an OpenWRT system. That means I may try to mount filesystems and have them fail, but this is pretty safe. Eventually I’ll find the right filesystem, and if not I’ll boot off flash. I do assume that the filesystem will be on partition 1, and that it will be an ext3 filesystem.
I hate that the standard pivotroot uses the /mnt mountpoint. I use /flash instead. When I’m looking for an OpenWRT filesystem I look for /flash (and also /sbin/init). (So I need to mkdir /flash on both the flash-based and USB root filesystems.)
Finally, I want to know what’s going on – especially with fsck – and a way to turn off the pivotroot easily. I put logs in /tmp/pivotroot.log, I make the power LED flash while I’m fsck’ing and searching for a root fs, and I have rcS check whether the pivotroot script is executable. A quick chmod -x /flash/etc/init.d/pivotroot, and I can boot off flash.
Enough yakking, here’s the code. /etc/init.d/pivotroot, first, but there’s more below:
#!/bin/sh
# From http://oldwiki.openwrt.org/UsbStorageHowto.html - DHK 7/14/09
#
# Greatly enhanced to select among (maybe) several USB Mass Storage devices
# Assumes that the root fs will be on an ext3 filesystem in partition 1,
# and looks for /flash and /jffs in the root directory. Also assumes that
# mounting and checking each candidate device is harmless. Runs e2fsck
# over partitions before mounting them; note that this causes spurious
# date-related complaints every time, since time has not been synchronized
# yet at the time pivotroot is run.
#
# Output from this script is stashed at /tmp/pivotroot.log.
#
# DHK 7/23/2009
# Set Power LED flashing while we look for OpenWRT root
if [ -f /etc/diag.sh ]; then
LED=1
. /etc/diag.sh
POW=`cat /proc/diag/led/power`
set_led power f
fi
# install needed modules for usb and the ext3 filesystem
# We could defer loading the filesystem modules until we know there's a useful partition.
# **NOTE** for usb2.0 replace "uhci" with "ehci_hcd"
# **NOTE** for ohci chipsets replace "uhci" with "usb-ohci"
# **NOTE** for WL-500gP usb-uhci not usb-ohci
echo -n "pivotroot loading kernel modules: "
for module in usbcore usb-uhci scsi_mod sd_mod usb-storage ext2 jbd ext3 ; do {
echo -n "$module "
insmod $module
}; done
echo
# this may need to be higher if your disk is slow to initialize
sleep 4s
PART1=`find /dev/scsi -name part1`
if [ -z $PART1 ]; then
echo No partitions, skipping pivotroot
else
# Look for a mountable USB stick with, maybe, OpenWRT on it
for dev in `find /dev/scsi -name part1`; do
echo pivotroot checking $dev
e2fsck -p $dev
fsck=$?
echo fsck status is $fsck
if [ $fsck -eq 2 ]; then
echo Corrected errors on $dev, need to reboot
sleep 5
reboot
elif [ $fsck -eq 1 ]; then
echo Corrected errors on $dev
elif [ $fsck -gt 2 ]; then
echo No usable filesystem on $dev
continue
fi
echo Mounting $dev
mount -t ext3 $dev /flash
mt=$?
if [ $mt -eq 0 ]; then
if [ -x /flash/sbin/init -a -d /flash/jffs -a -d /flash/flash ]; then
echo Found OpenWRT root on $dev
# Side-effect - leave /flash mounted
break;
else
echo "Missing /sbin/init, /jffs, or /flash (mount status was $mt)"
[ -x /flash/sbin/init ] || echo Failed -x /sbin/init
[ -d /flash/flash ] || echo Failed -x /flash
[ -d /flash/jffs ] || echo Failed -x /jffs
umount $dev
fi
else
echo mount status is $mt
fi
done
# if everything looks ok, do the pivot root
[ -x /flash/sbin/init ] && {
mount -o move /proc /flash/proc && \
pivot_root /flash /flash/flash && {
mount -o move /flash/dev /dev
mount -o move /flash/tmp /tmp
mount -o move /flash/jffs2 /jffs2 2>&-
mount -o move /flash/sys /sys 2>&-
}
}
fi
# Restore Power LED to previous state
[ $LED -eq 1 ] && set_led power $POW
Addition to /etc/init.d/rcS (put this before the line LOGGER=”cat”):
# Improved from http://oldwiki.openwrt.org/UsbStorageHowto.html
# Switch the root filesystem to USB, if present
# DHK 7/14/09
if [ $2 == "boot" -a -x /etc/init.d/pivotroot ] ; then
/etc/init.d/pivotroot > /tmp/pivotroot.log
fi
Finally, I add this line to the bottom of /etc/banner, but on the flash filesystem only. When I ssh in to an OpenWRT box I’ll know immediately how it booted:
Booted off internal flash
Remember, this works for me, but I make no guarantees it will work for you. Make sure you’ve saved your configuration before fooling with this stuff, and that you know how to reflash your router if you brick it.
hello, dhkaufman , I use your pivotroot script on my openwrt, it is wonderful. After the system boot from my sd card with a usb adapter, I install openvpn and miniupnpd, and I make init file for both of them in /etc/init.d/ and link file in /etc/rcS/ (all of them were in USB root filesystem), however, after a root, openvpn and miniupnd could auto start, then I install curl, it is the same.
I want to look your help, is it possible to auto start the software in USB root filesystem?
dhkaufman, I have already solved the problem, just add several commands after
# if everything looks ok, do the pivot root
[ -x /flash/sbin/init ] &}
fi
#add some commands
if [ -x /etc/init.d ]
then
/etc/init.d/miniupnpd start
/etc/init.d/openvpn start
fi
# Restore Power LED to previous state
[ $LED -eq 1 ] && set_led power $POW
thanks a lot!
@happybit, this is not quite right. You should not need to edit /etc/init.d/rcS for each package you add. Instead, you should create links in /etc/rc.d – but you should use the tools provided to do it. To create the links, run
/etc/init.d/openvpn enable
and similarly for miniupnpd. (Or you can use the “Enable Service” link on the System/Services page in the x-wrt Web UI, and I’m sure there’s something similar in the standard Web UI.) This creates a link for starting up the service, and one for shutting it down, and importantly it makes sure things happen in the right order.
All of this happens in the USB root filesystem, of course.
Enjoy!
for kaufman,
Sorry for my mistake in the first post?”/etc/rcS/” should be “/etc/rc.d” , that is to say:
I have already run
/etc/init.d/openvpn enable
/etc/init.d/miniupnpd enable
and check that links in /etc/rc.d are also created.
however, both of openvpn and miniupnpd couldn’t auto-start after reboot.
Then, I try to add the command below at the end of your /etc/init.d/pivotroot script manually.
#add some commands
if [ -x /etc/init.d ]
then
/etc/init.d/miniupnpd start
/etc/init.d/openvpn start
fi
then they react as what I think.
So, I wandered why the links in /etc/rc.d in the USB root filesystem are useless.
@happybit, I am glad that the links are really in /etc/rc.d.
/etc/rc.d is processed by calls to the run_scripts function in /etc/init.d/rcS, just after pivotroot is called. I’m just guessing here, but if you ran pivotroot at the end of rcS instead of in the middle, then /etc/rc.d would get processed from the flash root filesystem instead of the USB root filesystem and you would see the symptoms you describe.
I see that my original posting is ambiguous about this. The call to pivotroot should go in /etc/rcS before the line:
LOGGER=”cat”
This is probably clearer in the UsbStorageHowTo on the OpenWRT wiki.
dear kaufman ? thanks for your help,
I find that I ran pivotroot at the end of rcS instead of in the middle indeed, I will try to put it in /etc/rcS before the line:
LOGGER=”cat”
by the way, in your article Persistent /var for OpenWRT, Should I fix etc/init.d/syslog and etc/init.d/boot on both flash and USB root filesystem?
hi, kaufman
I have tried to put it in /etc/rcS before the line:
LOGGER=”cat”
there isn’t any affection, openvpn and miniupnpd can’t start up.
I also tried Persistent /var for OpenWRT, to fix etc/init.d/boot on flash root filesystem is ok, thanks!
@happybit, this still doesn’t sound right. Did you change the /etc/init.d/pivotroot on the flash filesystem? (pivotroot on the USB filesystem doesn’t do anything, it’s never called.)
As far as the persistent /var changes, if your system is working correctly, you would change /etc/init.d/syslog and boot on the USB filesystem. Since they’re in /etc/init.d, they get called via /etc/rc.d, and so they reside in the USB filesystem.
kaufman, it’s actually done on the flash filesystem, not on the usb filesystem, but openvpn and miniupnpd can’t start up either. I don’t why????
In my system, there is no /etc/init.d/syslog file, just /etc/init.d/boot file.
Awesome stuff. Though I was hoping you might have a solution to the problem of keeping /etc/config/system and /[flash|mnt]/config/system in sync. For example, if I change the hostname using LUCI that is saved in /etc/config/system. But upon reboot, the internal system file is used and it stays with the old hostname.
There is one howto (http://oldwiki.openwrt.org/OpenWrtDocs(2f)KamikazeConfiguration(2f)BootFromExternalMediaHowTo.html) which mentions doing a “bind mount” of /etc to avoid this . Their method is totally different from yours and the traditional OpenWRT method it is based on.
How to you avoid this problem?
Thanks
@nathane, that’s an interesting question, because I prefer that /etc on flash not be in sync with /etc on the USB drive. I wrote a blog post explaining why.
I have a problem, openwrt simply isn’t booting from flash, I think that something is wrong with usb flash drive but simply don’t know what.
My pivotroot.log
pivotroot loading kernel modules: usbcore usb-ohci scsi_mod sd_mod usb-storage e
pivotroot checking /dev/scsi/host0/bus0/target0/lun0/part1
/dev/scsi/host0/bus0/target0/lun0/part1: clean, 414/115200 files, 17189/459852 b
fsck status is 0
Mounting /dev/scsi/host0/bus0/target0/lun0/part1
mount status is 255
I forgot to mention something, pivotroot script is working fine after booting openwrt normally and then plunging usb stick and firing the script
@ScApi, if the pivotroot script works after you booted then the usual reason is you’re preloading the wrong kernel modules (foreach loop with insmod in it). The right kernel modules get loaded when the boot completes, and then when you try pivotroot again it magically works.
Another possibility is that your flash drive is an ext2 not an ext3 (since e2fsck works it out for itself, but the mount line is hardcoded to ext3).
Another possibility is that the /flash directory is missing. But these latter two possibilities don’t explain why the pivotroot would work after booting. For that, kernel modules are the most likely explanation.
I’ve switched to ext2, formatted usb stick to ext2, and finaly boot works, I think it has be something in image creation, sine 520GC has only 2mb flash, I removed almost everything from flash to add required modules, I’l try to change to ext3 in future but now another question, can i simply copy contents of image builder temp dir to usb stick (dir with I think all flash files) ? or it has to be copied from working flash ?
@ScApi, you can build the USB image however you want, you just need to be prepared for it not to work
. Since you can just pull the USB stick and power-cycle, it seems like the experiment is pretty low-risk. Let us know how it works out.
The space requirements (on flash) for ext3 are nearly the same as for ext2, but ext2 is a completely valid choice. The upside of ext3 is the journaling, which gives better recovery from an unclean shutdown. The downside is that the frequent journal-writing will kill your USB stick faster. I choose safety, and anticipate replacing USB sticks.
As I and You expected it didn’t worked
, this had something to do with firstboot script, but I didn’t digg deeper.
Something about the web management pakages.
X-Wrt is working except installation of packages via WebGui.
Gargoyle is working except graphs (their missing, code is pointing to empty tmp dir)
OpenWrt web – not yet tested, will try tommorow,
And something that I’v added to the script, include swap utils, and makeswap + swapon, it really speeds up my device.
I have seen and checked your script and is waiting for your swap option in the script to test if it works.
I have been trying to use my own but failed each time.