Ever on and on

In other posts I talked about booting my OpenWRT routers off an external USB stick. In this post I’ll describe how to make the /var directory tree persistent across reboots.

On a standard OpenWRT system, /var is a link to /tmp. This simplifies setup, but it means that /var gets wiped out on each reboot, because /tmp is a tmpfs (in-memory). Why do I care? Well, when you reboot you lose:

1. The package lists downloaded by opkg update
2. DHCP leases that OpenWRT handed out
3. Web pages that polipo, a caching proxy, has already loaded
4. System-specific Bluetooth configuration, especially paired devices

And I care about all of these. Also, the tmpfs is kind of small, which makes web caching harder.

Unfortunately, the assumption that /var is /tmp has leaked into a number of places, so you’ll need to fix those. Start with the USB Storage howto to setup your external filesystem, and with my pivotroot scripts as well. Make these changes when you’re booted off internal flash – don’t make them to your live filesystem.

I’ll assume your external USB stick is mounted on /mnt.

1. Undo the link. cd /mnt; rm var; mkdir -p var/etc
2. Fix the bootup scripts. Exceprts are below, but you need to fix references in etc/init.d/syslog and etc/init.d/boot on your USB stick

diffs for etc/init.d/boot:

--- /etc/init.d/boot    Sun May 10 21:09:45 2009
+++ boot        Sat Jan  1 00:08:59 2000
@@ -38,10 +38,14 @@
        config_load system
        config_foreach system_config system

-       mkdir -p /var/run
-       mkdir -p /var/log
-       mkdir -p /var/lock
-       mkdir -p /var/state
+       mkdir -p /tmp/run
+       [ ! -f /var/run ]   && ln -s /tmp/run   /var
+       mkdir -p /tmp/log
+       [ ! -f /var/log ]   && ln -s /tmp/log   /var
+       mkdir -p /tmp/lock
+       [ ! -f /var/lock ]  && ln -s /tmp/lock  /var
+       mkdir -p /tmp/state
+       [ ! -f /var/state ] && ln -s /tmp/state /var
        mkdir -p /tmp/.uci
        chmod 0700 /tmp/.uci
        touch /var/log/wtmp

diffs for etc/init.d/syslog:

--- /etc/init.d/syslog  Fri Jun 12 08:16:26 2009
+++ syslog      Sat Jan  1 00:09:22 2000
@@ -11,7 +11,7 @@
        local cfg="$1"
        local type file size ipaddr port IPCALC_CMD SYSLOG_CMD
        local DEFAULT_type="circular"
-       local DEFAULT_file="/var/log/messages"
+       local DEFAULT_file="/tmp/log/messages"
        local DEFAULT_size=16
        local DEFAULT_ipaddr=""
        local DEFAULT_port=514

Last time I introduced pivotroot scripts for OpenWRT. This time I’ll discuss my improvements.

The standard OpenWRT pivotroot is pretty basic. It needs to know where your USB disk will show up, and it assumes that the filesystem is in good shape. But I have OpenWRT systems with multiple USB ports, plus I’d like to be able to use USB hubs. And my OpenWRT systems tend to run until the power goes off – so I want to make sure the filesystem is clean before I boot off it.

In order to make things safer, I choose to run an ext3 filesystem. This is harder on the flash, because of the journal writes, but safer recovery for unclean shutdowns – which are the norm. A corrupted filesystem only gets worse, though, so I want to run fsck before I use the filesystem. (That means you’ll need to install e2fsprogs in the flash-based root filesystem.)

Since my USB sticks can move around, I search through all the available disk partitions to find one that looks like an OpenWRT system. That means I may try to mount filesystems and have them fail, but this is pretty safe. Eventually I’ll find the right filesystem, and if not I’ll boot off flash. I do assume that the filesystem will be on partition 1, and that it will be an ext3 filesystem.

I hate that the standard pivotroot uses the /mnt mountpoint. I use /flash instead. When I’m looking for an OpenWRT filesystem I look for /flash (and also /sbin/init). (So I need to mkdir /flash on both the flash-based and USB root filesystems.)

Finally, I want to know what’s going on – especially with fsck – and a way to turn off the pivotroot easily. I put logs in /tmp/pivotroot.log, I make the power LED flash while I’m fsck’ing and searching for a root fs, and I have rcS check whether the pivotroot script is executable. A quick chmod -x /flash/etc/init.d/pivotroot, and I can boot off flash.

Enough yakking, here’s the code. /etc/init.d/pivotroot, first, but there’s more below:

#!/bin/sh
# From http://oldwiki.openwrt.org/UsbStorageHowto.html - DHK 7/14/09
#
# Greatly enhanced to select among (maybe) several USB Mass Storage devices
# Assumes that the root fs will be on an ext3 filesystem in partition 1,
# and looks for /flash and /jffs in the root directory. Also assumes that
# mounting and checking each candidate device is harmless. Runs e2fsck
# over partitions before mounting them; note that this causes spurious
# date-related complaints every time, since time has not been synchronized
# yet at the time pivotroot is run.
#
# Output from this script is stashed at /tmp/pivotroot.log.
#
# DHK 7/23/2009

# Set Power LED flashing while we look for OpenWRT root
if [ -f /etc/diag.sh ]; then
  LED=1
  . /etc/diag.sh
  POW=`cat /proc/diag/led/power`
  set_led power f
fi

# install needed modules for usb and the ext3 filesystem
# We could defer loading the filesystem modules until we know there's a useful partition.
# **NOTE** for usb2.0 replace "uhci" with "ehci_hcd"
# **NOTE** for ohci chipsets replace "uhci" with "usb-ohci"
# **NOTE** for WL-500gP usb-uhci not usb-ohci
echo -n "pivotroot loading kernel modules: "
for module in usbcore usb-uhci scsi_mod sd_mod usb-storage ext2 jbd ext3 ; do {
  echo -n "$module "
  insmod $module
}; done
echo
# this may need to be higher if your disk is slow to initialize
sleep 4s

PART1=`find /dev/scsi -name part1`

if [ -z $PART1 ]; then
  echo No partitions, skipping pivotroot
else
  # Look for a mountable USB stick with, maybe, OpenWRT on it
  for dev in `find /dev/scsi -name part1`; do
    echo pivotroot checking $dev
    e2fsck -p $dev
    fsck=$?
    echo fsck status is $fsck
    if [ $fsck -eq 2 ]; then
      echo Corrected errors on $dev, need to reboot
      sleep 5
      reboot
    elif [ $fsck -eq 1 ]; then
      echo Corrected errors on $dev
    elif [ $fsck -gt 2 ]; then
      echo No usable filesystem on $dev
      continue
    fi
    echo Mounting $dev
    mount -t ext3 $dev /flash
    mt=$?
    if [ $mt -eq 0 ]; then
      if [ -x /flash/sbin/init -a -d /flash/jffs -a -d /flash/flash ]; then
        echo Found OpenWRT root on $dev
        # Side-effect - leave /flash mounted
        break;
      else
        echo "Missing /sbin/init, /jffs, or /flash (mount status was $mt)"
        [ -x /flash/sbin/init ] || echo Failed -x /sbin/init
        [ -d /flash/flash ]     || echo Failed -x /flash
        [ -d /flash/jffs ]      || echo Failed -x /jffs
        umount $dev
      fi
    else
      echo mount status is $mt
    fi
  done

  # if everything looks ok, do the pivot root
  [ -x /flash/sbin/init ] && {
          mount -o move /proc /flash/proc && \
          pivot_root /flash /flash/flash && {
                  mount -o move /flash/dev /dev
                  mount -o move /flash/tmp /tmp
                  mount -o move /flash/jffs2 /jffs2 2>&-
                  mount -o move /flash/sys /sys 2>&-
          }
  }
fi

# Restore Power LED to previous state
[ $LED -eq 1 ] && set_led power $POW

Addition to /etc/init.d/rcS (put this before the line LOGGER=”cat”):

# Improved from http://oldwiki.openwrt.org/UsbStorageHowto.html
# Switch the root filesystem to USB, if present
# DHK 7/14/09
if [ $2 == "boot" -a -x /etc/init.d/pivotroot ] ; then
        /etc/init.d/pivotroot > /tmp/pivotroot.log
fi

Finally, I add this line to the bottom of /etc/banner, but on the flash filesystem only. When I ssh in to an OpenWRT box I’ll know immediately how it booted:

Booted off internal flash

Remember, this works for me, but I make no guarantees it will work for you. Make sure you’ve saved your configuration before fooling with this stuff, and that you know how to reflash your router if you brick it.

I use OpenWRT for my wireless access point, and for some other projects. I often set up an OpenWRT box running off an external USB stick, instead of its internal flash. This gives me more storage space – for software packages but also pictures when I’m hosting a web site, etc. Another advantage of an external USB stick is that it’s cheap to replace when you burn out the flash, although I’ve never actually had this happen to me.

So how do you boot OpenWRT off an external USB stick? Well, midway through the boot process you change the root filesystem from the one on the internal flash to the one on the external stick. That means you need to check first that there’s a filesystem to boot off of. There’s a standard procedure on the OpenWRT wiki, using what’s called a pivotroot script.

In White Russian (the older OpenWRT release) the procedure was to replace /sbin/init with pivotroot. This is scary, because if you step wrong you’ve bricked your router – made it unbootable. In Kamikaze (the newer release), the recommendation has changed, and instead pivotroot is explicitly called from rcS, a very early bootup script. This is a better idea, and when I was done with my changes it was even safer.

What does a pivotroot script need to do? It needs to:

1. Make sure the kernel can recognize your USB stick, by loading kernel modules to support USB and USB storage devices
2. Make sure the kernel can recognize the filesystem on your USB stick, by loading more kernel modules. For example, I use the ext3 filesystem on my USB stick, and that’s not built in to the OpenWRT kernels
3. Check whether your USB stick is even present
4. Mount the filesystem on your USB stick
5. Finally, if everything else is OK, it remounts all the filesystems so that your USB stick is the root filesystem and the flash-based root filesystems are mounted elsewhere

Note that this procedure still uses the flash-based root filesystem to get partway booted. Any special software packages you need to boot up – all those kernel modules, for example – and the pivotroot script itself need to be on the flash-based root filesystem, and not the root filesystem you see once you’re booted. This also makes debugging a pivotroot script tricky, because by the time OpenWRT is booted the environment is very different from the one pivotroot sees.

Next time I’ll outline the changes I made to improve the standard OpenWRT pivotroot.