Ever on and on

A friend runs two separate subnets at home: one is for home computing/entertainment, and the other is work-related. It’s important to keep the entertainment users off the work LAN. We recently replaced his aging firewall with an ASUS WL-520gU running OpenWRT, so I had to work out how to create separate LANs under OpenWRT.

We picked the the 520gU because it has 1 WAN port, 4 LAN ports, and a USB port, and becuase it runs OpenWRT well. We set up the root filesystem on an external USB stick (see previous posts). Then we had to work out how to have two separate networks on the LAN side.

OpenWRT sets up networks by tying together ports into VLANs (see fuller discussion on the OpenWRT site). Each of these groups of ports becomes a separate sub-interface of the master ethernet interface. For example, the standard configuration on a 520gU has (as I said) 1 WAN port and 4 LAN ports. But all of those ports are connected to the same switch: the only difference between WAN and LAN is that they’re on different VLANs, and some labelling onthe outside of the box. The master ethernet interface is eth0; the standard OpenWRT LAN is eth0.0, and WAN is eth0.1.

All of this is set up on the OpenWRT Network / Interfaces configuration page, which manipulates the file /etc/config/network. I’ll show you the examples in /etc/config/network, partly because it’s easier to have text examples and partly because there are 2 admin UIs for OpernWRT (LUCI, which comes with OpenWRT, and X-WRT, a parallel project that I use). Here’s the default setup:

config 'switch' 'eth0'
        option 'vlan0' '1 2 3 4 5*'
        option 'vlan1' '0 5'

vlan0, as I said, is the standard LAN, and vlan1 is WAN. But wait, I said there were 5 ports (1 WAN and 4 LAN) but the ports here are numbered 0-5. What’s the 6th port? Well, it’s the internal switch connected to the CPU, and it’s included in every group of ports. If it weren’t then OpenWRT couldn’t see the packets.

Here’s the configuration with two different LANs:

config 'switch' 'eth0'
        option 'vlan0' '1 2 5*'
        option 'vlan1' '0 5'
        option 'vlan2' '3 4 5'

vlan1 is still the WAN port, but now vlan0 is one LAN network, and vlan2 is the other. Next time I’ll show how we did the firewall rules to complete the job. But before we tackle the firewall rules, I need to set up the WiFi.

WiFi is different, and isn’t on the 6-port ethernet switch. Instead, WiFi works with its own master interface, and sub-interfaces for each WiFi network that you’re either a client on, or an access point for. Then you can bridge the WiFi subinterface to a LAN subinterface, to create a network with both WiFi and LAN access types.

The standard OpenWRT configuration bridges the LAN to wl0, the default and first WiFi subinterface. We were able to use this same configuration to bridge wl0 to the first LAN, eth0.0.

In other posts I talked about booting my OpenWRT routers off an external USB stick. In this post I’ll describe how to make the /var directory tree persistent across reboots.

On a standard OpenWRT system, /var is a link to /tmp. This simplifies setup, but it means that /var gets wiped out on each reboot, because /tmp is a tmpfs (in-memory). Why do I care? Well, when you reboot you lose:

1. The package lists downloaded by opkg update
2. DHCP leases that OpenWRT handed out
3. Web pages that polipo, a caching proxy, has already loaded
4. System-specific Bluetooth configuration, especially paired devices

And I care about all of these. Also, the tmpfs is kind of small, which makes web caching harder.

Unfortunately, the assumption that /var is /tmp has leaked into a number of places, so you’ll need to fix those. Start with the USB Storage howto to setup your external filesystem, and with my pivotroot scripts as well. Make these changes when you’re booted off internal flash – don’t make them to your live filesystem.

I’ll assume your external USB stick is mounted on /mnt.

1. Undo the link. cd /mnt; rm var; mkdir -p var/etc
2. Fix the bootup scripts. Exceprts are below, but you need to fix references in etc/init.d/syslog and etc/init.d/boot on your USB stick

diffs for etc/init.d/boot:

--- /etc/init.d/boot    Sun May 10 21:09:45 2009
+++ boot        Sat Jan  1 00:08:59 2000
@@ -38,10 +38,14 @@
        config_load system
        config_foreach system_config system

-       mkdir -p /var/run
-       mkdir -p /var/log
-       mkdir -p /var/lock
-       mkdir -p /var/state
+       mkdir -p /tmp/run
+       [ ! -f /var/run ]   && ln -s /tmp/run   /var
+       mkdir -p /tmp/log
+       [ ! -f /var/log ]   && ln -s /tmp/log   /var
+       mkdir -p /tmp/lock
+       [ ! -f /var/lock ]  && ln -s /tmp/lock  /var
+       mkdir -p /tmp/state
+       [ ! -f /var/state ] && ln -s /tmp/state /var
        mkdir -p /tmp/.uci
        chmod 0700 /tmp/.uci
        touch /var/log/wtmp

diffs for etc/init.d/syslog:

--- /etc/init.d/syslog  Fri Jun 12 08:16:26 2009
+++ syslog      Sat Jan  1 00:09:22 2000
@@ -11,7 +11,7 @@
        local cfg="$1"
        local type file size ipaddr port IPCALC_CMD SYSLOG_CMD
        local DEFAULT_type="circular"
-       local DEFAULT_file="/var/log/messages"
+       local DEFAULT_file="/tmp/log/messages"
        local DEFAULT_size=16
        local DEFAULT_ipaddr=""
        local DEFAULT_port=514

Last time I introduced pivotroot scripts for OpenWRT. This time I’ll discuss my improvements.

The standard OpenWRT pivotroot is pretty basic. It needs to know where your USB disk will show up, and it assumes that the filesystem is in good shape. But I have OpenWRT systems with multiple USB ports, plus I’d like to be able to use USB hubs. And my OpenWRT systems tend to run until the power goes off – so I want to make sure the filesystem is clean before I boot off it.

In order to make things safer, I choose to run an ext3 filesystem. This is harder on the flash, because of the journal writes, but safer recovery for unclean shutdowns – which are the norm. A corrupted filesystem only gets worse, though, so I want to run fsck before I use the filesystem. (That means you’ll need to install e2fsprogs in the flash-based root filesystem.)

Since my USB sticks can move around, I search through all the available disk partitions to find one that looks like an OpenWRT system. That means I may try to mount filesystems and have them fail, but this is pretty safe. Eventually I’ll find the right filesystem, and if not I’ll boot off flash. I do assume that the filesystem will be on partition 1, and that it will be an ext3 filesystem.

I hate that the standard pivotroot uses the /mnt mountpoint. I use /flash instead. When I’m looking for an OpenWRT filesystem I look for /flash (and also /sbin/init). (So I need to mkdir /flash on both the flash-based and USB root filesystems.)

Finally, I want to know what’s going on – especially with fsck – and a way to turn off the pivotroot easily. I put logs in /tmp/pivotroot.log, I make the power LED flash while I’m fsck’ing and searching for a root fs, and I have rcS check whether the pivotroot script is executable. A quick chmod -x /flash/etc/init.d/pivotroot, and I can boot off flash.

Enough yakking, here’s the code. /etc/init.d/pivotroot, first, but there’s more below:

#!/bin/sh
# From http://oldwiki.openwrt.org/UsbStorageHowto.html - DHK 7/14/09
#
# Greatly enhanced to select among (maybe) several USB Mass Storage devices
# Assumes that the root fs will be on an ext3 filesystem in partition 1,
# and looks for /flash and /jffs in the root directory. Also assumes that
# mounting and checking each candidate device is harmless. Runs e2fsck
# over partitions before mounting them; note that this causes spurious
# date-related complaints every time, since time has not been synchronized
# yet at the time pivotroot is run.
#
# Output from this script is stashed at /tmp/pivotroot.log.
#
# DHK 7/23/2009

# Set Power LED flashing while we look for OpenWRT root
if [ -f /etc/diag.sh ]; then
  LED=1
  . /etc/diag.sh
  POW=`cat /proc/diag/led/power`
  set_led power f
fi

# install needed modules for usb and the ext3 filesystem
# We could defer loading the filesystem modules until we know there's a useful partition.
# **NOTE** for usb2.0 replace "uhci" with "ehci_hcd"
# **NOTE** for ohci chipsets replace "uhci" with "usb-ohci"
# **NOTE** for WL-500gP usb-uhci not usb-ohci
echo -n "pivotroot loading kernel modules: "
for module in usbcore usb-uhci scsi_mod sd_mod usb-storage ext2 jbd ext3 ; do {
  echo -n "$module "
  insmod $module
}; done
echo
# this may need to be higher if your disk is slow to initialize
sleep 4s

PART1=`find /dev/scsi -name part1`

if [ -z $PART1 ]; then
  echo No partitions, skipping pivotroot
else
  # Look for a mountable USB stick with, maybe, OpenWRT on it
  for dev in `find /dev/scsi -name part1`; do
    echo pivotroot checking $dev
    e2fsck -p $dev
    fsck=$?
    echo fsck status is $fsck
    if [ $fsck -eq 2 ]; then
      echo Corrected errors on $dev, need to reboot
      sleep 5
      reboot
    elif [ $fsck -eq 1 ]; then
      echo Corrected errors on $dev
    elif [ $fsck -gt 2 ]; then
      echo No usable filesystem on $dev
      continue
    fi
    echo Mounting $dev
    mount -t ext3 $dev /flash
    mt=$?
    if [ $mt -eq 0 ]; then
      if [ -x /flash/sbin/init -a -d /flash/jffs -a -d /flash/flash ]; then
        echo Found OpenWRT root on $dev
        # Side-effect - leave /flash mounted
        break;
      else
        echo "Missing /sbin/init, /jffs, or /flash (mount status was $mt)"
        [ -x /flash/sbin/init ] || echo Failed -x /sbin/init
        [ -d /flash/flash ]     || echo Failed -x /flash
        [ -d /flash/jffs ]      || echo Failed -x /jffs
        umount $dev
      fi
    else
      echo mount status is $mt
    fi
  done

  # if everything looks ok, do the pivot root
  [ -x /flash/sbin/init ] && {
          mount -o move /proc /flash/proc && \
          pivot_root /flash /flash/flash && {
                  mount -o move /flash/dev /dev
                  mount -o move /flash/tmp /tmp
                  mount -o move /flash/jffs2 /jffs2 2>&-
                  mount -o move /flash/sys /sys 2>&-
          }
  }
fi

# Restore Power LED to previous state
[ $LED -eq 1 ] && set_led power $POW

Addition to /etc/init.d/rcS (put this before the line LOGGER=”cat”):

# Improved from http://oldwiki.openwrt.org/UsbStorageHowto.html
# Switch the root filesystem to USB, if present
# DHK 7/14/09
if [ $2 == "boot" -a -x /etc/init.d/pivotroot ] ; then
        /etc/init.d/pivotroot > /tmp/pivotroot.log
fi

Finally, I add this line to the bottom of /etc/banner, but on the flash filesystem only. When I ssh in to an OpenWRT box I’ll know immediately how it booted:

Booted off internal flash

Remember, this works for me, but I make no guarantees it will work for you. Make sure you’ve saved your configuration before fooling with this stuff, and that you know how to reflash your router if you brick it.