Ever on and on

In a comment on my earlier pivotroot post, nathane says:

I was hoping you might have a solution to the problem of keeping /etc/config/system and /[flash|mnt]/config/system in sync.

and mentions another howto that does a bind mount of /etc between the flash and USB filesystems, so they stay in sync.

There’s an obvious problem with a bind mount of /etc, which is that the contents of /etc/config depend on the software in the rest of the filesystem. Let’s say you’re booted to the USB root filesystem. If you install a new package, or a new version of a current package, then you might inadvertently create a config file that’s incompatible with the software on the flash root filesystem. Most likely you won’t discover the incompatibility until it causes a significant problem. (Imagine if the webif or ssh doesn’t come up when you do an emergency boot off the flash root filesystem!)

My perspective is that having a completely separate root filesystem in flash is a feature, not a bug. When I deliver OpenWRT routers to remote locations, I configure the flash filesystem with the minimum requirements for remote maintenance: generally, this is the firewall, dynamic dns, and ssh with a key. Any added-value software goes on the USB root filesystem – generally this software is bigger or has more complex configuration.

If something goes wrong in the remote location, especially if somebody misconfigures OpenWRT, then recovery is easy: pull out the USB drive and reboot. (If the USB drive dies, simply rebooting is enough.) There’s a separate, known good configuration which will be sufficient for me to have remote access.

In a past life I helped design embedded network devices that got delivered to remote (unstaffed) locations. We used dual flash partitions and a custom boot ROM, to ensure that the devices would always be able to boot. Even a cosmic ray flipping the wrong bit wouldn’t brick the device: it would simply boot from the alternate partition. Similarly, software upgrades that affected the flash image would only be written to one partition, so you could fallback by booting the alternate.

My strategy for OpenWRT borrows from this prior experience, and also from the hidden “recovery” partition on many Windows computers, where there’s a stripped-down partition to allow restoring the original software in case of disaster.

A friend runs two separate subnets at home: one is for home computing/entertainment, and the other is work-related. It’s important to keep the entertainment users off the work LAN. We recently replaced his aging firewall with an ASUS WL-520gU running OpenWRT, so I had to work out how to create separate LANs under OpenWRT.

We picked the the 520gU because it has 1 WAN port, 4 LAN ports, and a USB port, and becuase it runs OpenWRT well. We set up the root filesystem on an external USB stick (see previous posts). Then we had to work out how to have two separate networks on the LAN side.

OpenWRT sets up networks by tying together ports into VLANs (see fuller discussion on the OpenWRT site). Each of these groups of ports becomes a separate sub-interface of the master ethernet interface. For example, the standard configuration on a 520gU has (as I said) 1 WAN port and 4 LAN ports. But all of those ports are connected to the same switch: the only difference between WAN and LAN is that they’re on different VLANs, and some labelling onthe outside of the box. The master ethernet interface is eth0; the standard OpenWRT LAN is eth0.0, and WAN is eth0.1.

All of this is set up on the OpenWRT Network / Interfaces configuration page, which manipulates the file /etc/config/network. I’ll show you the examples in /etc/config/network, partly because it’s easier to have text examples and partly because there are 2 admin UIs for OpernWRT (LUCI, which comes with OpenWRT, and X-WRT, a parallel project that I use). Here’s the default setup:

config 'switch' 'eth0'
        option 'vlan0' '1 2 3 4 5*'
        option 'vlan1' '0 5'

vlan0, as I said, is the standard LAN, and vlan1 is WAN. But wait, I said there were 5 ports (1 WAN and 4 LAN) but the ports here are numbered 0-5. What’s the 6th port? Well, it’s the internal switch connected to the CPU, and it’s included in every group of ports. If it weren’t then OpenWRT couldn’t see the packets.

Here’s the configuration with two different LANs:

config 'switch' 'eth0'
        option 'vlan0' '1 2 5*'
        option 'vlan1' '0 5'
        option 'vlan2' '3 4 5'

vlan1 is still the WAN port, but now vlan0 is one LAN network, and vlan2 is the other. Next time I’ll show how we did the firewall rules to complete the job. But before we tackle the firewall rules, I need to set up the WiFi.

WiFi is different, and isn’t on the 6-port ethernet switch. Instead, WiFi works with its own master interface, and sub-interfaces for each WiFi network that you’re either a client on, or an access point for. Then you can bridge the WiFi subinterface to a LAN subinterface, to create a network with both WiFi and LAN access types.

The standard OpenWRT configuration bridges the LAN to wl0, the default and first WiFi subinterface. We were able to use this same configuration to bridge wl0 to the first LAN, eth0.0.

In other posts I talked about booting my OpenWRT routers off an external USB stick. In this post I’ll describe how to make the /var directory tree persistent across reboots.

On a standard OpenWRT system, /var is a link to /tmp. This simplifies setup, but it means that /var gets wiped out on each reboot, because /tmp is a tmpfs (in-memory). Why do I care? Well, when you reboot you lose:

1. The package lists downloaded by opkg update
2. DHCP leases that OpenWRT handed out
3. Web pages that polipo, a caching proxy, has already loaded
4. System-specific Bluetooth configuration, especially paired devices

And I care about all of these. Also, the tmpfs is kind of small, which makes web caching harder.

Unfortunately, the assumption that /var is /tmp has leaked into a number of places, so you’ll need to fix those. Start with the USB Storage howto to setup your external filesystem, and with my pivotroot scripts as well. Make these changes when you’re booted off internal flash – don’t make them to your live filesystem.

I’ll assume your external USB stick is mounted on /mnt.

1. Undo the link. cd /mnt; rm var; mkdir -p var/etc
2. Fix the bootup scripts. Exceprts are below, but you need to fix references in etc/init.d/syslog and etc/init.d/boot on your USB stick

diffs for etc/init.d/boot:

--- /etc/init.d/boot    Sun May 10 21:09:45 2009
+++ boot        Sat Jan  1 00:08:59 2000
@@ -38,10 +38,14 @@
        config_load system
        config_foreach system_config system

-       mkdir -p /var/run
-       mkdir -p /var/log
-       mkdir -p /var/lock
-       mkdir -p /var/state
+       mkdir -p /tmp/run
+       [ ! -f /var/run ]   && ln -s /tmp/run   /var
+       mkdir -p /tmp/log
+       [ ! -f /var/log ]   && ln -s /tmp/log   /var
+       mkdir -p /tmp/lock
+       [ ! -f /var/lock ]  && ln -s /tmp/lock  /var
+       mkdir -p /tmp/state
+       [ ! -f /var/state ] && ln -s /tmp/state /var
        mkdir -p /tmp/.uci
        chmod 0700 /tmp/.uci
        touch /var/log/wtmp

diffs for etc/init.d/syslog:

--- /etc/init.d/syslog  Fri Jun 12 08:16:26 2009
+++ syslog      Sat Jan  1 00:09:22 2000
@@ -11,7 +11,7 @@
        local cfg="$1"
        local type file size ipaddr port IPCALC_CMD SYSLOG_CMD
        local DEFAULT_type="circular"
-       local DEFAULT_file="/var/log/messages"
+       local DEFAULT_file="/tmp/log/messages"
        local DEFAULT_size=16
        local DEFAULT_ipaddr=""
        local DEFAULT_port=514