One thing that bugs me about this setup is that I hardcoded some IP addresses. This is a really bad idea, in fact half the reason DNS exists is to stop people from doing this. Someday the VPN administrators could decide to renumber their gateway, and then I would have to connect to each and every WRT and update the configuration, or all my customers would be cut off from the VPN.

I could stop hardcoding the VPN gateway pretty easily, by having it auto-update whenever the WRT boots. Look up the correct IP address and write it to /etc/hosts. (You need to be careful not to trigger the VPN autoconnect when you do the lookup, of course.)

The DNS servers are a little harder to fix. To look them up, we need to actually connect the VPN, and then pull out the nameservers that the VPN tells us to use. But this could be done automatically each time the VPN is connected, and it would make the whole setup more robust. Don’t forget to restart dnsmasq after you reconfigure it.

Another thing that bothers me is that there are a lot of gyrations before the VPN gets connected: a packet goes to a private IP address, matches on the vpn_trigger rule, which queues a message for the ulog daemon, which writes to a file, which causes the file to exist, and also causes tail to create some output, which finally causes the script to start vpnc. Whew!

One suggestion is to use inotify to watch the file, which is a good idea but still requires a file. (Also I don’t see inotify in the OpenWRT Kamikaze package list.) Another possibility would be to write a ulog plugin so that I can omit the file altogether. That might make VPN startup quicker, which would be a good thing.

Finally, if a customer wanted it, there are more sophisticated setups possible. It might be useful to connect to more than one VPN (either serially or at the same time), or to have the different LAN ports on the OpenWRT connect to different VPNs. All of these are possible if the scenario warrants it.

What are your ideas?

This process requires some knowledge about your VPN setup. To keep my post from getting too long, I’m assuming that you know your VPN domain name and IP address range, that you can quickly figure out the IP addresses of hosts and DNS servers, and that you have a working vpnc config file.

The first step is installing additional packages on OpenWRT. You can install these from the web interface, or using opkg install at the OpenWRT shell:

1. vpnc, the Cisco VPN client
2. ulogd-mod-extra, which pulls down the ulog daemon
3. kmod-ipt-ulog, kernel modules for iptables and ulog
4. iptables-mod-ulog, part of the tool for adding rules to iptables

Next, we need to make sure you can always reach the VPN gateway host. So we configure it into /etc/hosts. Look up the IP address (using nslookup, dig, or a similar tool) and add a line to /etc/hosts like:

aaa.bbb.ccc.ddd vpn.example.org


Now that we know we can reach the VPN gateway, we will redirect dnsmasq to always use the internal servers for the VPN’s domain. Look up the domain nameservers using nslookup or dig from inside the VPN, or just look at the nameservers in /etc/resolv.conf when you’re connected from your PC. Then edit /etc/config/dhcp on OpenWRT and add lines like this:

# EXAMPLE.ORG private servers
list server '/example.org/aaa.bbb.ccc.ddd'
list server '/example.org/aaa.bbb.ccc.eee'


Now, we need the script that will manage the VPN connection. Cut and paste this code into /usr/bin/autostart-vpn.sh:

#!/bin/sh
#
# Autostart vpnc
#
# From http://www.kaufmanfamily.net/blog/2010/05/how-to-auto-connect-a-cisco-vpn-with-openwrt
# DHK 4/17/2010

MYPID=/var/run/autostart-vpnc.pid
LOGFILE=/var/log/ulogd.syslogemu
PIDFILE=/var/run/vpnc/pid

is_vpn_connected() {
        connected=0
        if [ -s $PIDFILE ]; then
                ps=`ps | awk -v pid=\`cat $PIDFILE\` '$1 == pid && $5 == "vpnc" { print $0 }'`
                if [ -n "$ps" ]; then
                        connected=1
                fi
        fi
}

# Fill in our PID file
echo $$ > $MYPID

# Loop, monitoring the VPN
while true; do
	is_vpn_connected

	if [ $connected -eq 0 ]; then
		# VPN is not connected. Wait for a request, then start it

		# Wait for a log message, denoting that someone is trying to connect
		while [ ! -e $LOGFILE ]; do
			sleep 10
		done
		pkt=`tail -0 -f $LOGFILE | head -1`
		logger "autoconnect[$$] connecting due to $pkt"

		# Start the vpn
		date >> /tmp/autoconnect-vpnc.log
		vpnc >> /tmp/autoconnect-vpnc.log

		# Let the VPN settle
		sleep 2
	else
		# VPN is connected. Wait for it to drop, then clean up

		# Wait for the VPN to disconnect
		while [ $connected -eq 1 ]; do
			is_vpn_connected

			# sleep 10 seconds, check again
			sleep 10
		done
		logger "autoconnect[$$] disconnected"

		# Clean up route table
		if [ -f /var/run/vpnc/defaultroute ]; then
			outsidegw=`awk '{print $3}' /var/run/vpnc/defaultroute`
			currentgw=`netstat -rn | awk '$1 == "0.0.0.0" && $4 == "UG" {print $2}'`
			if [ "X$outsidegw" != "X$currentgw" ]; then
				if [ "X$currentgw" != "X" ]; then
					route delete default gw $currentgw
				fi

				# Restore exactly what was saved, except vpnc syntax is slightly wrong
				route add `sed -e 's/via/gw/;' /var/run/vpnc/defaultroute`
			fi
		fi

		# Clean up resolv.conf
		resolvconf -d
	fi
done

We’re almost there. Cut and paste the following code into /etc/init.d/autostart-vpn; this is the startup script that creates the iptables rules and starts the last script at boot time. Make sure you edit the script to list the correct networks for your VPN, and check that the locations (hardcoded, unfortunately) for inserting vpn_trigger in the FORWARD and OUTPUT rulesets makes sense:

#!/bin/sh /etc/rc.common
#
# From http://www.kaufmanfamily.net/blog/2010/05/how-to-auto-connect-a-cisco-vpn-with-openwrt
# DHK 4/17/2010
#
# Start after dnsmasq
START=80

VPN_NETWORKS="10.0.0.0/8 192.152.0.0/16"
EXT_IF="eth0.1"

check_firewall_rules() {
	trigger=`iptables -L vpn_trigger 2>/dev/null`

	if [ "X$trigger" == "X" ]; then
		# Fill in vpn_trigger ruleset
		iptables -N vpn_trigger

		for net in $VPN_NETWORKS; do
			iptables -A vpn_trigger -o $EXT_IF --dest "$net" -j ULOG
		done

		# Hook vpn_trigger into OUTPUT and FORWARD rules
		# Ought to do something smarter than hardcoding the position
		iptables -I OUTPUT  4 -j vpn_trigger
		iptables -I FORWARD 4 -j vpn_trigger
	fi
}

start() {
	check_firewall_rules

	/usr/bin/autostart-vpnc.sh &
}

stop() {
	if [ -f /var/run/autostart-vpnc.pid ]; then
		kill `cat /var/run/autostart-vpnc.pid`
	fi
}

One detail I skipped last time is that vpnc, as packaged for OpenWRT Kamikaze, will stomp on your resolv.conf file. Its default configuration just doesn’t work on OpenWRT. (The issue is that OpenWRT puts the WAN resolv.conf details in a non-standard place.) There’s an easy fix for this, though. Cut and paste the following code into /sbin/resolvconf. vpnc will find resolvconf and use it to manage /etc/resolv.conf correctly.

#!/bin/sh
#
# Simple resolvconf manager to integrate vpnc better with WRT
#
# Update /tmp/resolv.conf.auto, NOT /etc/resolv.conf. This affects the
# DNS resolver operation, which is actually the right thing.
#
# Usage:
#
#   resolvconf -a [if] < new-resolvconf
#
#   resolvconf -d [if
#
# We ignore the [if] argument.
#
# From http://www.kaufmanfamily.net/blog/2010/05/how-to-auto-connect-a-cisco-vpn-with-openwrt
# DHK 4/13/2010

BACKUP=/tmp/resolv.conf.bak
RESOLV=/tmp/resolv.conf.auto

if [ $1 == "-a" ]; then
	# Change resolv.conf

	if [ ! -e $BACKUP ]; then
		cp $RESOLV $BACKUP
	fi

	cat > $RESOLV
fi

if [ $1 == "-d" ]; then
	# Restore original resolv.conf

	if [ -e $BACKUP ]; then
		mv $BACKUP $RESOLV
	fi
fi

Now is a good time to make sure you’ve installed your VPN configuration into /etc/vpnc/default.conf. It’s a good idea to test out your vpnc config on another machine before running it on OpenWRT.

OK, let’s enable the services we need. You can do this from the OpenWRT web interface, or the command line:

/etc/init.d/ulogd enable
/etc/init.d/autostart-vpnc enable


Reboot your OpenWRT to get all the services set up. You’ll want to watch the system message log, so in one ssh connection run the log reader:

logread -f


and in another ssh connection start pinging a host in the VPN:

ping somehost.example.org


You should see a message in the system log, and after a short delay you’ll start getting ping responses. Make sure to test the auto-connect from a host plugged in to your OpenWRT’s LAN port as well as from the shell: if auto-connect works directly from the OpenWRT shell, but not from the LAN, then your iptables OUTPUT rule is correct but your FORWARD rule isn’t. (If the problem is reversed, then the rules are reversed.)

If auto-connection doesn’t work, you can check the log at /tmp/autoconnect-vpnc.log and then debug the process step-by-step:

1. First, check that your vpnc configuration works:
   vpnc
2. Then, check that the vpn_trigger iptables rule is being called by looking at the packet counts:
   iptables -L vpn_trigger -v
3. If vpn_trigger is being called, make sure that ulogd is writing to the correct file:
   cat /var/log/ulogd.syslogemu
4. check that the autoconnect script is actually running with ps

The hardest thing to check is that you have your DNS setup correct. I usually do this by checking the vpn_trigger rules first, then use nslookup to query a behind-the-vpn host.

I started with OpenWRT (of course!) and vpnc. So far I’ve deployed this setup using OpenWRT Kamikaze, versions 8.09.1 and 8.09.2. I’ll describe how the process works first, and then show the details in my next post.

The first thing we need to do is intercept DNS queries for the secure network. (Most secure networks will have a private DNS server behind the firewall, with lots of information that’s not in the public version.) OpenWRT uses the dnsmasq name resolver, and happily dnsmasq provides for exactly this case. So, we configure dnsmasq to know that the servers for the VPN domain are on its private addresses, and not to use the public servers.

Some companies might have the name of the VPN gateway in the same domain as their private, through the VPN environment. This could be a problem: how do we find the IP address for the gateway, if we need the VPN to talk to the domain servers? The easy answer for now is to just hardcode it in /etc/hosts on the OpenWRT.

OK, so we dealt with DNS name resolution, but how do we get the IP packets to go through the VPN? Actually, once vpnc connects, it will set up routing on the OpenWRT for us. So the only tricky bit is starting up vpnc at just the right time.

One way to do this would be to start vpnc whenever the WRT boots. This works, pretty much, but might be considered hostile by the VPN administrators. Instead we want to start the vpn tunnel whenever a user tries to access a machine in the private environment. To do this, we need to capture IP packets being routed to private IP addresses. This sounds like a job for iptables!

iptables is part of the Linux kernel which lets you define rules for handling IP packets. It does all kinds of cool things – filtering, QOS, NAT – but all we need it to do is tell us when someone is trying to connect through the VPN, so that we can start up the VPN tunnel. To do this, I used the iptables “ULOG” target: ULOG writes a message to a userspace daemon called ulogd, which acts like a hub (kind of like syslogd) and routes the messages to other programs. I kept it simple and just wrote the message to a file.

To tie it all together, I wrote a script which monitors the ulog file. The script gets automatically started at boot time: if the VPN isn’t up yet, it just monitors the ulog file. When a packet trace appears in the ulog file, we know it’s time to start the VPN up. Once the VPN is connected, the script hangs out, waiting for it to get shut down — either manually or by an idle timer. If the VPN gets shut down, then the script goes back to watching the ulog file.

OK, so we have DNS requests going to the private servers, and IP traffic causes the VPN to autoconnect. We’re done, right? Well, almost. It turns out that sometimes vpnc shuts down messily. It can leave the OpenWRT without a default route, or it can leave the resolv.conf file pointing to behind-the-tunnel servers. Either one of these problems is enough to prevent reconnecting the VPN! So I added a cleanup step to my script, which makes sure that vpnc has cleaned up after itself.

Add some logging and a simple web interface, and we’re all set. Details in my next post.

Once again, I’ll show the configuration from /etc/config files. The first thing we need to do is define a network for the new LAN (in fact, I renamed the old LAN as well). The type is “bridge” so we can bridge it with the wireless LAN, as we noted last time. The interface is eth0.2, which is the new VLAN. This looks something like:

config 'interface' 'homelan'
option 'type' 'bridge'
option 'ifname' 'eth0.2'


Finally I wrote some iptables rules to prevent forwarding between the two LANs, and to prevent access to the router itself.
Then we need to add a section to the dhcp configuration. In this case we don’t want to run DHCP on this interface, so I set “ignore” to 1.

config 'dhcp'
option 'interface' 'homelan'
option 'ignore' '1'


Finally, and this is key, we need to add some sections to the firewall config file. First we need to define a “zone”, which is an OpenWRT concept. This (typical) zone allows packets to be transmitted or received on the homelan interface, but not to be forwarded between interfaces:

config 'zone'
option 'name' 'homelan'
option 'input' 'ACCEPT'
option 'output' 'ACCEPT'
option 'forward' 'REJECT'


Then we configure forwarding from the homelan into the WAN:

config 'forwarding'
option 'src' 'homelan'
option 'dest' 'wan'
option 'mtu_fix' '1'


We picked the the 520gU because it has 1 WAN port, 4 LAN ports, and a USB port, and becuase it runs OpenWRT well. We set up the root filesystem on an external USB stick (see previous posts). Then we had to work out how to have two separate networks on the LAN side.

OpenWRT sets up networks by tying together ports into VLANs (see fuller discussion on the OpenWRT site). Each of these groups of ports becomes a separate sub-interface of the master ethernet interface. For example, the standard configuration on a 520gU has (as I said) 1 WAN port and 4 LAN ports. But all of those ports are connected to the same switch: the only difference between WAN and LAN is that they’re on different VLANs, and some labelling onthe outside of the box. The master ethernet interface is eth0; the standard OpenWRT LAN is eth0.0, and WAN is eth0.1.

All of this is set up on the OpenWRT Network / Interfaces configuration page, which manipulates the file /etc/config/network. I’ll show you the examples in /etc/config/network, partly because it’s easier to have text examples and partly because there are 2 admin UIs for OpernWRT (LUCI, which comes with OpenWRT, and X-WRT, a parallel project that I use). Here’s the default setup:

config 'switch' 'eth0'
        option 'vlan0' '1 2 3 4 5*'
        option 'vlan1' '0 5'

vlan0, as I said, is the standard LAN, and vlan1 is WAN. But wait, I said there were 5 ports (1 WAN and 4 LAN) but the ports here are numbered 0-5. What’s the 6th port? Well, it’s the internal switch connected to the CPU, and it’s included in every group of ports. If it weren’t then OpenWRT couldn’t see the packets.

Here’s the configuration with two different LANs:

config 'switch' 'eth0'
        option 'vlan0' '1 2 5*'
        option 'vlan1' '0 5'
        option 'vlan2' '3 4 5'

vlan1 is still the WAN port, but now vlan0 is one LAN network, and vlan2 is the other. Next time I’ll show how we did the firewall rules to complete the job. But before we tackle the firewall rules, I need to set up the WiFi.

WiFi is different, and isn’t on the 6-port ethernet switch. Instead, WiFi works with its own master interface, and sub-interfaces for each WiFi network that you’re either a client on, or an access point for. Then you can bridge the WiFi subinterface to a LAN subinterface, to create a network with both WiFi and LAN access types.

The standard OpenWRT configuration bridges the LAN to wl0, the default and first WiFi subinterface. We were able to use this same configuration to bridge wl0 to the first LAN, eth0.0.

On a standard OpenWRT system, /var is a link to /tmp. This simplifies setup, but it means that /var gets wiped out on each reboot, because /tmp is a tmpfs (in-memory). Why do I care? Well, when you reboot you lose:

1. The package lists downloaded by opkg update
2. DHCP leases that OpenWRT handed out
3. Web pages that polipo, a caching proxy, has already loaded
4. System-specific Bluetooth configuration, especially paired devices

And I care about all of these. Also, the tmpfs is kind of small, which makes web caching harder.

Unfortunately, the assumption that /var is /tmp has leaked into a number of places, so you’ll need to fix those. Start with the USB Storage howto to setup your external filesystem, and with my pivotroot scripts as well. Make these changes when you’re booted off internal flash – don’t make them to your live filesystem.

I’ll assume your external USB stick is mounted on /mnt.

1. Undo the link. cd /mnt; rm var; mkdir -p var/etc
2. Fix the bootup scripts. Exceprts are below, but you need to fix references in etc/init.d/syslog and etc/init.d/boot on your USB stick

diffs for etc/init.d/boot:

--- /etc/init.d/boot    Sun May 10 21:09:45 2009
+++ boot        Sat Jan  1 00:08:59 2000
@@ -38,10 +38,14 @@
        config_load system
        config_foreach system_config system

-       mkdir -p /var/run
-       mkdir -p /var/log
-       mkdir -p /var/lock
-       mkdir -p /var/state
+       mkdir -p /tmp/run
+       [ ! -f /var/run ]   && ln -s /tmp/run   /var
+       mkdir -p /tmp/log
+       [ ! -f /var/log ]   && ln -s /tmp/log   /var
+       mkdir -p /tmp/lock
+       [ ! -f /var/lock ]  && ln -s /tmp/lock  /var
+       mkdir -p /tmp/state
+       [ ! -f /var/state ] && ln -s /tmp/state /var
        mkdir -p /tmp/.uci
        chmod 0700 /tmp/.uci
        touch /var/log/wtmp

diffs for etc/init.d/syslog:

--- /etc/init.d/syslog  Fri Jun 12 08:16:26 2009
+++ syslog      Sat Jan  1 00:09:22 2000
@@ -11,7 +11,7 @@
        local cfg="$1"
        local type file size ipaddr port IPCALC_CMD SYSLOG_CMD
        local DEFAULT_type="circular"
-       local DEFAULT_file="/var/log/messages"
+       local DEFAULT_file="/tmp/log/messages"
        local DEFAULT_size=16
        local DEFAULT_ipaddr=""
        local DEFAULT_port=514

The standard OpenWRT pivotroot is pretty basic. It needs to know where your USB disk will show up, and it assumes that the filesystem is in good shape. But I have OpenWRT systems with multiple USB ports, plus I’d like to be able to use USB hubs. And my OpenWRT systems tend to run until the power goes off – so I want to make sure the filesystem is clean before I boot off it.

In order to make things safer, I choose to run an ext3 filesystem. This is harder on the flash, because of the journal writes, but safer recovery for unclean shutdowns – which are the norm. A corrupted filesystem only gets worse, though, so I want to run fsck before I use the filesystem. (That means you’ll need to install e2fsprogs in the flash-based root filesystem.)

Since my USB sticks can move around, I search through all the available disk partitions to find one that looks like an OpenWRT system. That means I may try to mount filesystems and have them fail, but this is pretty safe. Eventually I’ll find the right filesystem, and if not I’ll boot off flash. I do assume that the filesystem will be on partition 1, and that it will be an ext3 filesystem.

I hate that the standard pivotroot uses the /mnt mountpoint. I use /flash instead. When I’m looking for an OpenWRT filesystem I look for /flash (and also /sbin/init). (So I need to mkdir /flash on both the flash-based and USB root filesystems.)

Finally, I want to know what’s going on – especially with fsck – and a way to turn off the pivotroot easily. I put logs in /tmp/pivotroot.log, I make the power LED flash while I’m fsck’ing and searching for a root fs, and I have rcS check whether the pivotroot script is executable. A quick chmod -x /flash/etc/init.d/pivotroot, and I can boot off flash.

Enough yakking, here’s the code. /etc/init.d/pivotroot, first, but there’s more below:

#!/bin/sh
# From http://oldwiki.openwrt.org/UsbStorageHowto.html - DHK 7/14/09
#
# Greatly enhanced to select among (maybe) several USB Mass Storage devices
# Assumes that the root fs will be on an ext3 filesystem in partition 1,
# and looks for /flash and /jffs in the root directory. Also assumes that
# mounting and checking each candidate device is harmless. Runs e2fsck
# over partitions before mounting them; note that this causes spurious
# date-related complaints every time, since time has not been synchronized
# yet at the time pivotroot is run.
#
# Output from this script is stashed at /tmp/pivotroot.log.
#
# DHK 7/23/2009

# Set Power LED flashing while we look for OpenWRT root
if [ -f /etc/diag.sh ]; then
  LED=1
  . /etc/diag.sh
  POW=`cat /proc/diag/led/power`
  set_led power f
fi

# install needed modules for usb and the ext3 filesystem
# We could defer loading the filesystem modules until we know there's a useful partition.
# **NOTE** for usb2.0 replace "uhci" with "ehci_hcd"
# **NOTE** for ohci chipsets replace "uhci" with "usb-ohci"
# **NOTE** for WL-500gP usb-uhci not usb-ohci
echo -n "pivotroot loading kernel modules: "
for module in usbcore usb-uhci scsi_mod sd_mod usb-storage ext2 jbd ext3 ; do {
  echo -n "$module "
  insmod $module
}; done
echo
# this may need to be higher if your disk is slow to initialize
sleep 4s

PART1=`find /dev/scsi -name part1`

if [ -z $PART1 ]; then
  echo No partitions, skipping pivotroot
else
  # Look for a mountable USB stick with, maybe, OpenWRT on it
  for dev in `find /dev/scsi -name part1`; do
    echo pivotroot checking $dev
    e2fsck -p $dev
    fsck=$?
    echo fsck status is $fsck
    if [ $fsck -eq 2 ]; then
      echo Corrected errors on $dev, need to reboot
      sleep 5
      reboot
    elif [ $fsck -eq 1 ]; then
      echo Corrected errors on $dev
    elif [ $fsck -gt 2 ]; then
      echo No usable filesystem on $dev
      continue
    fi
    echo Mounting $dev
    mount -t ext3 $dev /flash
    mt=$?
    if [ $mt -eq 0 ]; then
      if [ -x /flash/sbin/init -a -d /flash/jffs -a -d /flash/flash ]; then
        echo Found OpenWRT root on $dev
        # Side-effect - leave /flash mounted
        break;
      else
        echo "Missing /sbin/init, /jffs, or /flash (mount status was $mt)"
        [ -x /flash/sbin/init ] || echo Failed -x /sbin/init
        [ -d /flash/flash ]     || echo Failed -x /flash
        [ -d /flash/jffs ]      || echo Failed -x /jffs
        umount $dev
      fi
    else
      echo mount status is $mt
    fi
  done

  # if everything looks ok, do the pivot root
  [ -x /flash/sbin/init ] && {
          mount -o move /proc /flash/proc && \
          pivot_root /flash /flash/flash && {
                  mount -o move /flash/dev /dev
                  mount -o move /flash/tmp /tmp
                  mount -o move /flash/jffs2 /jffs2 2>&-
                  mount -o move /flash/sys /sys 2>&-
          }
  }
fi

# Restore Power LED to previous state
[ $LED -eq 1 ] && set_led power $POW

Addition to /etc/init.d/rcS (put this before the line LOGGER=”cat”):

# Improved from http://oldwiki.openwrt.org/UsbStorageHowto.html
# Switch the root filesystem to USB, if present
# DHK 7/14/09
if [ $2 == "boot" -a -x /etc/init.d/pivotroot ] ; then
        /etc/init.d/pivotroot > /tmp/pivotroot.log
fi

Finally, I add this line to the bottom of /etc/banner, but on the flash filesystem only. When I ssh in to an OpenWRT box I’ll know immediately how it booted:

Booted off internal flash

Remember, this works for me, but I make no guarantees it will work for you. Make sure you’ve saved your configuration before fooling with this stuff, and that you know how to reflash your router if you brick it.

So how do you boot OpenWRT off an external USB stick? Well, midway through the boot process you change the root filesystem from the one on the internal flash to the one on the external stick. That means you need to check first that there’s a filesystem to boot off of. There’s a standard procedure on the OpenWRT wiki, using what’s called a pivotroot script.

In White Russian (the older OpenWRT release) the procedure was to replace /sbin/init with pivotroot. This is scary, because if you step wrong you’ve bricked your router – made it unbootable. In Kamikaze (the newer release), the recommendation has changed, and instead pivotroot is explicitly called from rcS, a very early bootup script. This is a better idea, and when I was done with my changes it was even safer.

What does a pivotroot script need to do? It needs to:

1. Make sure the kernel can recognize your USB stick, by loading kernel modules to support USB and USB storage devices
2. Make sure the kernel can recognize the filesystem on your USB stick, by loading more kernel modules. For example, I use the ext3 filesystem on my USB stick, and that’s not built in to the OpenWRT kernels
3. Check whether your USB stick is even present
4. Mount the filesystem on your USB stick
5. Finally, if everything else is OK, it remounts all the filesystems so that your USB stick is the root filesystem and the flash-based root filesystems are mounted elsewhere

Note that this procedure still uses the flash-based root filesystem to get partway booted. Any special software packages you need to boot up – all those kernel modules, for example – and the pivotroot script itself need to be on the flash-based root filesystem, and not the root filesystem you see once you’re booted. This also makes debugging a pivotroot script tricky, because by the time OpenWRT is booted the environment is very different from the one pivotroot sees.

Next time I’ll outline the changes I made to improve the standard OpenWRT pivotroot.