Yet Another IPv6 Setup

Last weekend I got the bright idea to give IPv6 another attempt on my network. I had previously tried it a while back, tunneling straight from my Cisco router. However, I had an older Cisco router that could only do IPv6 on a "testing" build of IOS. Being sick and tired of potential "issues" with this build, I wound up just ditching IPv6 for the time being. At the time I had a single static IP, and I did not have any other good configuration options.

These days I have a connection with multiple static IPs, so I have more options available to myself now. My current network config is also rather interesting, so allow me to illustrate:
{Internet} ---->(Cisco 4500 rtr)---->(FreeBSD firewall)====>{Multiple internal subnets}

Basically I've banished all NAT to that Cisco, which does the common port-translating NAT for most machines on my network. However, it also does 1:1 (bi-directional) NAT for my firewall and server machines. The advantage of 1:1 NAT is that you only translate the network address, and nothing else. As such, you can use it for a lot more than just the usual restrictive TCP and UDP setup you have with port-translating NAT. Of course 1:1 NAT does just translate network addresses, so you need to configure your firewall as if your machines did have public addresses.

So coming out of the Cisco router, I have my private address range (with some public IPs mapped to some of the private IPs). Just behind it, the FreeBSD firewall takes the next step. First, it filters out any traffic I don't want going into my network (obviously). Second, it takes this private address range and subnets it further. (the internal side of the box is a VLAN trunk to my switches) Yes, I have multiple subnets internally. This lets me separate different types of traffic for the purposes of flexibility and/or security.

Basically, I wanted to connect my various internal networks to the IPv6 Internet, by way of this FreeBSD firewall. (FYI, the system is running FreeBSD 6.0-RELEASE at the time of this writing, and is named "Tritanium") To accomplish this, I had two main options at my disposal:

• Use a "tunnel broker" service (i.e. Hurricane Electric, Hexago, or Sixxs)
• Use a 6to4 tunnel (RFC3056)

The last time around, I used a tunnel broker. However, this method depends on having an active account with an external service. While that does work, it is a bit of an annoyance it would be nice to do without.
As such, I decided to attempt 6to4 this time around. The 6to4 method works by directly mapping your public IPv4 address into an IPv6 /48 subnet. Then your border router essentially tunnels IPv6 packets directly inside IPv4 packets (as IP protocol 41). What's really cool about this is that you don't need any external services or configurations. If you go to an IPv6 site, and ping your local 6to4 address, you will see the inbound packets while sniffing your external interface. So, with all that being set, time to get on to an account of my experiences:

Step 1: Figure out your IPv6 address
This is probably the easiest step of the entire adventure. You just take your public IPv4 address (yes, it does have to be a public routable address), convert it to hexadecimal, and tack it onto the end of the 6to4 prefix (2002). For the sake of this writeup, lets assume our public address is "12.34.56.78". In hex, that translates to "0C22384E". So that translates into the following 6to4 subnet:
2002:c22:384e::/48

(IPv6 lets you omit leading zeros and abbreviate the end of the address, in case you were wondering.)

Step 2: Configure the 6to4 tunnel
This was probably one of the most frustrating steps, despite the fact that it looks like it should be the easiest. I blame my configuration more than anything else, though. You see, while the external interface on Tritanium maps directly to a public IP address, it actually has a private IP address itself.

In short, you have to configure FreeBSD's stf(4) interface with your 6to4 address, and then setup routing. However, I had a bit of a problem. You see, for this to work in both directions, two things had to happen. First, Tritanium had to have something telling it that it did indeed have a relationship with its public IP. Second, certain sanity checks (that prevent you from using stf with private IPs) had to be bypassed.

The first step was easy. I just created an alias on Tritanium's external interface with its public IP address, and a /32 netmask:
# ifconfig fxp1 inet 12.34.56.78 netmask 0xffffffff alias

The second step turned out to be a lot more involved. What's going to happen is that Tritanium will be receiving incoming 6to4 packets where the IPv4 address (1:1 translated to a private IP by the Cisco router) will not match the IPv6 address (based on our public IP) contained within. Let's just say that it does not work out of the box. Upon reading the stf man page, it does however tell us that the: "Ingress filter can be turned off by IFF_LINK2 bit". (this is the "link2" flag you can pass to ifconfig when setting up an interface)

Glossing over what was an entire night of frustration and debugging, let's just say that LINK2 doesn't really do much of anything. The stf interface driver has a lot of sanity checks, some failing with my configuration, and the "ingress filter" block of code that LINK2 disables isn't one of those checks.

The fix I ultimately came up with involved fixing the source code (if_stf.c) to make the LINK2 flag disable the sanity checks that were failing on my setup. The result of my fix can be summed up in this patch. (yes, it is against 6.0-RELEASE, but it shouldn't be hard to adapt to a newer version)

Once that file was patched, and the kernel module reloaded, the next step was pretty simple:
# ifconfig stf0 create
# ifconfig stf0 inet6 2002:c22:384e::1 prefixlen 16 link2

The third step involves setting up routing. For this, we need to create a route to a public 6to4 router. I took the easy way with this one, as there is a public "anycast" address for your nearest 6to4 router. That address is 192.88.99.1 (in IPv4), or 2002:c058:6301:: (in 6to4 IPv6). So I set my default IPv6 route to that:
# route add -inet6 default 2002:c058:6301::

Step 3: Internal subnets and routing
First I set IPv6 addresses on my internal interfaces, using subnets of the /48 that I got with 6to4:
# ifconfig vlan1 inet6 2002:c22:384e:1::1 prefixlen 64
# ifconfig vlan2 inet6 2002:c22:384e:2::1 prefixlen 64
# ifconfig vlan3 inet6 2002:c22:384e:3::1 prefixlen 64
# ifconfig vlan4 inet6 2002:c22:384e:4::1 prefixlen 64

Then I enabled IPv6 forwarding:
sysctl net.inet6.ip6.forwarding=1

Finally, I enabled rtadvd(8) in my rc.conf, and also told it which interfaces to run on (a subset of the ones above), and then started it:
# /etc/rc.d/rtadvd start

In case you were wondering, "rtadvd" is the router advertisement daemon. Using it, all my internal IPv6-enabled systems will automatically learn their IPv6 network addresses and routers. Pretty cool, eh?

Step 4: The firewall
While the IPv6 Internet is probably not yet anywhere near as hazardous as the IPv4 internet, chances are that you still want some level of protection. Since I used to use OpenBSD for my firewalls in the past, I had become accustomed to using pf(4). Unfortunately, I discovered that pf has a very annoying problem with my configuration. Just having pf enabled (even with all rules flushed) seemed to inhibit IPv6 packet forwarding! It was actually kinda strange how it behaved. I could talk normally on the IPv6 Internet from Tritanium directly. However, only ICMP worked correctly from my internal machines. Outbound TCP and UDP packets were never forwarded across Tritanium, while inbound ones worked just fine.

What's the solution? Use ipfw(8) instead of pf, and your problem will be solved. Just make sure you configure the IPv4 side of ipfw so that IP protocol 41 packets are permitted unscathed. (my version wouldn't let me specifically allow proto 41, for some strange reason, so I just permitted all IP packets that I hadn't explicitly blocked with some other rules elsewhere in my configuration.)

Step 5: And there was much rejoicing!
I'm now connected to the IPv6 internet, after a week's worth of evening tinkering. Yippee!
I may eventually put all my configurations into rc.conf (I had some difficulties when I first tried, and gave up soon afterwards), but right now most of this stuff is just running out of rc.local on the machine.