A while back I created a highly available ipsec tunnel from our colocation facility at RackSpace to Amazon EC2/VPC using open source utilities such as OpenSwan & Heartbeat/Pacemaker.  

Amazon provides a HA solution out of the box using VPC but the problem with it is that it requires the your ipsec terminator on the colo side support BGP.  Unfortunately our Cisco ASAs do not support BGP so it was either spend a bunch of money buying something that would support BGP or come up with an open source solution.  Spending a bunch of money was not an option for us so I came up with a solution to our problems using our Cisco ASAs on the colo end and an openswan instance on the other side.  Other people have done this but I've yet to see anyone that's done in a highly available way.  We find that our EC2 instances will sometimes reboot themselves for no clear reason so only having 1 instance that could do this was not acceptable.  This is also required when for some reason you need to take down that instance for an upgrade or a number of other reasons.  We could not afford any downtime so a 2nd instance that could take over when needed was essential.

The trick was to use Amazon's new (released in late 2011) interface feature which allows you to attach and detach interfaces and their associated ips (internal and elastic) at will.  We can then use Heartbeat to monitor the servers and promote (detach interface from other instance, attach to this instance & manipulate routing tables) servers based on the reachability of the other instance.

Multiple routing tables on each instance were required so that the system always maintained a reachable interface via eth0 regardless if they had the eth1 interface which is the interface all other NAT & ipsec traffic went through.  This is done by applying scope to the routing tables based on where the traffic came in from.

I had to come up with a couple of custom scripts that use the ec2 command line tools to query and change interface location as well as change the routing tables on the individual box based on whether or not it had eth1.

Someday maybe I'll detail more of the specifics on how this was done but it is possible if anyone is out there looking.

I had a hard time figuring this out and there seems to be lots of conflicting information out there so I thought I'd write down my thoughts about this immediately after I figured it out.

I was trying to get a fully automated network install of Ubuntu 10.04 working with a static ip address that gets set up in the preseed file.  My entire preseed was working but the static ip address was not.  The example preseed has the following entries:

# If you prefer to configure the network manually, uncomment this line and
# the static network configuration below.
#d-i netcfg/disable_dhcp boolean true

# If you want the preconfiguration file to work on systems both with and
# without a dhcp server, uncomment these lines and the static network
# configuration below.
#d-i netcfg/dhcp_failed note
#d-i netcfg/dhcp_options select Configure network manually

# Static network configuration.
#d-i netcfg/get_nameservers string 192.168.1.1
#d-i netcfg/get_ipaddress string 192.168.1.42
#d-i netcfg/get_netmask string 255.255.255.0
#d-i netcfg/get_gateway string 192.168.1.1
#d-i netcfg/confirm_static boolean true

Great...so I should just be able to uncomment the disable_dhcp and the Static network configuration entries and it should work right? WRONG. I went through various permutations and could not get it to work. Finally, I broke down and decided to RTFM and found this:

http://d-i.alioth.debian.org/manual/en.amd64/apbs04.html#preseed-network

It reads:

Although preseeding the network configuration is normally not possible when using network preseeding (using “preseed/url”), you can use the following hack to work around that, for example if you'd like to set a static address for the network interface. The hack is to force the network configuration to run again after the preconfiguration file has been loaded by creating a “preseed/run” script containing the following commands:

killall.sh; netcfg

So....I added the following to my preseed.cfg file

d-i preseed/early_command string /bin/killall.sh; /bin/netcfg

and boom the static network config works. Yay!