Firewall Failover with CARP,PF and PFSYNC in OpenBSD 4.2 under VMWare ESX3

CARP Setup
----------

Reference
http://www.openbsd.org/faq/pf/carp.html


CARP Master
-----------
# cat /etc/hostname.pcn1
# ifconfig carp0 create
# ifconfig pcn1 up

# ifconfig carp0 vhid 2 pass mysecretpassword carpdev pcn1 advbase 1 advskew 1 state master PUB.LIC.IPA.DDR netmask 255.255.255.0 broadcast PUB.LIC.IPA.DDR

# echo "inet PUB.LIC.IPA.DDR 255.255.255.0 PUB.LIC.IPA.DDR vhid 2 pass mysecretpassword carpdev pcn1 advbase 1 advskew 1 state master" > /etc/hostname.carp0

Note : Enable Promiscuous Mode for Virtual Switch related with pcn1

# ifconfig carp1 create

# ifconfig carp1 vhid 1 pass mysecretpassword carpdev pcn0 advbase 1 advskew 1 state master 192.168.0.254 netmask 255.255.255.0 broadcast 192.168.0.255

Note : Enable Promiscuous Mode for Virtual Switch related with pcn0

# vi /etc/pf.conf

pass out on $lan_if proto carp keep state

# pfctl -f /etc/pf.conf

# sysctl -w net.inet.carp.allow=1
# vi /etc/sysctl.conf
net.inet.carp.allow=1

# echo "inet 192.168.0.254 255.255.255.0 192.168.0.255 vhid 1 pass mysecretpassword carpdev pcn0 advbase 1 advskew 1 state master" > /etc/hostname.carp1

# reboot

CARP Backup
-----------

# cat /etc/hostname.pcn1
# ifconfig carp0 create
# ifconfig pcn1 up

# ifconfig carp0 vhid 2 pass mysecretpassword carpdev pcn1 advbase 1 advskew 1 state backup PUB.LIC.IPA.DDR netmask 255.255.255.0 broadcast PUB.LIC.IPA.DDR

# echo "inet PUB.LIC.IPA.DDR 255.255.255.0 PUB.LIC.IPA.DDR vhid 2 pass mysecretpassword carpdev pcn1 advbase 1 advskew 1 state backup" > /etc/hostname.carp0

Note : Enable Promiscuous Mode for Virtual Switch related with pcn1

# ifconfig carp1 create

# ifconfig carp1 vhid 1 pass mysecretpassword carpdev pcn0 advbase 1 advskew 1 state backup 192.168.0.254 netmask 255.255.255.0 broadcast 192.168.0.255

# vi /etc/pf.conf
pass out on $lan_if proto carp keep state

# pfctl -f /etc/pf.conf

# sysctl -w net.inet.carp.allow=1
# vi /etc/sysctl.conf
net.inet.carp.allow=1

Note : Enable Promiscuous Mode for Virtual Switch related with pcn0

# echo "inet 192.168.0.254 255.255.255.0 192.168.0.255 vhid 1 pass mysecretpassword carpdev pcn0 advbase 1 advskew 1 state backup" > /etc/hostname.carp1

# reboot

But VMWare will not allow CARP device to be used. It will not accept any request to a real hardware unless it is
addresed to it. As a solution we have to recompile VMWare Kernel OR We have to enable Promiscuous Mode for virtual NIC and Virtual Switch.

Reference
---------
http://www.ogris.de/docs/vmware-server-vrrp.html
http://paul.ouderkirk.ca/2006/10/11/network-experiments-with-vmware-server
http://paul.ouderkirk.ca/2006/10/11/network-experiments-with-vmware-server

When using a vswitch, both the virtual adapter and vswitch must have promiscuous mode enabled
So here I enabled the Promiscuous Mode.

1. Login to the Virtual Infrastructure Client
2. Then Right Click on the Virtual Switch listed in the right bottom of the Main Window.
3. Thereby we will get the "PropertiesWindow"
4. Go to the Security Tab -> Edit Promiscuous mode to "Accept"
Thats all. The CARP Interface should work now

References
----------
http://pubs.vmware.com/vi35/server_config/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_adv_netwk.6.4.html
http://communities.vmware.com/thread/33241
http://www.virtualscoop.org/?q=comment/reply/9

Packet Filter (PF)
------------------
# vi /etc/rc.conf.local
# pf=YES

# pfctl -e

Enabled IP forwarding

# sysctl -w sysctl net.inet.ip.forwarding=1
# echo "sysctl net.inet.ip.forwarding=1" >> /etc/sysctl.conf
# sysctl -p /etc/sysctl.conf


PFSYNC Setup
------------

Master PF Server
----------------
# ifconfig pfsync0 syncdev pcn0 [ syncpeer 192.168.0.221 ]
# ifconfig pfsync0 up
# echo "up syncdev pcn0" > /etc/hostname.pfsync0
# vi /etc/pf.conf
sync_if=$lan_if
pass on $sync_if proto pfsync
pfctl -f /etc/pf.conf

# sysctl -w net.inet.carp.preempt=1
# vi /etc/sysctl.conf
net.inet.carp.preempt=1

# reboot

Backup PF Server
----------------
# ifconfig pfsync0 syncdev pcn0 [ syncpeer 192.168.0.220 ]
# ifconfig pfsync0 up
# echo "up syncdev pcn0" > /etc/hostname.pfsync0
# vi /etc/pf.conf
sync_if=$lan_if
pass on $sync_if proto pfsync
pfctl -f /etc/pf.conf

# sysctl -w net.inet.carp.preempt=1
# vi /etc/sysctl.conf
net.inet.carp.preempt=1
# reboot


##################################
/etc/pf.conf
##################################

# $OpenBSD: pf.conf,v 1.31 2006/01/30 12:20:31 camield Exp $
#
# See pf.conf(5) and /usr/share/pf for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.

############# OPTIONS,LISTS #######################

lan_if="pcn0"
wan_if ="pcn1"
int_carp_if="carp1"
ext_carp_if="carp0"
sync_if=$lan_if

INT_CARP_IP=192.168.0.254
EXT_CARP_IP=PUB.LIC.IPA.DDR
#PUBLIC_IP=$INT_CARP_IP
#PUBLIC_IP=192.168.0.254
REAL_SERVER=192.168.0.50

LAN_NODES = "{ 192.168.0.1, 192.168.0.2, 192.168.0.3, 192.168.0.4, 192.168.0.5 }"

scrub in all

########### NAT #########################################
############ Redirect HTTP,MySQL,SMTP,IMAP and FTP connections to REAL SERVER
rdr on $wan_if proto tcp from any to $EXT_CARP_IP port {80,3306,25,143,21} -> $REAL_SERVER
rdr on $lan_if proto tcp from any to $INT_CARP_IP port {80,3306,25,143,21} -> $REAL_SERVER

### SNAT Outgoing FTP,HTTP,SMTP,IMAP and MySQL traffic through LAN Interface
nat on $lan_if proto tcp from any to $REAL_SERVER port { 21,80,25,143,3306 } -> $lan_if
nat on $lan_if proto tcp from $REAL_SERVER to any port { 21,80,25,143,3306 } -> $lan_if

nat on $wan_if proto tcp from any to $INT_CARP_IP port { 21,80,25,143,3306 } -> $int_carp_if
nat on $wan_if proto tcp from $INT_CARP_IP to any port { 21,80,25,143,3306 } -> $int_carp_if

### Internet sharing for CLUSTERNODES
nat on $wan_if proto tcp from $LAN_NODES to any -> $wan_if
nat on $wan_if proto udp from $LAN_NODES to any -> $wan_if

## No NAT for others in LAN
no nat on $wan_if proto tcp from $lan_if:network to any

################### FILTER ################################
pass in quick on carp1
pass on $sync_if proto pfsync
pass in quick on $lan_if proto tcp to port 22
pass in quick on $lan_if from $REAL_SERVER
pass in quick on $lan_if to $REAL_SERVER
pass in quick on $lan_if proto tcp from $LAN_NODES to $lan_if
pass in quick on $lan_if proto udp from $LAN_NODES to $lan_if

### Prevent SYNFLOOD Attack
pass in quick on $wan_if proto tcp from any to $EXT_CARP_IP port {80,443,21,3306,25,143} flags S/SA synproxy state
pass in quick on $lan_if all allow-opts

###### pass out quick on $lan_if proto carp
pass out quick on carp1
pass out on $lan_if proto carp keep state
pass out on $lan_if all
pass out on $lan_if proto tcp from $INT_CARP_IP to any port {21,53,80,443}

## Possible Spoofing
block in quick from urpf-failed label uRPF
pass in quick from $lan_if
pass in quick on $lan_if from $lan_if
block in all

################# ATTACK Preventions ##################
set skip on lo0
antispoof for $ext_carp_if inet
antispoof for $lan_if inet

######################################################################
# block in on $lan_if from any to any

## table { 192.168.0.0/24 }
## table const { 192.168.0.0/16, 172.16.0.0/12,10.0.0.0/8 }

## table persist

## block in on $int_if from { , } to any
## table { 192.168.0.24/8 !192.0.0.0/24 }
## pass in on $int_if from to any
## block in on $int_if from to any