HOWTO check step-by-step a Layer 4 LVS-NAT setup for problems


Q.1 Can the real server ping client?

	rs# ping -n client

A.1 Yes => good
A.2 No => bad

	Some settings for the director:

	Linux 2.2/2.4:
	ipchains -A forward -s RIP -j MASQ

	Linux 2.4:
	iptables -t nat -A POSTROUTING -s RIP -j MASQUERADE

Q.2 Traceroute to client goes through LVS box and reaches the client?

	traceroute -n -s RIP CLIENT_IP

A.1 Yes => good
A.2 No => bad

	same ipchains command as in Q.1

	For client and server on same physical media use these
	in the director:

	echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
	echo 0 > /proc/sys/net/ipv4/conf/<DEV>/send_redirects


Q.3 Is the traffic forwarded from the LVS box, in both directions?

	For all interfaces on director:
	tcpdump -ln host CLIENT_IP

	The right sequence, i.e. the IP addresses and ports on each
	step (the reversed for the in->out direction are not shown):

	CLIENT
	   | CIP:CPORT -> VIP:VPORT
	   |		||
	   |		\/
 out	   | CIP:CPORT -> VIP:VPORT
 ||	LVS box
 \/	   | CIP:CPORT -> RIP:RPORT
 in	   |		||
	   |		\/
	   | CIP:CPORT -> RIP:RPORT
	   +
	REAL SERVER

A.1 Yes, in both directions => good (for Layer 4, probably not for L7)
A.2 The packets from the real server are dropped => bad:

	- rp_filter protection on the incoming interface, probably
	hit from local client
	- firewall rules drop the replies

A.3 The packets from the real servers leave the director unchanged

	- missing -j MASQ ipchains rule in the LVS box

	For client and server on same physical media:

	The packets simply does not reach the director. The real
	server is ICMP redirected to the client. In director:

	echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
	echo 0 > /proc/sys/net/ipv4/conf/<DEV>/send_redirects

A.4 All packets from the client are dropped

	- the requests are received on wrong interface with rp_filter
	protection
	- firewall rules drop the requests

A.5 The client connections are refused or are served from service
in the LVS box

	- client and LVS are on same host => not valid
	- the packets are not marked from the firewall and don't hit
	firewall mark based virtual service

Q.4 Is the traffic replied from the real server?

	For the outgoing interface on real server:

	tcpdump -ln host CLIENT_IP

A.1 Yes, SYN+ACK => good
A.2 TCP RST => bad, No listening real service
A.3 ICMP message => bad, Blocked from Firewall/No listening service
A.4 The same request packet leaves the real server => missing accept
	rules or RIP is not defined
A.5 No reply => real server problem:

	- the rp_filter protection drops the packets
	- the firewall drops the request packets
	- the firewall drops the replies

A.6 Replies goes through another device or don't go to the LVS
box =? bad

	- the route to the client is direct and so don't pass the LVS
	box, for example:

		- client on the LAN
		- client and real server on same host

	- wrong route to the LVS box is used => use another

	Check the route:

	rs# ip route get CLIENT_IP from RIP


The result: start the following tests

rs# tcpdump -ln host CIP
rs# traceroute -n -s RIP CIP
lvs# tcpdump -ln host CIP
client# tcpdump -ln host CIP


For more deep problems use tcpdump -len, i.e. sometimes the link layer
addresses help a bit.


For FTP:

	LVS-NAT in Linux 2.2 requires:

	- modprobe ip_masq_ftp (before 2.2.19)
	- modprobe ip_masq_ftp in_ports=21 (2.2.19+)

	LVS-NAT in Linux 2.4 requires:

	- ip_vs_ftp

	LVS-DR/TUN require persistent flag


	FTP reports with debug mode enabled are useful:

	# ftp
	ftp> debug
	ftp> open my.virtual.ftp.service
	ftp> ...
	ftp> dir
	ftp> passive
	ftp> dir

	There are reports that sometimes the status strings reported
	from the FTP real servers are not matched with the string
	constants encoded in the kernel FTP support. For example,
	Linux 2.2.19 matches
	"227 Entering Passive Mode (xxx,xxx,xxx,xxx,ppp,ppp)"


Julian Anastasov