• use 2 distinct NICs
• ensure connection tracking is off !!!
• add to haproxy.cfg

option http-server-close

• add to /etc/sysctl.conf

net.ipv4.ip_local_port_range = 1024 65000

• If still not good, try the following (or buy a HW load balancer)

net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 10000
net.ipv4.tcp_max_syn_backlog = 10000
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_tw_buckets = 400000  <------- not sure
net.ipv4.tcp_max_orphans = 60000      <------- not sure

Learn additional tips and trinks on http://blog.exceliance.fr/category/haproxy-2/