2023年7月19日发(作者：)

Keepalived+LVS+Nginx热双机互备负载均衡

作者：范征元

mail:fzhy163@

体系架构：

在Keepalived + Nginx高可靠负载均衡架构中，keepalived负责实现High-availability (HA) 功能控制前端机VIP(虚拟网络地址)，当有设备发生故障时，热备服务器可以瞬间将VIP自动切换过来，实际运行中体验只有2秒钟切换时间，DNS服务可以负责前端VIP的负载均衡。

简单原理：

LVS_DR_MASTER、LVS_DR_BACKUP两台服务器均通过keepalived软件把eth0网卡绑上一个虚拟IP（VIP1）地址200.200.200.200，此VIP1当前由谁承载着服务就绑定在谁的eth0上，当LVS_DR_MASTER发生故障时，LVS_DR_BACKUP会通过/etc/keepalived/文件中设置的心跳时间advert_int 1检查，无法获取LVS_DR_MASTER正常状态后瞬间切换到LVS_DR_BACKUP上来实现热双机负载均衡，当LVS_DR_MASTER恢复后keepalived会通过priority 参数判断优先权将虚拟VIP1地址200.200.200.200重新绑定给LVS_DR_MASTER的eth0网卡；同理，虚拟IP（VIP2）地址200.200.200.199把先前的LVS_DR_BACKUP当做了主服务器，LVS_DR_MASTER当成了辅服务器，来实现热双机互备负载均衡。

硬件环境： vmware 7.1.2网卡Host-only模式接入

系统软件环境：

两台机器安装：centos4.3 + lnmp (linux version Red Hat 3.4.6-11) 32 位，分别命名为:

LVS_DR_MASTER，LVS_DR_BACKUP；默认LVS_DR_MASTER作主机，LVS_DR_BACKUP作热备。

Direct Routing：直接路由模式

CLIENT：

地址：200.200.200.2

子网掩码：255.255.255.0

默认网关：200.200.200.1

LVS_DR_MASTER：

ip：200.200.200.10（主服务器）

子网掩码：255.255.255.0

默认网关：200.200.200.1

vip1（LVS_DR_MASTER）：200.200.200.200

vip2（LVS_DR_BACKUP）：200.200.200.199

LVS_DR_BACKUP：

ip：200.200.200.11（备服务器）

子网掩码：255.255.255.0

默认网关：200.200.200.1

vip1（LVS_DR_BACKUP）：200.200.200.200

vip2（LVS_DR_MASTER）：200.200.200.199

准备工作：

分别在LVS_DR_MASTER、LVS_DR_BACKUP两台服务器安装nginx

分别在LVS_DR_MASTER、LVS_DR_BACKUP两台服务器创建网页显示文件

LVS_DR_MASTER：

echo "LVS_DR_MASTER 200.200.200.10" > /home/wwwroot/

LVS_DR_BACKUP：

echo "LVS_DR_BACKUP 200.200.200.11" > /home/wwwroot/

同步服务器的系统时间

# ntpdate 8 Dec 11:56:59 ntpdate[10531]: adjust time server 192.43.244.18 offset 0.009136 sec

查看当前kernels环境

# uname –a

Linux omain #1 Tue Oct 19 16:47:55 EDT 2010 i686 i686 i386 GNU/Linux

软连接当前kernels目录到/usr/src/linux ，否则无法支持IPVS

# ln -s /usr/src/kernels/-i686/ /usr/src/linux

下载：

# wget /software/kernel-2.6/

# wget /software/

安装ipvsadm（lvs管理查看工具）：

# tar zxvf

# cd ipvsadm-1.24

# make

# make install

查看ipvsadm是否安装正确

# watch ipvsadm –ln

安装keepalived（HA）：

# tar zxvf

# cd keepalived-1.1.20

注意项

Centos5.0以下需要修改以下configure才能通过。

# vi /usr/src/linux/include/linux/types.h

/*

typedef __u16 __bitwise __sum16;

typedef __u32 __bitwise __wsum;

*/

# ./configure --prefix=/usr/local/keepalived

看到提示如下状态

Keepalived configuration

------------------------

Keepalived version : 1.1.20

Compiler : gcc

Compiler flags : -g -O2

Extra Lib : -lpopt -lssl -lcrypto

Use IPVS Framework : Yes

IPVS sync daemon support : Yes

Use VRRP Framework : Yes

Use Debug flags : No # make

# make install

# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

# mkdir /etc/keepalived

# cp /usr/local/keepalived/etc/keepalived/ /etc/keepalived/

# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

编写LVS_DR_MASTER keepalived配置文件

# vi /usr/local/keepalived/etc/keepalived/

global_defs{

notification_email{

fzhy163@

}

notification_email_from fzhy163@

smtp_server

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_script chk_http {

script "/usr/local/keepalived/nginx_" //监控脚本

interval 9 //监控时间，以秒为单位

weight 1 //权重值，数值越大权重越高

}

vrrp_instance VI_1{

state MASTER //实例状态state ，只有MASTER,BACKUP两种必需大写单词，MASTER为工作状态，BACKUP为备用状态，当MASTER失效时，BACKUP会自动把状态BACKUP变成MASTER;当MASTER系统恢复时，BACKUP从MASTER恢复到BACKUP状态

interface eth0

priority 100

advert_int 1

authentication{

}

track_script {

chk_http //执行监控的服务

}

virtual_ipaddress{

200.200.200.200 //1_vip， 定义虚拟IP，可以有多个，分行写入

}

auth_type PASS //验证类型主要有PASS、AH 两种，通常使用的类型为PASS，据说AH 使用时有问题

auth_pass 123456 //验证密码为明文，主从服务器要一致

//监控网卡

//虚拟路由编号，主辅要一致 virtual_router_id 50

//LVS负载均衡标识，在一个网络内，它是唯一标识

//权重值MASTER一定要大于BACKUP

//检查间隔时间，单位为1秒 }

vrrp_instance VI_2{

}

state BACKUP

interface eth0

virtual_router_id 49

proiority 99

advert_int 1

authentication{

}

virtual_ipaddress{

}

200.200.200.199 //2_vip

auth_type PASS

auth_pass 1111

编写LVS_DR_BACKUP keepalived配置文件

global_defs{

notification_email{

fzhy163@

}

notification_email_from fzhy163@

smtp_server

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_script chk_http {

script "/usr/local/keepalived/nginx_"

interval 9

weight 1

}

vrrp_instance VI_1{

state BACKUP

interface eth0

virtual_router_id 50

priority 99

advert_int 1

authentication{

}

virtual_ipaddress{

200.200.200.200 //1_vip

auth_type PASS

auth_pass 123456 }

}

vrrp_instance VI_2{

state MASTER

interface eth0

virtual_router_id 49

proiority 100

advert_int 1

authentication{

}

track_script {

chk_http

}

}

virtual_ipaddress{

200.200.200.199 //2_vip

}

auth_type PASS

auth_pass 1111

服务层检查脚本补充

当keepalived发现当LVS_DR_MASTER服务器nginx无法正常使用时， keepalived是无法检测到服务层故障来切换到LVS_DR_BACKUP服务器，我认为如果nginx服务挂掉了，我觉得就很难再起来，所以我把keepalived 也杀掉了，再有，类似nagios服务监控软件也会给你报警。

# vi /usr/local/keepalived/nginx_

#!/bin/sh

A=`ps -C nginx --no-header |wc -l` //查看是否有 nginx进程数并把值赋给变量A

if [ $A -eq 0 ];then //如果没有进程则值得为零

/usr/local/nginx/sbin/nginx

sleep 5

if [ `ps -C nginx --no-header |wc -l` -eq 0 ];then

killall keepalived //结束 keepalived 进程

fi

fi

提示：要给/usr/local/keepalived/nginx_加入可执行权限

# chmod 744 /usr/local/keepalived/nginx_

注意：

一定要在主服务器中相应LVS_DR_MASTER地方加入内容（具体见先前配置）

1.

vrrp_script chk_http {

script "/usr/local/keepalived/nginx_" //监控脚本 interval 9 //监控时间

weight 1 //权重值，数值越大权重越高

}

2.

track_script {

chk_http //执行监控的服务

}

启动keepalived服务

# service keepalived start

Starting keepalived: [ OK ]

建议使用：

# /usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/

-D 显示在日志记录

-f 指定配置文件目录

确认keepalived已启动

# ps -aux|grep keepalived

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ

root 5227 0.0 0.2 4896 696 ? Ss 18:15 0:00 keepalived -D

root 5228 0.0 0.4 4948 1276 ? S 18:15 0:00 keepalived -D

root 5229 0.0 0.4 4948 1036 ? S 18:15 0:00 keepalived -D

root 5654 0.0 0.2 3820 664 pts/1 S+ 18:19 0:00 grep keepalived

设置keepalived随服务器一起启动

# echo “/usr/local/keepalived/sbin/keepalived -D -f /etc/keepalived/” >> /etc/rc.d/

查看LVS_DR_MASTER上eth0接口在启动keepalived前后变化

启动keepalived之前，查看LVS_DR_MASTER主服务器 keepalived虚拟IP绑定状况，ifconfig无法查看到

# ip a

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:22:3d:17 brd ff:ff:ff:ff:ff:ff inet 200.200.200.10/24 brd 200.200.200.255 scope global eth0

inet6 fe80::20c:29ff:fe22:3d17/64 scope link

valid_lft forever preferred_lft forever

3: sit0: mtu 1480 qdisc noop

link/sit 0.0.0.0 brd 0.0.0.0

启动keepalived之后

# ip a

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:22:3d:17 brd ff:ff:ff:ff:ff:ff

inet 200.200.200.10/24 brd 200.200.200.255 scope global eth0

inet 200.200.200.200/32 scope global eth0

inet6 fe80::20c:29ff:fe22:3d17/64 scope link

valid_lft forever preferred_lft forever

3: sit0: mtu 1480 qdisc noop

link/sit 0.0.0.0 brd 0.0.0.0

可以看到主服务器的200.200.200.200虚拟 IP 已经挂接在网卡 eth0 上。

可以正常查看到keepalived启动日志

# /tail -100 /var/log/messages

Dec 8 20:54:42 localhost Keepalived: Starting Keepalived v1.1.20 (12/02,2010)

Dec 8 20:54:42 localhost Keepalived: Starting Healthcheck child process, pid=3894

Dec 8 20:54:42 localhost Keepalived: Starting VRRP child process, pid=3896

Dec 8 20:54:42 localhost Keepalived_vrrp: Netlink reflector reports IP 200.200.200.10 added

Dec 8 20:54:42 localhost Keepalived_vrrp: Registering Kernel netlink reflector

Dec 8 20:54:42 localhost Keepalived_vrrp: Registering Kernel netlink command channel

Dec 8 20:54:42 localhost Keepalived_vrrp: Registering gratutious ARP shared channel

Dec 8 20:54:43 localhost kernel: IPVS: Registered protocols (TCP, UDP, AH, ESP)

Dec 8 20:54:43 localhost kernel: IPVS: Connection hash table configured (size=4096, memory=32Kbytes)

Dec 8 20:54:44 localhost kernel: IPVS: ipvs loaded.

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Netlink reflector reports IP 200.200.200.10 added

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Registering Kernel netlink reflector

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Registering Kernel netlink command channel

Dec 8 20:54:44 localhost Keepalived_vrrp: Opening file '/etc/keepalived/'.

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Opening file '/etc/keepalived/'.

Dec 8 20:54:44 localhost Keepalived_vrrp: Configuration is using : 38035 Bytes

Dec 8 20:54:44 localhost Keepalived_vrrp: Using LinkWatch kernel Dec 8 20:54:44 localhost Keepalived_vrrp: VRRP_Instance(VI_2) Entering BACKUP STATE

Dec 8 20:54:44 localhost Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Configuration is using : 4811 Bytes

Dec 8 20:54:44 localhost Keepalived_healthcheckers: Using LinkWatch kernel

Dec 8 20:54:45 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Dec 8 20:54:45 localhost udevd[1485]: udev done!

Dec 8 20:54:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Dec 8 20:54:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Dec 8 20:54:46 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 200.200.200.200

Dec 8 20:54:46 localhost Keepalived_vrrp: Netlink reflector reports IP 200.200.200.200 added

Dec 8 20:54:46 localhost Keepalived_healthcheckers: Netlink reflector reports IP 200.200.200.200 added

Dec 8 20:54:48 localhost Keepalived_vrrp: VRRP_Instance(VI_2) Transition to MASTER STATE

Dec 8 20:54:49 localhost Keepalived_vrrp: VRRP_Instance(VI_2) Entering MASTER STATE

Dec 8 20:54:49 localhost Keepalived_vrrp: VRRP_Instance(VI_2) setting protocol VIPs.

Dec 8 20:54:49 localhost Keepalived_vrrp: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 200.200.200.199

Dec 8 20:54:49 localhost Keepalived_vrrp: Netlink reflector reports IP 200.200.200.199 added

Dec 8 20:54:49 localhost Keepalived_healthcheckers: Netlink reflector reports IP 200.200.200.199 added

Dec 8 20:54:51 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 200.200.200.200

Dec 8 20:54:53 localhost udevd[1485]: udev done!

Dec 8 20:54:54 localhost Keepalived_vrrp: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 200.200.200.199

日志显示，此时地址为200.200.200.200的VI_1在当前服务器上正常运行

查看LVS_DR_BACKUP辅服务器开启keepalived虚拟IP绑定状况

# ip a

1: lo: mtu 16436 qdisc noqueue

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 brd 127.255.255.255 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: mtu 1500 qdisc pfifo_fast qlen 1000

link/ether 00:0c:29:f0:95:68 brd ff:ff:ff:ff:ff:ff

inet 200.200.200.11/24 brd 200.200.200.255 scope global eth0

inet 200.200.200.199/32 scope global eth0

inet6 fe80::20c:29ff:fef0:9568/64 scope link

valid_lft forever preferred_lft forever

3: sit0: mtu 1480 qdisc noop

link/sit 0.0.0.0 brd 0.0.0.0

可以看到在LVS_DR_MASTER正常运行的时候200.200.200.200虚拟IP不会挂接在辅服务器网卡 eth0 上。

验证测试

1. 当LVS_DR_MASTER、LVS_DR_BACKUP服务器nginx均正常工作时， CLIENT通过浏览器访问

200.200.200.10

LVS_DR_MASTER 200.200.200.10

200.200.200.11

LVS_DR_BACKUP 200.200.200.11

200.200.200.200

LVS_DR_MASTER 200.200.200.10

200.200.200.199

LVS_DR_BACKUP 200.200.200.11

2. 当LVS_DR_MASTER服务器nginx出现故障，LVS_DR_BACKUP正常工作时，CLIENT通过浏览器访问

200.200.200.10

无法访问

200.200.200.11

LVS_DR_BACKUP 200.200.200.11

200.200.200.200

LVS_DR_BACKUP 200.200.200.11

200.200.200.199

LVS_DR_BACKUP 200.200.200.11

3. 当LVS_DR_MASTER正常工作时，LVS_DR_BACKUP服务器nginx出现故障，CLIENT通过浏览器访问

200.200.200.10

LVS_DR_MASTER 200.200.200.10

200.200.200.11

无法访问

200.200.200.200

LVS_DR_BACKUP 200.200.200.10

200.200.200.199

LVS_DR_BACKUP 200.200.200.10

4. 当LVS_DR_MASTER、LVS_DR_BACKUP服务器nginx均出现故障时，CLIENT通过浏览器访问

200.200.200.10 无法访问

200.200.200.11

无法访问

200.200.200.200

无法访问

200.200.200.199

无法访问