揭秘企業(yè)級(jí)web負(fù)載均衡完美架構(gòu)（圖）
原創(chuàng)【51CTO.com獨(dú)家特稿】相信很多朋友對(duì)企業(yè)級(jí)的負(fù)載均衡高可用實(shí)例非常感興趣，此篇文章根據(jù)成熟的線上環(huán)境而寫，旨在幫助大家迅速架構(gòu)一個(gè)企業(yè)級(jí)的負(fù)載均衡高可用的web環(huán)境。
此系統(tǒng)架構(gòu)僅映射內(nèi)網(wǎng)VIP的80及443端口于外網(wǎng)的Juniper防火墻下，其它端口均關(guān)閉，內(nèi)網(wǎng)所有機(jī)器均關(guān)閉iptables及ipfw防火墻；外網(wǎng)DNS指向即通過(guò)Juniper映射出來(lái)的外網(wǎng)地址，而此映射的地址對(duì)映的其實(shí)是內(nèi)網(wǎng)VIP地址。這里說(shuō)下端口的問(wèn)題，有的朋友可能會(huì)很疑惑，這樣映射端口行不？通過(guò)項(xiàng)目實(shí)踐得知，這樣完全是可行的，php-cgi需要的9000端口及MySQL的3306端口均可走內(nèi)網(wǎng)，完全不影響業(yè)務(wù)系統(tǒng)的運(yùn)行。
另外，我維護(hù)的電子商務(wù)網(wǎng)站并發(fā)大約在1000左右，此時(shí),Nginx+Apache集群運(yùn)行得非常穩(wěn)定，尤其是apache，并沒(méi)有想象中那般弱；其實(shí)，在內(nèi)存足夠(>=8G)的情況，測(cè)試時(shí)不連數(shù)據(jù)庫(kù)的話，單臺(tái)apache+php5能頂?shù)米?000并發(fā)，而且相當(dāng)穩(wěn)定。在網(wǎng)站升級(jí)架構(gòu)方面，我不贊成全面淘汰生級(jí)，錦上添花式的升級(jí)會(huì)更好。
第一部分：Nginx+Keepalived的說(shuō)明及環(huán)境說(shuō)明
喜歡看我博客或文章的朋友都知道，我一直主力推崇Nginx+Keepalived作web的負(fù)載均衡高可用架構(gòu)，并積極將其用于項(xiàng)目方案中；Nginx負(fù)載均衡作服務(wù)器遇到的故障一般有①服務(wù)器網(wǎng)線松動(dòng)等網(wǎng)絡(luò)故障；②服務(wù)器硬件故障從而crash；③nginx服務(wù)死掉；遇到前二者情況，keeaplived是能起到HA的作用的；然而遇到③種情況就沒(méi)有辦法了，但可以通過(guò)shell監(jiān)控解決這問(wèn)題，從而實(shí)現(xiàn)真正意義上的負(fù)載均衡高可用。此篇的最新更新時(shí)間為2010年6月25號(hào)，下面將其安裝步驟詳細(xì)說(shuō)明下：
環(huán)境：
- centos5.3(64位)、nginx-0.7.51、keepalived-1.1.15
- 主nginx負(fù)載均衡器：192.168.0.154
- 輔nginx負(fù)載均衡器：192.168.9.155
- vip：192.168.0.188
#p#
第二部分：分別安裝Nginx負(fù)載均衡器及相關(guān)配置腳本
先安裝Nginx負(fù)載均衡器，nginx負(fù)載的配置就用一般的模板來(lái)配置了
- #添加運(yùn)行nginx的用戶和組www
- groupadd www
- useradd -g www www
- wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.8.tar.gz
- tar zxvf pcre-7.8.tar.gz
- cd pcre-7.8/
- ./configure
- make && make install
- wget http://sysoev.ru/nginx/nginx-0.7.51.tar.gz
- tar zxvf nginx-0.7.51.tar.gz
- cd nginx-0.7.51/
- ./configure --user=www --group=www --prefix=/usr/local/webserver/nginx --with-http_stub_status_module --with-http_ssl_module
- make && make install
配置nginx負(fù)載均衡器的配置文件vim /usr/local/nginx/conf/nginx.conf,此篇文章僅僅只是我的某項(xiàng)目的配置文檔，純80轉(zhuǎn)發(fā)；如果對(duì)nginx配置有https要求的可參考張宴的相關(guān)文章。
- user www www;
- worker_processes 8;
- pid /usr/local/nginx/logs/nginx.pid;
- worker_rlimit_nofile 65535;
- events
- {
- use epoll;
- worker_connections 65535;
- }
- http{
- include mime.types;
- default_type application/octet-stream;
- server_names_hash_bucket_size 128;
- client_header_buffer_size 32k;
- large_client_header_buffers 4 32k;
- client_max_body_size 8m;
- sendfile on;
- tcp_nopush on;
- keepalive_timeout 60;
- tcp_nodelay on;
- fastcgi_connect_timeout 300;
- fastcgi_send_timeout 300;
- fastcgi_read_timeout 300;
- fastcgi_buffer_size 64k;
- fastcgi_buffers 4 64k;
- fastcgi_busy_buffers_size 128k;
- fastcgi_temp_file_write_size 128k;
- gzip on;
- gzip_min_length 1k;
- gzip_buffers 4 16k;
- gzip_http_version 1.0;
- gzip_comp_level 2;
- gzip_types text/plain application/x-javascript text/css application/xml;
- gzip_vary on;
- upstream backend
- {
- server 192.168.1.102:80;
- server 192.168.1.103:80;
- server 192.168.1.105:80;
- }
- server {
- listen 80;
- server_name www.yuhongchun027.com;
- location / {
- root /var/www ;
- index index.jsp index.htm index.html;
- proxy_redirect off;
- proxy_set_header Host $host;
- proxy_set_header X-Real-IP $remote_addr;
- proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
- proxy_pass http://backend;
- }
- location /nginx {
- access_log on;
- auth_basic "NginxStatus";
- auth_basic_user_file /usr/local/nginx/htpasswd;
- }
- log_format access '$remote_addr - $remote_user [$time_local] "$request" '
- '$status $body_bytes_sent "$http_referer" '
- '"$http_user_agent" $http_x_forwarded_for';
- access_log /var/log/access.log access;
- }
- }
小節(jié)：
第一部分和第二部分講的是如何通過(guò)安裝Nginx來(lái)達(dá)到負(fù)載均衡后端web集群的過(guò)程，Nginx能實(shí)現(xiàn)自動(dòng)切換后端有故障的web服務(wù)器；但Nginx負(fù)載均衡器出了問(wèn)題怎么辦呢，它們之間是如何實(shí)現(xiàn)無(wú)故障轉(zhuǎn)移的呢？#p#
第三部分：安裝Keepalived，讓其分別作web及Nginx的HA
安裝keepalived，并將其做成服務(wù)模式，方便以后調(diào)試。
- wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
- #tar zxvf keepalived-1.1.15.tar.gz
- #cd keepalived-1.1.15
- #./configure --prefix=/usr/local/keepalived
- #make
- #make install
- #cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
- #cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
- #cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
- #mkdir /etc/keepalived
- #cd /etc/keepalived/
- vim keepalived.conf
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- yuhongchun027@163.com
- }
- notification_email_from keepalived@chtopnet.com
- smtp_server 127.0.0.1
- smtp_connect_timeout 30
- router_id LVS_DEVEL
- }
- vrrp_instance VI_1 {
- state MASTER
- interface eth0
- virtual_router_id 51
- mcast_src_ip 192.168.0.154 <==主nginx的IP地址
- priority 100
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass chtopnet
- }
- virtual_ipaddress {
- 192.168.0.188 <==vip地址
- }
- }
- #service keepalived start
我們來(lái)看一下日志：
- [root@ltos ~]# tail /var/log/messages
- Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering new address record for 192.168.0.188 on eth0.
- Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering new address record for 192.168.0.154 on eth0.
- Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering HINFO record with values 'I686'/'LINUX'.
- Oct 6 03:25:23 ltos avahi-daemon[2306]: Withdrawing address record for fe80::20c:29ff:feb9:eeab on eth0.
- Oct 6 03:25:23 ltos avahi-daemon[2306]: Withdrawing address record for 192.168.0.154 on eth0.
- Oct 6 03:25:23 ltos avahi-daemon[2306]: Host name conflict, retrying with <ltos-31>
很顯然vrrp已經(jīng)啟動(dòng)，我們還可以通過(guò)命令來(lái)檢查
- [root@ltos html]# ip addr
- 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
- link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
- inet 127.0.0.1/8 scope host lo
- inet6 ::1/128 scope host
- valid_lft forever preferred_lft forever
- 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
- link/ether 00:0c:29:ba:9b:e7 brd ff:ff:ff:ff:ff:ff
- inet 192.168.0.154/24 brd 192.168.0.255 scope global eth0
- inet 192.168.0.188/32 scope global eth0
- inet6 fe80::20c:29ff:feba:9be7/64 scope link
- valid_lft forever preferred_lft forever
- 3: sit0: <NOARP> mtu 1480 qdisc noop
- link/sit 0.0.0.0 brd 0.0.0.0
說(shuō)明vip已經(jīng)啟動(dòng)，這樣主服務(wù)器就配置好了，輔機(jī)的配置大致一樣，除了配置文件有少部分的變化，下面貼出輔機(jī)的配置文件：
- ! Configuration File for keepalived
- global_defs {
- notification_email {
- yuhongchun027@163.com
- }
- notification_email_from keepalived@chtopnet.com
- smtp_server 127.0.0.1
- smtp_connect_timeout 30
- router_id LVS_DEVEL
- }
- vrrp_instance VI_1 {
- state BACKUP
- interface eth0
- virtual_router_id 51
- mcast_src_ip 192.168.0.155 <==輔nginx的IP的地址
- priority 100
- advert_int 1
- authentication {
- auth_type PASS
- auth_pass chtopnet
- }
- virtual_ipaddress {
- 192.168.0.188
- }
- }
#p#
第四部分：針對(duì)Keepalived的不足，用Nginx_pid.sh來(lái)監(jiān)控nginx進(jìn)程，實(shí)現(xiàn)真正意義上的負(fù)載均衡高可用。
針對(duì)Nginx+Keepalived，編寫nginx監(jiān)控腳本nginx_pid.sh，此腳本思路其實(shí)也很簡(jiǎn)單，即放置在后臺(tái)一直監(jiān)控nginx進(jìn)程；如進(jìn)程消失，嘗試重啟nginx，如是失敗則立即停掉本機(jī)的keepalived服務(wù)，讓另一臺(tái)負(fù)載均衡器接手，此腳本直接從生產(chǎn)環(huán)境下載：
- vim /root/nginx_pid.sh
- #!/bin/bash
- while :
- do
- nginxpid=`ps -C nginx --no-header | wc -l`
- if [ $nginxpid -eq 0 ];then
- /usr/local/nginx/sbin/nginx
- sleep 5
- nginxpid=`ps -C nginx --no-header | wc -l`
- if [ $nginxpid -eq 0 ];then
- /etc/init.d/keepalived stop
- fi
- fi
- sleep 5
- done
然后置于后臺(tái)運(yùn)行 sh /root/nginx_pid.sh &，這種寫法是錯(cuò)誤的，這樣你用root用戶logout后，此進(jìn)程會(huì)消失；正確寫法為nohup/bin/bash /root/nginx_pid.sh &,附帶下注釋:如果你正在運(yùn)行一個(gè)進(jìn)程，而且你覺(jué)得在退出帳戶時(shí)該進(jìn)程還不會(huì)結(jié)束，那么可以使用nohup命令。該命令可以在你退出root帳戶之后繼續(xù)運(yùn)行相應(yīng)的進(jìn)程。nohup就是不掛起的意思( no hang up)，哈哈，差點(diǎn)老馬失蹄了。
后記：
我的線上環(huán)境網(wǎng)絡(luò)非常復(fù)雜，這也是LVS+Keepalived失敗的原因。目前此套架構(gòu)在1000并發(fā)的電子商務(wù)網(wǎng)站非常穩(wěn)定，帶來(lái)的直接影響就是nginx_backup一直處于閑置狀態(tài)。相對(duì)于張宴的雙機(jī)輪詢而言，我感覺(jué)他的可能更加完美，因?yàn)槟壳拔业腘ginx僅僅只做了負(fù)載均衡器，如果以后有機(jī)會(huì)我會(huì)嘗試做負(fù)載均衡器/反向代理加速。
【51CTO.com獨(dú)家特稿，非經(jīng)授權(quán)謝絕轉(zhuǎn)載，合作媒體轉(zhuǎn)載請(qǐng)注明原文出處及作者！】
