?LVS是中國(guó)人發(fā)起的項(xiàng)目，真是意外呀!大家可以看http://www.douzhe.com/linuxtips/1665.html
我是從最初的HA(高可用性)開始的，別人的例子是用VMWARE，可以做試驗(yàn)但不能實(shí)際應(yīng)用，我又沒有光纖卡的Share Storage，于是就選用ISCSI，成功后又發(fā)現(xiàn)ISCSI+EXT3不能用于LVS，倒最后發(fā)現(xiàn)GFS可用，我最終成功配成可實(shí)際應(yīng)用的LVS，前后斷斷續(xù)續(xù)花了四個(gè)月，走了很多彎路。我花了三天時(shí)間寫下這篇文章，希望對(duì)大家有用。
這里要感謝linuxfans.org、linuxsir.com、chinaunix.com以及其它很多網(wǎng)站，很多資料都是從他們的論壇上找到的。參考文檔及下載點(diǎn)
a.http://www.gyrate.org/misc/gfs.txt
b.http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/cluster-suite/index.html
http://www.redhat.com/docs/manuals/csgfs/admin-guide/index.html
c.ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHGFS/SRPMS
d.http://distro.ibiblio.org/pub/linux/distributions/caoslinux/centos/3.1/contrib/i386/RPMS/
???????????????? LVS結(jié)構(gòu)圖：??????????????eth0=10.3.1.101????????????eth0:1=10.3.1.254????????????????Load Balance?????????????????? Router??????????????eth1=192.168.1.71????????????eth1:1=192.168.1.1??????????????????|???????? |??????????????????|???????? |????????????????Real1??????Real2?????? eth0=192.168.1.68??eth0=192.168.1.67????????????(eth0 gateway=192.168.1.1)?????? eth1=192.168.0.1---eth1=192.168.0.2??????????????????(雙機(jī)互聯(lián)線)?????????????????????? |?????????????????????? |??????????????????????GFS???????????????????? ISCSI????????????????Share storage??????????????eth0=192.168.1.124
1.Setup ISCSI Server
Server: PIII 1.4,512M, Dell 1650,Redhat 9,IP=192.168.1.124
從http://iscsitarget.sourceforge.net/下載ISCSI TARGET的Source code
(http://sourceforge.net/project/showfiles.php?group_id=108475&package_id=117141)
我選了iscsitarget-0.3.8.tar.gz,要求kernel 2.4.29
從kernel.org下載kernel 2.4.29,解開編譯重啟后編譯安裝iscsitarget-0.3.8：
#make KERNELSRC=/usr/src/linux-2.4.29
#make KERNELSRC=/usr/src/linux-2.4.29 install
#cp ietd.conf /etc
#vi /etc/ietd.conf

# Example iscsi target configuration## Everything until the first target definition belongs# to the global configuration.# Right now this is only the user configuration used# during discovery sessions:# Users, who can access this target# (no users means anyone can access the target)User iscsiuser 1234567890abcTarget iqn.2005-04.com.my:storage.disk2.sys1.iraw1????????User iscsiuser 1234567890abc????????Lun 0 /dev/sda5 fileio????????Alias iraw1Target iqn.2005-04.com.my:storage.disk2.sys1.iraw2????????User iscsiuser 1234567890abc????????Lun 1 /dev/sda6 fileio????????Alias iraw2Target iqn.2005-04.com.my:storage.disk2.sys2.idisk????????User iscsiuser 1234567890abc????????Lun 2 /dev/sda3 fileio????????Alias idiskTarget iqn.2005-04.com.my:storage.disk2.sys2.icca????????User iscsiuser 1234567890abc????????Lun 3 /dev/sda7 fileio????????Alias icca

說明：password 長(zhǎng)度必須不小于12個(gè)字符, Alias是別名, 不知為何這個(gè)別名在Client端顯示不出來.
分區(qū)：我只有一個(gè)SCSI盤，所以：

/dev/sda3: Share storage,容量越大越好/dev/sda5: raw1, 建Cluster要的rawdevice, 我給了900M/dev/sda6: raw2, 建Cluster要的rawdevice, 我給了900M/dev/sda7: cca, 建GFS要的,我給了64M(/dev/sda4是Extended分區(qū),在其中建了sda5,6,7)

#Reboot，用service iscsi-target start啟ISCSI server(我覺得比建議的好，可以
用service iscsi-target status看狀態(tài))
2.Setup ISCSI Client(on two real server)
Server: PIII 1.4,512M, Dell 1650,Redhat AS3U4(用AS3U5更好),2.4.21-27.EL
#vi /etc/iscsi.conf

DiscoveryAddress=192.168.1.124OutgoingUsername=iscsiuserOutgoingPassword=1234567890abcUsername=iscsiuserPassword=1234567890abcLoginTimeout=15IncomingUsername=iscsiuserIncomingPassword=1234567890abcSendAsyncTest=yes

#service iscsi restart
#iscsi-ls -l
..., 精簡(jiǎn)如下:
/dev/sdb:iraw2
/dev/sdc:iraw1
/dev/sdd:idisk
/dev/sde:icca
注意: 在real server中ISCSI device的順序很重要，兩個(gè)real server中一定要一樣，如不一樣
　　　就改ISCSI Server中的設(shè)置，多試幾次
3.Install Redhat Cluster suite
先下載Cluster Suite的ISO, AS3的我是從ChinaUnix.net找到的下載點(diǎn), 安裝clumanager和
redhat-config-cluster。沒有Cluster Suite的ISO也沒關(guān)系，從
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/3ES/en/RHCS/SRPMS/下載
clumanager-1.2.xx.src.rpm,redhat-config-cluster-1.0.x.src.rpm,編譯后安裝，應(yīng)該更好:
#rpm -Uvh clumanager-1.2.26.1-1.src.rpm
#rpmbuild -bs /usr/src/redhat/SPECS/clumanager.spec
#rpmbuild --rebuild --target i686 /usr/src/redhat/SRPMS/clumanager-1.2.26.1-1.src.rpm
還有redhat-config-cluster-1.0.x.src.rpm，也裝好
4.Setup Cluster as HA module
詳細(xì)步驟我就不寫了，網(wǎng)上有很多文章，我也是看了別人的文章學(xué)會(huì)的，不過人家是用VMWARE，
而我是用真的機(jī)子+ISCSI，raw device就是/dev/sdb,/dev/sdc, 然后就
mount /dev/sdd /u01, mkfs.ext3 /u01 ......
設(shè)好后會(huì)發(fā)現(xiàn)ISCSI有問題：同時(shí)只能有一個(gè)Client聯(lián)接寫盤，如果
兩個(gè)Client同時(shí)聯(lián)ISCSI的Share Storge,一個(gè)Client寫，另一個(gè)Client是看不到的，而且此時(shí)文
件系統(tǒng)已經(jīng)破壞了，Client重聯(lián)ISCSI時(shí)會(huì)發(fā)現(xiàn)文件是壞的，用fsck也修復(fù)不了。
ISCSI真的是雞肋嗎？
　NO！從GOOGLE上我終于查到ISCSI只有用Cluster File System才能真正用于Share Storage!
而Redhat買下的GFS就是一個(gè)!
5.Setup GFS on ISCSI
GFS只有Fedora Core4才自帶了，而GFS又一定要用到Cluster Suite產(chǎn)生的/etc/cluster.xml文件，
我沒見FC4有Cluster Suite,真不知Redhat給FC4帶GFS干嘛，饞人嗎？
好，閑話少說，下載：c處的GFS-6.0.2.20-2.src.rpm，按a處的gfs.txt編譯安裝，不過關(guān)于
cluster.ccs,fence.ccs,nodes.ccs的設(shè)置沒說，看b的文檔，我總算弄出來了，都存在
/root/cluster下，存在別的地方也行，不過我不知道有沒有錯(cuò)，我沒有光纖卡，文檔又沒講ISCSI
的例子，不過GFS能啟動(dòng)的。
#cat cluster.ccs

cluster {????????name = "Cluster_1"????????lock_gulm {????????????servers = ["cluster1", "cluster2"]????????????heartbeat_rate = 0.9????????????allowed_misses = 10????????}}

注：name就是Cluster Suite設(shè)置的Cluster name, servers就是Cluster member的Hostname，別忘
了加進(jìn)/etc/hosts;allowed_misses我開始設(shè)為1，結(jié)果跑二天GFS就會(huì)死掉，改為10就沒死過了。
#cat fence.ccs

fence_devices{????????admin {??????????????agent = "fence_manual"????????}}

#cat nodes.ccs

nodes {?? cluster1 {??????ip_interfaces {???????? hsi0 = "192.168.0.1"??????}??????fence {???????? human {????????????admin {?????????????? ipaddr = "192.168.0.1"????????????}???????? }??????}?? }?? cluster2 {??????ip_interfaces {???????? hsi0 = "192.168.0.2"??????}??????fence {???????? human {????????????admin {?????????????? ipaddr = "192.168.0.2"????????????}???????? }??????}?? }}

注：ip就是心跳線的ip
這三個(gè)文件建在/root/cluster下，先建立Cluster Configuration System：
a.#vi /etc/gfs/pool0.cfg
poolname pool0
minor 1 subpools 1
subpool 0 8 1 gfs_data
pooldevice 0 0 /dev/sde1
b.#pool_assemble -a pool0
c.#ccs_tool create /root/cluster /dev/pool/pool0
d.#vi /etc/sysconfig/gfs
CCS_ARCHIVE="/dev/pool/pool0"
再Creating a Pool Volume，就是我們要的共享磁盤啦，
a.#vi /etc/gfs/pool1.cfg
poolname pool1
minor 2 subpools 1
subpool 0 128 1 gfs_data
pooldevice 0 0 /dev/sdd1
b.#pool_assemble -a pool1
c.#gfs_mkfs -p lock_gulm -t Cluster_1:gfs1 -j 8 /dev/pool/pool1
d.#mount -t gfs -o noatime /dev/pool/pool1 /u01
下面是個(gè)GFS的啟動(dòng)腳本，注意real1和real2必須同時(shí)啟動(dòng)lock_gulmd進(jìn)程，第一臺(tái)lock_gulmd
會(huì)成為Server并等Client的lock_gulmd，幾十秒后沒有響應(yīng)會(huì)fail，GFS啟動(dòng)失敗。Redhat建議
GFS盤不要寫進(jìn)/etc/fstab。
#cat gfstart.sh

#!/bin/shdepmod -amodprobe poolmodprobe lock_gulmmodprobe gfssleep 5service iscsi startsleep 20service rawdevices restartpool_assemble -a pool0pool_assemble -a pool1service ccsd startservice lock_gulmd startmount -t gfs /dev/pool/pool1 /s02 -o noatimeservice gfs status
6. Setup Linux LVS
LVS是章文嵩博士發(fā)起和領(lǐng)導(dǎo)的優(yōu)秀的集群解決方案，許多商業(yè)的集群產(chǎn)品，比如RedHat的Piranha，Turbolinux公司的Turbo Cluster等，都是基于LVS的核心代碼的。
我的系統(tǒng)是Redhat AS3U4，就用Piranha了。從rhel-3-u5-rhcs-i386.iso安裝piranha-0.7.10-2.i386.rpm,ipvsadm-1.21-9.ipvs108.i386.rpm (http://distro.ibiblio.org/pub/linux/distributions/caoslinux/centos/3.1/contrib/i386/RPMS/) 裝完后service httpd start & service piranha-gui start，就可以從http://xx.xx.xx.xx:3636管理或設(shè)置了，當(dāng)然了，手工改/etc/sysconfig/ha/lvs.cf也一樣。
#cat /etc/sysconfig/ha/lvs.cf

serial_no = 80primary = 10.3.1.101service = lvsrsh_command = sshbackup_active = 0backup = 0.0.0.0heartbeat = 1heartbeat_port = 1050keepalive = 6deadtime = 18network = natnat_router = 192.168.1.1 eth1:1nat_nmask = 255.255.255.0reservation_conflict_action = preemptdebug_level = NONEvirtual lvs1 {???? active = 1???? address = 10.3.1.254 eth0:1???? vip_nmask = 255.255.255.0???? fwmark = 100???? port = 80???? persistent = 60???? pmask = 255.255.255.255???? send = "GET / HTTP/1.0\r\n\r\n"???? expect = "HTTP"???? load_monitor = ruptime???? scheduler = wlc???? protocol = tcp???? timeout = 6???? reentry = 15???? quiesce_server = 1???? server Real1 {???????? address = 192.168.1.68???????? active = 1???????? weight = 1???? }???? server Real2 {???????? address = 192.168.1.67???????? active = 1???????? weight = 1???? }}virtual lvs2 {???? active = 1???? address = 10.3.1.254 eth0:1???? vip_nmask = 255.255.255.0???? port = 21???? send = "\n"???? use_regex = 0???? load_monitor = ruptime???? scheduler = wlc???? protocol = tcp???? timeout = 6???? reentry = 15???? quiesce_server = 0???? server ftp1 {???????? address = 192.168.1.68???????? active = 1???????? weight = 1???? }???? server ftp2 {???????? address = 192.168.1.67???????? active = 1???????? weight = 1???? }}

設(shè)置完后service pulse start, 別忘了把相關(guān)的client加進(jìn)/etc/hosts
#iptables -t mangle -A PREROUTING -p tcp -d 10.3.1.254/32 --dport 80 -j MARK --set-mark 100
#iptables -t mangle -A PREROUTING -p tcp -d 10.3.1.254/32 --dport 443 -j MARK --set-mark 100
#iptables -A POSTROUTING -t nat -p tcp -s 10.3.1.0/24 --sport 20 -j MASQUERADE
運(yùn)行以上三行命令并存入/etc/rc.d/rc.local，用ipvsadm看狀態(tài)：
#ipvsadm

IP Virtual Server version 1.0.8 (size=65536)Prot LocalAddress:Port Scheduler Flags??-> RemoteAddress:Port?????????? Forward Weight ActiveConn InActConnTCP??10.3.1.254:ftp wlc??-> cluster2:ftp??????????????????Masq????1??????0??????????0??-> cluster1:ftp??????????????????Masq????1??????0??????????0FWM??100 wlc persistent 60??-> cluster1:0????????????????????Masq????1??????0??????????0??-> cluster2:0????????????????????Masq????1??????0??????????0

注意：a.Firewall Mark可以不要，我反正是加了，文檔說有https的話加上，值我選了100，
b.Virtual IP別加進(jìn)/etc/hosts，我上過當(dāng)，80端口時(shí)有時(shí)無的，
c.eth0:1,eth1:1是piranha產(chǎn)生的，別自己手工設(shè)置，我干過這畫蛇添足的事，網(wǎng)上有
些帖子沒說清，最后是看Redhat的文檔才弄清楚的。
d.The LVS router can monitor the load on the various real servers by using
either rup or ruptime. If you select rup from the drop-down menu, each real
server must run the rstatd service. If you select ruptime, each real server
must run the rwhod service.Redhat的原話，就是如選rup的監(jiān)控模式real server上
都要運(yùn)行rstatd進(jìn)程，如選ruptime就要運(yùn)行rwhod進(jìn)程。
e.Real Server同Router相聯(lián)的網(wǎng)卡的Gateway必須是Router的那塊網(wǎng)卡的VIP，舉本例：
Router的eth1同兩個(gè)real server的eth0相聯(lián)，如VIP eth1:1=192.168.1.1，則real
server 的eth0的Gateway=192.168.1.1
echo "1" > /proc/sys/net/ipv4/ip_forward
q:在兩臺(tái)real server上分別
#vi /usr/local/tomcat/conf/tomcat-users.xml
加下面一行，允許頁(yè)面管理：

r:在兩臺(tái)real server上分別
#service httpd restart
#/usr/local/tomcat/bin/startup.sh
s:打開http://1092.168.1.68:8080和http://1092.168.1.67:8080，選Tomcat Manager，用
manager/tomcat登錄，虛擬目錄/myjsp和/local應(yīng)該Start了
在兩臺(tái)機(jī)子上分別打開網(wǎng)頁(yè)http://10.3.1.254，選WEB Local，可以看到一臺(tái)顯示：
"test page on real server 1"，另一臺(tái)為"test page on real server 2",同時(shí)在Router上
ipvsadm可以看到每個(gè)real server的聯(lián)接數(shù)
8.設(shè)置FTP服務(wù)
#vi /etc/vsftpd/vsftp.conf，在兩臺(tái)real server上分別加入以下幾行：

anon_root=/u01/ftprootlocal_root=/u01/ftprootsetproctitle_enable=YES

#service vsftpd start
現(xiàn)在LVM+GFS+ISCSI+TOMCAT就設(shè)置好了，我們可以用Apache Jmeter來測(cè)試LVM的性能，兩臺(tái)機(jī)子上分別運(yùn)行jmeter，都指向10.3.1.254/myjsp/testoracle.jsp，各200個(gè)threads同時(shí)運(yùn)行，在Router上用ipvsadm可以監(jiān)控，Oracle Server的性能可要好，否則大量的http進(jìn)程會(huì)hang在real server上，ipvsadm也會(huì)顯示有個(gè)real server失去了。測(cè)試時(shí)real server的CPU idle會(huì)降到70%，而Router的CPU idle幾乎不動(dòng)。
??????????????????????????????????????????????????????文章引用自：大行的BLOG

集群LVS+GFS+ISCSI+TOMCAT