HA Cluster with FreeBSD
Jump to navigation
Jump to search
High Availability Cluster
This is a 2-node cluster with a floating IP address cluster file system for client mail service
Setup 2 basic FreeBSD 11.1 servers
Install the following packages (all nodes):
net/pacemaker
(net/corosync)
net-mgmt/crmsh
net/glusterfs
Edit /usr/local/etc/corosync/corosync.conf (all nodes):
totem {
version: 2
cluster_name: mail01
transport: udpu
interface {
ringnumber: 0
bindnetaddr: 10.0.1.0
broadcast: yes
mcastport: 5405
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
nodelist {
node {
ring0_addr: 10.0.1.131
name: mail01a
nodeid: 1
}
node {
ring0_addr: 10.0.1.132
name: mail01b
nodeid: 2
}
}
logging {
to_syslog: yes
timestamp: on
}
Create /usr/local/etc/coresync/service.d/pcmk (all nodes):
service {
name: pacemaker
ver: 1
}
run /usr/local/sbin/corosync-keygen on either node and cp /usr/local/etc/corosync/authkey to other node
chmod 400 /usr/local/etc/corosync/authkey
Edit /boot/loader.conf and add the following (all nodes):
fuse_load="YES"
Edit /etc/rc.conf and add the following (all nodes):
#
# CLUSTERING
#
corosync_enable="YES"
corosync_notifyd_enable="YES"
pacemaker_enable="YES"
glusterd_enable="YES"
reboot
Start the corosync service (all nodes):
service corosync start
Check corosync setup (all nodes):
/usr/local/sbin/corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.1.131)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.1.132)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
Start the pacemaker service (all nodes):
service pacemaker start
Check the cluster status (any node):
/usr/local/bin/crm status
Stack: corosync
Current DC: mail01a (version 1.1.16-94ff4df51a) - partition with quorum
Last updated: Wed Dec 13 10:29:27 2017
Last change: Wed Dec 13 10:29:12 2017 by hacluster via crmd on mail01a
2 nodes configured
0 resources configured
Online: [ mail01a mail01b ]
No resources
If desired to see realtime status, use crm_mon (any node):
/usr/local/sbin/crm_mon
To show a one-shot status use crm_mon -1 (any node):
/usr/local/sbin/crm_mon -1
Show the cluster configuration (any node):
/usr/local/bin/crm configure show
node 1: mail01a
node 2: mail01b
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.16-94ff4df51a \
cluster-infrastructure=corosync \
cluster-name=mail01
Add some additional configuration options (any node):
/usr/local/bin/crm configure property stonith-enabled=false
/usr/local/bin/crm configure property no-quorum-policy=ignore
/usr/local/bin/crm configure rsc_defaults migration-threshold=1
/usr/local/bin/crm configure rsc_defaults resource-stickiness=100
Adding a primitive (any node):
view ocf paramters (any node):
/usr/local/bin/crm ra meta IPaddr
Add the floating IP address:
/usr/local/bin/crm configure primitive ClusterIP ocf:heartbeat:IPaddr nic=em0 ip=10.0.1.130 cidr_netmask=24 broadcast=10.0.1.255
op monitor interval=30s
In some cases, a route may be needed:
#/usr/local/bin/crm configure primitive ClusterRoute ocf:heartbeat:Route destination=default device=em0 gateway=10.0.1.1
A static IP should be assigned to the nic, if not, then add the following to /etc/sysctl.conf (all nodes):
/sbin/sysctl -w net.ipv4.conf.all.promote_secondaries=1
Other useful parameters for IPaddr:
local_start_script
local_stop_script
Determine which node has the floating IP address and reboot it to test failover of the floating IP address to the other node
Remove a resource:
/usr/local/bin/crm resource stop ClusterIP
/usr/local/bin/crm configure delete ClusterIP
Remove an attribute:
/usr/local/sbin/crm_attribute --id=cib-bootstrap-options-migration-threshold --name=migration-threshold --delete
Remove resource options:
/usr/local/bin/crm configure delete rsc-options
Localtion of ocf files:
/usr/local/lib/ocf/resource.d/
Description of crm_* commands and more: http://clusterlabs.org/man/pacemaker/
Pacemaker Tutorial:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/
Adding a new disk (/dev/da1) to the cluster (all nodes):
/sbin/newfs -U /dev/da1
/bin/mkdir -p /data/brick1
Edit /etc/fstab (all nodes):
/dev/da1 /data/brick1 ufs rw 1 1
Mount the new file system:
/sbin/mount -a
reboot all nodes
Configure trusted pool:
On first node:
/usr/local/sbin/gluster peer probe mail01b
On second node:
/usr/local/sbin/gluster peer probe mail01a
Check gluster peer status (all nodes):
/usr/local/sbin/gluster peer status
Number of Peers: 1
Hostname: mail01b
Uuid: 05835b3d-53ed-48dc-b767-8c225dc3bd8e
State: Peer in Cluster (Connected)
Create the glusterfs volume path (all nodes):
/bin/mkdir -p /data/brick1/gv0
Start the replication (any node):
/usr/local/sbin/gluster volume create gv0 replica 2 mail01a:/data/brick1/gv0 mail01b:/data/brick1/gv0
Show the gluster volume info (any node):
/usr/local/sbin/gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 2d2fdec1-d219-487f-b860-fb844c840923
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: mail01a:/data/brick1/gv0
Brick2: mail01b:/data/brick1/gv0
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
Additional options (any node):
/usr/local/sbin/gluster volume set gv0 nfs.disable off
/usr/local/sbin/gluster volume set gv0 nfs.export-volumes on
/usr/local/sbin/gluster volume set gv0 nfs.rpc-auth-allow 10.0.1.*
Display options (any node):
/usr/local/sbin/gluster volume get gv0 nfs.disable
Mounting the gluster volume:
On the first node:
/usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01a /mnt
On the second node:
/usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01b /mnt
Other methods:
/usr/local/sbin/mount_glusterfs mail01a:/gv0 /mnt
/usr/local/sbin/mount_glusterfs mail01b:/gv0 /mnt
Remote Clients:
pkg install net/glusterfs
Edit /boot/loader.conf and add the following:
fuse_load="YES"
reboot
Edit /etc/fstab:
mail01:/gv0 /mnt fusefs rw,failok,late,mountprog=/usr/local/sbin/mount_glusterfs 0 0
/sbin/mount -a
GlusterFS Quick Start Guide:
http://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/
|