HA Cluster with FreeBSD
High Availability Cluster
This is a 2-node cluster with a floating IP address cluster file system for client mail service Setup 2 basic FreeBSD 11.1 servers
Install the following packages (all nodes): net/pacemaker (net/corosync) net-mgmt/crmsh net/glusterfs
Edit /usr/local/etc/corosync/corosync.conf (all nodes): totem {
version: 2
cluster_name: mail01
transport: udpu
interface {
ringnumber: 0
bindnetaddr: 10.0.1.0
broadcast: yes
mcastport: 5405
}
}
quorum {
provider: corosync_votequorum two_node: 1
}
nodelist {
node {
ring0_addr: 10.0.1.131
name: mail01a
nodeid: 1
}
node {
ring0_addr: 10.0.1.132
name: mail01b
nodeid: 2
}
}
logging {
to_syslog: yes timestamp: on
}
Create /usr/local/etc/coresync/service.d/pcmk (all nodes): service {
name: pacemaker
ver: 1
}
run /usr/local/sbin/corosync-keygen on either node and cp /usr/local/etc/corosync/authkey to other node chmod 400 /usr/local/etc/corosync/authkey
Edit /boot/loader.conf and add the following (all nodes): fuse_load="YES"
Edit /etc/rc.conf and add the following (all nodes):
- CLUSTERING
corosync_enable="YES" corosync_notifyd_enable="YES" pacemaker_enable="YES" glusterd_enable="YES"
reboot
Start the corosync service (all nodes): service corosync start
Check corosync setup (all nodes): /usr/local/sbin/corosync-cmapctl | grep members runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.1.131) runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.1.status (str) = joined runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.1.132) runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.2.status (str) = joined
Start the pacemaker service (all nodes): service pacemaker start
Check the cluster status (any node): /usr/local/bin/crm status Stack: corosync Current DC: mail01a (version 1.1.16-94ff4df51a) - partition with quorum Last updated: Wed Dec 13 10:29:27 2017 Last change: Wed Dec 13 10:29:12 2017 by hacluster via crmd on mail01a
2 nodes configured 0 resources configured
Online: [ mail01a mail01b ]
No resources
If desired to see realtime status, use crm_mon (any node): /usr/local/sbin/crm_mon To show a one-shot status use crm_mon -1 (any node): /usr/local/sbin/crm_mon -1
Show the cluster configuration (any node): /usr/local/bin/crm configure show node 1: mail01a node 2: mail01b property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.16-94ff4df51a \
cluster-infrastructure=corosync \
cluster-name=mail01
Add some additional configuration options (any node): /usr/local/bin/crm configure property stonith-enabled=false /usr/local/bin/crm configure property no-quorum-policy=ignore /usr/local/bin/crm configure rsc_defaults migration-threshold=1 /usr/local/bin/crm configure rsc_defaults resource-stickiness=100
Adding a primitive (any node):
view ocf paramters (any node): /usr/local/bin/crm ra meta IPaddr
Add the floating IP address: /usr/local/bin/crm configure primitive ClusterIP ocf:heartbeat:IPaddr nic=em0 ip=10.0.1.130 cidr_netmask=24 broadcast=10.0.1.255 op monitor interval=30s
In some cases, a route may be needed:
- /usr/local/bin/crm configure primitive ClusterRoute ocf:heartbeat:Route destination=default device=em0 gateway=10.0.1.1
A static IP should be assigned to the nic, if not, then add the following to /etc/sysctl.conf (all nodes): /sbin/sysctl -w net.ipv4.conf.all.promote_secondaries=1
Other useful parameters for IPaddr: local_start_script local_stop_script
Determine which node has the floating IP address and reboot it to test failover of the floating IP address to the other node
Remove a resource: /usr/local/bin/crm resource stop ClusterIP /usr/local/bin/crm configure delete ClusterIP
Remove an attribute: /usr/local/sbin/crm_attribute --id=cib-bootstrap-options-migration-threshold --name=migration-threshold --delete
Remove resource options: /usr/local/bin/crm configure delete rsc-options
Localtion of ocf files: /usr/local/lib/ocf/resource.d/
Description of crm_* commands and more: http://clusterlabs.org/man/pacemaker/
Pacemaker Tutorial: http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/
Adding a new disk (/dev/da1) to the cluster (all nodes): /sbin/newfs -U /dev/da1 /bin/mkdir -p /data/brick1
Edit /etc/fstab (all nodes): /dev/da1 /data/brick1 ufs rw 1 1
Mount the new file system: /sbin/mount -a
reboot all nodes
Configure trusted pool: On first node: /usr/local/sbin/gluster peer probe mail01b On second node: /usr/local/sbin/gluster peer probe mail01a Check gluster peer status (all nodes): /usr/local/sbin/gluster peer status Number of Peers: 1
Hostname: mail01b Uuid: 05835b3d-53ed-48dc-b767-8c225dc3bd8e State: Peer in Cluster (Connected)
Create the glusterfs volume path (all nodes): /bin/mkdir -p /data/brick1/gv0
Start the replication (any node): /usr/local/sbin/gluster volume create gv0 replica 2 mail01a:/data/brick1/gv0 mail01b:/data/brick1/gv0
Show the gluster volume info (any node): /usr/local/sbin/gluster volume info Volume Name: gv0 Type: Replicate Volume ID: 2d2fdec1-d219-487f-b860-fb844c840923 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: mail01a:/data/brick1/gv0 Brick2: mail01b:/data/brick1/gv0 Options Reconfigured: nfs.disable: on transport.address-family: inet
Additional options (any node): /usr/local/sbin/gluster volume set gv0 nfs.disable off /usr/local/sbin/gluster volume set gv0 nfs.export-volumes on /usr/local/sbin/gluster volume set gv0 nfs.rpc-auth-allow 10.0.1.*
Display options (any node): /usr/local/sbin/gluster volume get gv0 nfs.disable
Mounting the gluster volume:
On the first node: /usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01a /mnt On the second node: /usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01b /mnt
Other methods: /usr/local/sbin/mount_glusterfs mail01a:/gv0 /mnt /usr/local/sbin/mount_glusterfs mail01b:/gv0 /mnt
Remote Clients: pkg install net/glusterfs
Edit /boot/loader.conf and add the following: fuse_load="YES"
reboot
Edit /etc/fstab: mail01:/gv0 /mnt fusefs rw,failok,late,mountprog=/usr/local/sbin/mount_glusterfs 0 0
/sbin/mount -a
GlusterFS Quick Start Guide: http://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/