HA Cluster with FreeBSD

From The Datapanic Wiki
Jump to navigation Jump to search
High Availability Cluster

This is a 2-node cluster with a floating IP address cluster file system for client mail service
Setup 2 basic FreeBSD 11.1 servers

Install the following packages (all nodes):
net/pacemaker
(net/corosync)
net-mgmt/crmsh
net/glusterfs

Edit /usr/local/etc/corosync/corosync.conf (all nodes):
totem {
  version: 2
  cluster_name: mail01
  transport: udpu
  interface {
    ringnumber: 0
    bindnetaddr: 10.0.1.0
    broadcast: yes
    mcastport: 5405
  }
}

quorum {
  provider: corosync_votequorum
  two_node: 1
}

nodelist {
  node {
    ring0_addr: 10.0.1.131
    name: mail01a
    nodeid: 1
  }
  node {
    ring0_addr: 10.0.1.132
    name: mail01b
    nodeid: 2
  }
}

logging {
  to_syslog: yes
  timestamp: on
}

Create /usr/local/etc/coresync/service.d/pcmk (all nodes):
service {
        name: pacemaker
        ver: 1
}

run /usr/local/sbin/corosync-keygen on either node  and cp /usr/local/etc/corosync/authkey to other node
chmod 400 /usr/local/etc/corosync/authkey

Edit /boot/loader.conf and add the following (all nodes):
fuse_load="YES"

Edit /etc/rc.conf and add the following (all nodes):
#
# CLUSTERING
#
corosync_enable="YES"
corosync_notifyd_enable="YES"
pacemaker_enable="YES"
glusterd_enable="YES"

reboot

Start the corosync service (all nodes):
service corosync start

Check corosync setup (all nodes):
/usr/local/sbin/corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.1.131)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.1.132)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined

Start the pacemaker service (all nodes):
service pacemaker start

Check the cluster status (any node):
/usr/local/bin/crm status
Stack: corosync
Current DC: mail01a (version 1.1.16-94ff4df51a) - partition with quorum
Last updated: Wed Dec 13 10:29:27 2017
Last change: Wed Dec 13 10:29:12 2017 by hacluster via crmd on mail01a

2 nodes configured
0 resources configured

Online: [ mail01a mail01b ]

No resources

If desired to see realtime status, use crm_mon (any node):
/usr/local/sbin/crm_mon
To show a one-shot status use crm_mon -1 (any node):
/usr/local/sbin/crm_mon -1

Show the cluster configuration (any node):
/usr/local/bin/crm configure show
node 1: mail01a
node 2: mail01b
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=1.1.16-94ff4df51a \
        cluster-infrastructure=corosync \
        cluster-name=mail01

Add some additional configuration options (any node):
/usr/local/bin/crm configure property stonith-enabled=false
/usr/local/bin/crm configure property no-quorum-policy=ignore
/usr/local/bin/crm configure rsc_defaults migration-threshold=1
/usr/local/bin/crm configure rsc_defaults resource-stickiness=100

Adding a primitive (any node):

view ocf paramters (any node):
/usr/local/bin/crm ra meta IPaddr

Add the floating IP address:
/usr/local/bin/crm configure primitive ClusterIP ocf:heartbeat:IPaddr nic=em0 ip=10.0.1.130 cidr_netmask=24 broadcast=10.0.1.255
op monitor interval=30s

In some cases, a route may be needed:
#/usr/local/bin/crm configure primitive ClusterRoute ocf:heartbeat:Route destination=default device=em0 gateway=10.0.1.1

A static IP should be assigned to the nic, if not, then add the following to /etc/sysctl.conf (all nodes):
/sbin/sysctl -w net.ipv4.conf.all.promote_secondaries=1

Other useful parameters for IPaddr:
local_start_script
local_stop_script

Determine which node has the floating IP address and reboot it to test failover of the floating IP address to the other node

Remove a resource:
/usr/local/bin/crm resource stop ClusterIP
/usr/local/bin/crm configure delete ClusterIP

Remove an attribute:
/usr/local/sbin/crm_attribute --id=cib-bootstrap-options-migration-threshold --name=migration-threshold --delete

Remove resource options:
/usr/local/bin/crm configure delete rsc-options

Localtion of ocf files:
/usr/local/lib/ocf/resource.d/

Description of crm_* commands and more: http://clusterlabs.org/man/pacemaker/

Pacemaker Tutorial:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/

Adding a new disk (/dev/da1) to the cluster (all nodes):
/sbin/newfs -U /dev/da1
/bin/mkdir -p /data/brick1

Edit /etc/fstab (all nodes):
/dev/da1        /data/brick1    ufs     rw      1       1

Mount the new file system:
/sbin/mount -a

reboot all nodes

Configure trusted pool:
On first node:
/usr/local/sbin/gluster peer probe mail01b
On second node:
/usr/local/sbin/gluster peer probe mail01a
Check gluster peer status (all nodes):
/usr/local/sbin/gluster peer status
Number of Peers: 1

Hostname: mail01b
Uuid: 05835b3d-53ed-48dc-b767-8c225dc3bd8e
State: Peer in Cluster (Connected)

Create the glusterfs volume path (all nodes):
/bin/mkdir -p /data/brick1/gv0

Start the replication (any node):
/usr/local/sbin/gluster volume create gv0 replica 2 mail01a:/data/brick1/gv0 mail01b:/data/brick1/gv0

Show the gluster volume info (any node):
/usr/local/sbin/gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 2d2fdec1-d219-487f-b860-fb844c840923
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: mail01a:/data/brick1/gv0
Brick2: mail01b:/data/brick1/gv0
Options Reconfigured:
nfs.disable: on
transport.address-family: inet

Additional options (any node):
/usr/local/sbin/gluster volume set gv0 nfs.disable off
/usr/local/sbin/gluster volume set gv0 nfs.export-volumes on
/usr/local/sbin/gluster volume set gv0 nfs.rpc-auth-allow 10.0.1.*

Display options (any node):
/usr/local/sbin/gluster volume get gv0 nfs.disable

Mounting the gluster volume:

On the first node:
/usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01a /mnt
On the second node:
/usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01b /mnt

Other methods:
/usr/local/sbin/mount_glusterfs mail01a:/gv0 /mnt
/usr/local/sbin/mount_glusterfs mail01b:/gv0 /mnt

Remote Clients:
pkg install net/glusterfs

Edit /boot/loader.conf and add the following:
fuse_load="YES"

reboot

Edit /etc/fstab:
mail01:/gv0   /mnt            fusefs  rw,failok,late,mountprog=/usr/local/sbin/mount_glusterfs 0 0

/sbin/mount -a

GlusterFS Quick Start Guide:
http://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/