HA Cluster with FreeBSD

From The Datapanic Wiki
Revision as of 09:18, 10 May 2018 by Datapanic (talk | contribs) (Created page with "High Availability Cluster This is a 2-node cluster with a floating IP address cluster file system for client mail service Setup 2 basic FreeBSD 11.1 servers Install the foll...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

High Availability Cluster

This is a 2-node cluster with a floating IP address cluster file system for client mail service Setup 2 basic FreeBSD 11.1 servers

Install the following packages (all nodes): net/pacemaker (net/corosync) net-mgmt/crmsh net/glusterfs

Edit /usr/local/etc/corosync/corosync.conf (all nodes): totem {

 version: 2
 cluster_name: mail01
 transport: udpu
 interface {
   ringnumber: 0
   bindnetaddr: 10.0.1.0
   broadcast: yes
   mcastport: 5405
 }

}

quorum {

 provider: corosync_votequorum
 two_node: 1

}

nodelist {

 node {
   ring0_addr: 10.0.1.131
   name: mail01a
   nodeid: 1
 }
 node {
   ring0_addr: 10.0.1.132
   name: mail01b
   nodeid: 2
 }

}

logging {

 to_syslog: yes
 timestamp: on

}

Create /usr/local/etc/coresync/service.d/pcmk (all nodes): service {

       name: pacemaker
       ver: 1

}

run /usr/local/sbin/corosync-keygen on either node and cp /usr/local/etc/corosync/authkey to other node chmod 400 /usr/local/etc/corosync/authkey

Edit /boot/loader.conf and add the following (all nodes): fuse_load="YES"

Edit /etc/rc.conf and add the following (all nodes):

  1. CLUSTERING

corosync_enable="YES" corosync_notifyd_enable="YES" pacemaker_enable="YES" glusterd_enable="YES"

reboot

Start the corosync service (all nodes): service corosync start

Check corosync setup (all nodes): /usr/local/sbin/corosync-cmapctl | grep members runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.1.131) runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.1.status (str) = joined runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.1.132) runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.2.status (str) = joined

Start the pacemaker service (all nodes): service pacemaker start

Check the cluster status (any node): /usr/local/bin/crm status Stack: corosync Current DC: mail01a (version 1.1.16-94ff4df51a) - partition with quorum Last updated: Wed Dec 13 10:29:27 2017 Last change: Wed Dec 13 10:29:12 2017 by hacluster via crmd on mail01a

2 nodes configured 0 resources configured

Online: [ mail01a mail01b ]

No resources

If desired to see realtime status, use crm_mon (any node): /usr/local/sbin/crm_mon To show a one-shot status use crm_mon -1 (any node): /usr/local/sbin/crm_mon -1

Show the cluster configuration (any node): /usr/local/bin/crm configure show node 1: mail01a node 2: mail01b property cib-bootstrap-options: \

       have-watchdog=false \
       dc-version=1.1.16-94ff4df51a \
       cluster-infrastructure=corosync \
       cluster-name=mail01

Add some additional configuration options (any node): /usr/local/bin/crm configure property stonith-enabled=false /usr/local/bin/crm configure property no-quorum-policy=ignore /usr/local/bin/crm configure rsc_defaults migration-threshold=1 /usr/local/bin/crm configure rsc_defaults resource-stickiness=100

Adding a primitive (any node):

view ocf paramters (any node): /usr/local/bin/crm ra meta IPaddr

Add the floating IP address: /usr/local/bin/crm configure primitive ClusterIP ocf:heartbeat:IPaddr nic=em0 ip=10.0.1.130 cidr_netmask=24 broadcast=10.0.1.255 op monitor interval=30s

In some cases, a route may be needed:

  1. /usr/local/bin/crm configure primitive ClusterRoute ocf:heartbeat:Route destination=default device=em0 gateway=10.0.1.1

A static IP should be assigned to the nic, if not, then add the following to /etc/sysctl.conf (all nodes): /sbin/sysctl -w net.ipv4.conf.all.promote_secondaries=1

Other useful parameters for IPaddr: local_start_script local_stop_script

Determine which node has the floating IP address and reboot it to test failover of the floating IP address to the other node

Remove a resource: /usr/local/bin/crm resource stop ClusterIP /usr/local/bin/crm configure delete ClusterIP

Remove an attribute: /usr/local/sbin/crm_attribute --id=cib-bootstrap-options-migration-threshold --name=migration-threshold --delete

Remove resource options: /usr/local/bin/crm configure delete rsc-options

Localtion of ocf files: /usr/local/lib/ocf/resource.d/

Description of crm_* commands and more: http://clusterlabs.org/man/pacemaker/

Pacemaker Tutorial: http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/

Adding a new disk (/dev/da1) to the cluster (all nodes): /sbin/newfs -U /dev/da1 /bin/mkdir -p /data/brick1

Edit /etc/fstab (all nodes): /dev/da1 /data/brick1 ufs rw 1 1

Mount the new file system: /sbin/mount -a

reboot all nodes

Configure trusted pool: On first node: /usr/local/sbin/gluster peer probe mail01b On second node: /usr/local/sbin/gluster peer probe mail01a Check gluster peer status (all nodes): /usr/local/sbin/gluster peer status Number of Peers: 1

Hostname: mail01b Uuid: 05835b3d-53ed-48dc-b767-8c225dc3bd8e State: Peer in Cluster (Connected)

Create the glusterfs volume path (all nodes): /bin/mkdir -p /data/brick1/gv0

Start the replication (any node): /usr/local/sbin/gluster volume create gv0 replica 2 mail01a:/data/brick1/gv0 mail01b:/data/brick1/gv0

Show the gluster volume info (any node): /usr/local/sbin/gluster volume info Volume Name: gv0 Type: Replicate Volume ID: 2d2fdec1-d219-487f-b860-fb844c840923 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: mail01a:/data/brick1/gv0 Brick2: mail01b:/data/brick1/gv0 Options Reconfigured: nfs.disable: on transport.address-family: inet

Additional options (any node): /usr/local/sbin/gluster volume set gv0 nfs.disable off /usr/local/sbin/gluster volume set gv0 nfs.export-volumes on /usr/local/sbin/gluster volume set gv0 nfs.rpc-auth-allow 10.0.1.*

Display options (any node): /usr/local/sbin/gluster volume get gv0 nfs.disable

Mounting the gluster volume:

On the first node: /usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01a /mnt On the second node: /usr/local/sbin/glusterfs --log-file=/var/log/gluster.log --log-level=INFO --volfile-id=gv0 --volfile-server=mail01b /mnt

Other methods: /usr/local/sbin/mount_glusterfs mail01a:/gv0 /mnt /usr/local/sbin/mount_glusterfs mail01b:/gv0 /mnt

Remote Clients: pkg install net/glusterfs

Edit /boot/loader.conf and add the following: fuse_load="YES"

reboot

Edit /etc/fstab: mail01:/gv0 /mnt fusefs rw,failok,late,mountprog=/usr/local/sbin/mount_glusterfs 0 0

/sbin/mount -a

GlusterFS Quick Start Guide: http://docs.gluster.org/en/latest/Quick-Start-Guide/Quickstart/