Red Hat Knowledgebase

Limitation: This only applies for Red Hat Enterprise Linux 4 Update 2 or later. Setting up the Global File System (GFS) on Red Hat Enterprise Linux 4 is a relatively simple process, but does require several steps. This brief article provides a general overview of all necessary steps; more detail of each step can be found in separate Knowledge Base Articles. It is assumed that Red Hat Enterprise Linux 4 is installed on each node and that each node is connected to the shared storage device. For this article, the example will refer to the shared storage device as /dev/sde. The examples in this article refer to a two node cluster.

1. Install the needed packages on all nodes. For a complete list of packages, view the GFS 6.1 user manual (http://www.redhat.com/docs/manuals/csgfs/browse/rh-gfs-en/ch-install.html#S1-PKG-INSTALL)

   NOTE: for each package that ends in "-kernel" there are variants for each kernel flavor. For example, for the SMP kernel, the packages to install would be cman-kernel-smp, GFS-kernel-smp, and so forth.

2. Modify the /etc/hosts file on each node
   • Do not alter the localhost line
   • Add a line for each node
   • Add a line for the heartbeat IP if needed
3. Run system-config-cluster on one node
   • Name the cluster (click on "Cluster", then click "Edit Cluster Properties")
   • Set up each node (click on "Cluster Nodes", then click "Add a Cluster Node")
   • Set up the fence device (Click on "Fence Devices", then click "Add a Fence Device")
   • Add a fence device to each node (Click on the Node name from the nodes created; click "Manage Fencing for This Node"; click "Add a New Fence Level"; click "Fence-Level-1; click "Add a New Fence to This Level"; finally, select the fence device and enter any required information such as port or switch number.
   • There is no need to create any “Services” or “Resources” at this time.
   • Save the configuration and copy the file /etc/cluster/cluster.conf to each node.
   • Start the cluster services:

     service ccsd start 
     service cman start
     service fenced start
     service clvmd start
     service gfs start
     

   • Run clustat on each node to verify that the cluster is operational
   • If the services fail to start:
     • Check to see the /etc/hosts file is accurate on each node.
     • Ensure the /etc/cluster/cluster.conf file is exactly the same on each node.
     • Ensure all the packages are installed on each node.
     • Do not move forward until the cluster services start without error.
4. Create logical volumes on the shared storage. CLVM is used to do this. It is a cluster-aware version of LVM. However, all the LVM commands are standard commands.
   • Check the /etc/lvm/lvm.conf file to make sure it includes the following lines:

     locking_type = 2
     locking_library = "/usr/lib/liblvm2clusterlock.so"
     

     Then run the following commands:

     pvcreate /dev/sde
     vgcreate volGFS01 /dev/sde
     lvcreate -L size in megabites volGFS01
     

     Each of the above commands should return output that indicates success. Run vgchange -ay to activate the new volumes. For more information on creating logical volumes, see other articles in the Knowledgebase.

5. Make and Mount the GFS file system:

   gfs_mkfs -p lock_dlm -t nameofcluster:gfs01 -j number of nodes /dev/volGFS/lvol0
   mkdir /mnt/gfs
   mount -t gfs /dev/volGFS/lvol0 /mnt/gfs
   

Note: The GFS volume group should now be seen on all nodes and can run the above mount command on each node.

Note: Red Hat Enterprise Linux 5 has a new locking_type = 3 in /etc/lvm/lvm.conf.

The logical volume manager can take a new locking_type = 3 to figure out the appropriate locking for clustered and non-clustered volumes.