Red Hat Knowledgebase

Release Found: Red Hat Enterprise Linux 3 Update 3

Symptom:
Your system boots from local disk(s) and you need to access a shared storage volume. The volume will only be accessed by a single host at any time but you have multiple redundant paths to the storage array. You are not aware of how to configure Red Hat Enterprise Linux to access a LUN (Logical Unit Number) that is exported by the array. Furthermore, you have a requirement for high-availability and scalability robustness.

Solution:
There are several ways to configure this setup, I will illustrate one method that usually works in most situations. However, there are several things to note with this configuration:

• It is assumed that you already have at least a basic understanding of shared storage, related components, and technology.
• Array activation and mounting at boot time, via the initial ramdisk, is currently not supported. Support for this will likely be added in a future Update release.
• There are known problems with using the Logical Volume Manager (LVM) at boot time with multi-path shared storage. If you have single path shared storage, this should not be a problem.
• You may experience difficulty using LVM for both your local drives and on the shared storage array. This problem is tied to the order of loading Kernel modules. Generally, it's best to load the LVM-mod LVM module after you load your Host-Bus Adapter (HBA) and multipath modules.
• You will need to write/configure a service script to control the mounting/unmounting of the shared storage drive(s). I will provide an example for doing this.
• If you are using multiple, redundant, switches and controllers then (depending on your specific hardware) the system may not failover properly in the event of a controller failure.
• In all cases, we recommend thorough testing of the shared storage configuration prior to actual deployment.
• Due to the wide range of hardware and hardware configurations, the setup I illustrate here may not work for you. If this is the case, you should contact your hardware vendor for information on how to proceed.

Assuming your HBA is an X-AMPLE-1000, the Kernel module is xample1000 and your storage LUN shows up as /dev/sda, /dev/sdb, /dev/sdc, and /dev/sdd.

Configuration steps:

1. Verify that the modules for md multipath and LVM are not loaded. These modules are normally called multipath, LVM-mod, and (for example) xample1000. You can display loaded modules with the lsmod command. The removal of modules is accomplished with the rmmod <module name> command.
2. Load the module for your HBA. Assuming that it is already configured in /etc/modules.conf, you can load it with the modprobe xample1000 command.
3. Add your devices to the devlabel facility. You do this with the devlabel add -d <device> -s /dev/exampleName --multipath command. Where <device> is the device name.
   For example:

    
   # rmmod LVM-mod
   # rmmod multipath
   # modprobe xample1000
   # devlabel add -d /dev/sda /dev/exampleName --multipath
   SYMLINK: /dev/exampleName_multipath0 -> /dev/sda
   Added /dev/exampleName_multipath0 to /etc/sysconfig/devlabel
   # devlabel add -d /dev/sdb /dev/exampleName --multipath
   SYMLINK: /dev/exampleName_multipath1 -> /dev/sdb
   Added /dev/exampleName_multipath1 to /etc/sysconfig/devlabel
   # devlabel add -d /dev/sdc /dev/exampleName --multipath
   SYMLINK: /dev/exampleName_multipath2 -> /dev/sdc
   Added /dev/exampleName_multipath2 to /etc/sysconfig/devlabel
   # devlabel add -d /dev/sdd /dev/exampleName --multipath
   SYMLINK: /dev/exampleName_multipath3 -> /dev/sdd
   Added /dev/exampleName_multipath3 to /etc/sysconfig/devlabel
   

4. Create, activate, and start the md multipath device with the

   mdadm --create /dev/md0 --level=multipath --raid-devices=4
      /dev/exampleName_multipath0 /dev/exampleName_multipath1
      /dev/exampleName_multipath2 /dev/exampleName_multipath3
   

   command.
5. Load the LVM module and scan all devices with the vgscan command.
6. Format the md0 device as a Physical Volume (PV) for use with LVM with the pvcreate /dev/md0 command.
7. Create a Volume Group (VG) with the vgcreate vg0 /dev/md0 command.
8. Create a new Logical Volume (LV) with the lvcreate -L <volume size> -n lv0 vg0 command.
9. Format the new LV with, for example:

   mke2fs -j /dev/vg0/lv0
   

10. Finally, mount the new LV, with for example the mount /dev/vg0/lv0 /mnt/test command
    Example output:

     
    # mdadm --create /dev/md0 --level=multipath --raid-devices=4 \
    > /dev/exampleName_multipath0 \
    > /dev/exampleName_multipath1 \
    > /dev/exampleName_multipath2 \
    > /dev/exampleName_multipath3
    
    mdadm: array /dev/md0 started.
    # vgscan
    vgscan -- reading all physical volumes (this may take a while...)
    vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
    vgscan -- WARNING: This program does not do a VGDA backup of your volume group
    
    # pvcreate /dev/md0
    pvcreate -- physical volume "/dev/md0" successfully created
    
    # vgcreate vg0 /dev/md0
    vgcreate -- INFO: using default physical extent size 32 MB
    vgcreate -- INFO: maximum logical volume size is 2 Terabyte
    vgcreate -- doing automatic backup of volume group "vg0"
    vgcreate -- volume group "vg0" successfully created and activated
    
    # lvcreate -L 1G -n lv0 vg0
    lvcreate -- doing automatic backup of "vg0"
    lvcreate -- logical volume "/dev/vg0/lv0" successfully created
    
    # mke2fs -j /dev/vg0/lv0
    mke2fs 1.32 (09-Nov-2002)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    131072 inodes, 262144 blocks
    13107 blocks (5.00%) reserved for the super user
    First data block=0
    8 block groups
    32768 blocks per group, 32768 fragments per group
    16384 inodes per group
    Superblock backups stored on blocks:
             32768, 98304, 163840, 229376
    
    Writing inode tables: done
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    This filesystem will be automatically checked every 38 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.
    # mount /dev/vg0/lv0 /mnt/test
    # touch /mnt/test/test_file
    # ls /mnt/test
    test_file
    

Note: Since you will most likely want this filesystem available on startup, I would suggest creating a simple sysvinit script. In this context, to activate and use the multipath device you will need to assemble it first. This is done with the mdadm --assemble <md device> <member devices> command. This will create the actual /dev/md? device file and start the array. You can then load the LVM module, run a vgscan, and mount the volume.
For example:

  
# mdadm --assemble /dev/md0 \
> /dev/sandisk_multipath0 \
> /dev/sandisk_multipath1 \
> /dev/sandisk_multipath2 \
> /dev/sandisk_multipath3
mdadm: /dev/md0 has been started with 1 drive and 3 spares.

# modprobe lvm-mod
# vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume group

# mount /dev/vg0/lv0 /mnt/whereever
 

You may run into trouble (as noted at the beginning of this FAQ) if you are also using LVM volumes for the rest of your system. Under some circumstances, the vgscan will scan the physical devices and exit with an error before even looking at the md? device. If this is the case, you will probably not be able to use LVM for both your system filesystems and your SAN filesystems. This issue among others will likely be fixed in future releases or updates.

Example sysvinit script (unsupported):

 
#!/bin/bash
#   
# chkconfig: 235 70 30
#    activate in runlevels 2, 3, and 5
#    start with a priority of 70 and stop with 30
# description: Start up multipath SAN device
# processname: sanstart

# source function library
. /etc/init.d/functions

RETVAL=0

start() {
	echo -n $"Starting multipath SAN devices: "
  mdadm --assemble /dev/md0 
    /dev/sandisk_multipath0 \
    /dev/sandisk_multipath1 \
    /dev/sandisk_multipath2 \
    /dev/sandisk_multipath3
  let "RETVAL += $?"
  modprobe lvm-mod
  let "RETVAL += $?"  
  vgchange -ay
  let "RETVAL += $?"  
  vgscan
  let "RETVAL += $?"
  mount /dev/vg0/lv0 /mnt/whereever
  let "RETVAL += $?"
  return $RETVAL
}

stop() {
	echo -n $"Shutting down multipath SAN devices: "
  umount /mnt/whereever
  let "RETVAL += $?"
  vgchange -an
  rmmod lvm-mod
  let "RETVAL += $?"  
  mdadm --stop /dev/md0
  let "RETVAL += $?"
  return $RETVAL  
}

case "$1" in
  start)
	start
	;;
  stop)
	stop
	;;
  restart|reload)
	stop
	start
	RETVAL=$?
	;;
  *)
	echo $"Usage: $0 {start|stop|restart}"
	exit 1
esac

exit $RETVAL