How to setup iSCSI on Linux (RedHat)

Linux, Unix - Linux ..., , , , , , , , , 4 Comments

Definitions:

iSCSI initiator : the endpoint that initiates a iSCSI session.  An iSCSI initiator sends SCSI commands over an IP network. It’s the client endpoint.

iSCSI Target : refers to a storage resource located on an iSCSI server (most of the time it’s a “storage array”). It’s the server endpoint.

LUNs (Logical Number Units): number used to identify a logical unit, which is a device addressed by the SCSI protocol (thus Fiber Channel or iSCSI). It usually represents slices of large RAID disk arrays.

IQN (iSCSI Qualified Name) : iSCSI name of the target or initiator.

 

On the Storage Server:

Enable and configure the iSCSI Target on your storage server.

Mine is a QNAP Turbo NAS. I’ve got 1 target with 5 LUNs configured.

SCSI Portal
 
X Enable iSCSI Target Service
 
iSCSI Service Port:           3260
 
mytarget (iqn.2004-04.com.qnap:ts-859:iscsi.mytarget.c5884b)     Connected
 
id:0 - lun1 ( 2024.00 GB)               Enabled
 
id:1 - lun2 ( 2024.00 GB)               Enabled
 
id:2 - lun3 ( 2024.00 GB)               Enabled
 
id:3 - lun4 ( 2024.00 GB)               Enabled
 
id:4 - lun5 ( 1804.13 GB)               Enabled

I have two network interfaces:

1-       for QNAP management, IP : 10.0.0.5

2-      iSCSI access, directly connected to the server : 192.168.0.1

 

For more security you can enable “LUN masking”. It will restrain iSCSI target to be accessed to only by the initiator of your client (the client initiator name IQN can be found on /etc/iscsi/initiatorname.iscsi).

 

On the Linux client (see tips ** for VMware configuration) :

Install “iscsi-initiator-utils” on the server that will connect to the iSCSI volume:

# rpm –Uvh iscsi-initiator-utils-6.2.0.865-6.el5.x86_64.rpm

Set up iscsi automatic start on boot and start iscsi services:

# chkconfig iscsid on
# service iscsid start
# chkconfig iscsi on
# service iscsi start

 

Discover your iSCSI targets:

# iscsiadm -m discovery -t st -p 192.168.0.1

In my case it will show 2 targets (one for each network connection):

192.168.0.1:3260 iqn.2004-04.com.qnap:ts-859:iscsi. mytarget.c5884b
10.0.0.5:3260 iqn.2004-04.com.qnap:ts-859:iscsi. mytarget.c5884b

I have 2 routes for the same target.

 

Log to the target through IP 192.168.0.1 :

# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-859:iscsi.mytarget.c5884b -p 192.168.0.1 -l

Add automatic login at boot :

# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-859:iscsi.mytarget.c5884b -p 192.168.0.1 --op update -n node.startup -v automatic

As I have another access to the target, I will disable it in order to not disturb the previous configuration:

# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-859:iscsi.mytarget.c5884b -p 10.0.0.5 --logout
# iscsiadm -m node -T iqn.2004-04.com.qnap:ts-859:iscsi.mytarget.c5884b -p 10.0.0.5 --op update -n node.startup -v manual

 

At this point you will see the iSCSI LUNs as block devices on your client.

On my system the five iSCSI block devices are /dev/sdc, sdd, sde, sdf and sdg.

 

You will have create partitions and format them to either standard Linux partition or LVM partition.

I chose LVM because I need large file systems.

You can use parted or fdisk (if < 2To), see article: ” How To Make Partitions Larger Than 2To With Parted GPT Support“.

So here is the result:

# fdisk -l
Disk /dev/sdc: 2173.2 GB, 2173253451776 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1      264216  2122314988+  8e  Linux LVM
Disk /dev/sdd: 2173.2 GB, 2173253451776 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      264216  2122314988+  8e  Linux LVM
Disk /dev/sde: 2173.2 GB, 2173253451776 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      264216  2122314988+  8e  Linux LVM
Disk /dev/sdf: 2173.2 GB, 2173253451776 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdf1               1      264216  2122314988+  8e  Linux LVM
Disk /dev/sdg: 1937.1 GB, 1937169711104 bytes
Device Boot      Start         End      Blocks   Id  System
/dev/sdg1               1      235514  1891766173+  8e  Linux LVM

 

Then create you LVM volume group and logical volume :

# pvcreate /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1
# vgcreate -s 256M vol_vg /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1
# lvcreate -l 28672 vol_vg -n vol_lv1
# lvcreate -l 10924 vol_vg -n vol_lv2

File system creation:

# mkfs -t ext3 -b 4096 -N 100000 /dev/vol_vg/vol_lv1 -L VOL1
# mkfs -t ext3 -b 4096 -N 100000 /dev/vol_vg/vol_lv2 -L VOL2

Then mount the file systems:

# mkdir –p /VOL1 /VOL2
# mount -t ext3 /dev/vol_vg/vol_lv1 /VOL1
# mount -t ext3 /dev/vol_vg/vol_lv2 /VOL2
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vol_vg-vol_lv1 7.0T  6.3T  395G  95% /VOL1
/dev/mapper/vol_vg-vol_lv2 2.7T  1.5T  1.2T  57% /VOL2

If you will want to automatically mount your iSCSI file systems at startup (see article: How To Use UUID And Blkid To Manage Devices).

Get the UUID for each file system:

# blkid /dev/vol_vg/vol_lv1
/dev/vol_vg/vol_lv1: LABEL="VOL1" UUID="4a496f92-6840-4736-a0d5-5b9916113835" SEC_TYPE="ext2" TYPE="ext3"
# blkid /dev/vol_vg/vol_lv2
/dev/vol_vg/vol_lv2: LABEL="VOL2" UUID="cab5e3ec-4797-4227-98e8-e9bca3c3f766" SEC_TYPE="ext2" TYPE="ext3"

The add UUIDs to /etc/fstab :

UUID=4a496f92-6840-4736-a0d5-5b9916113835       /VOL1   ext3 _netdev    0 0
UUID=cab5e3ec-4797-4227-98e8-e9bca3c3f766       /VOL2   ext3 _netdev    0 0

 

** Tip :

If your Linux is a VM on ESXi :

–          Dedicate a network adapter to connect directly the Storage Array to the VMware server.

CAT 5e/6 RJ45 cable (through dedicated hardware switch, if needed).

–          Create a “vSwitch” using the dedicated network adapter with vShere Client.

vswitch iscsi

vswitch iscsi

–          Add a network adapter using the new vSwitch on your virtual host configuration.

Now you have a direct iSCSI connection to you storage array. You can start the configuration.

How to use UUID and blkid to manage devices

Linux, Unix - Linux ..., , , , , , , , , , 3 Comments

Having trouble with device mapping at reboot (iscsi device mapping can change at every reboot) ?

Use UUID (Universal Unique Identifier)!

 

Say you have two iSCSI targets on your Linux. They show, for example, as disks /dev/sdc and /dev/sdd the first time you discover them (with iscsiadm) :

# sfdisk -s

/dev/sda:  20971520
/dev/sdb: 104857600
/dev/sdc: 2122317824
/dev/sdd: 2122317824

Using fdisk or parted you will create the devices /dev/sdc1 and /dev/sdd1 :

# sfdisk -l /dev/sdc

Disk /dev/sdc: 264216 cylinders, 255 heads, 63 sectors/track

Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdc1          0+ 264215  264216- 2122314988+  83  Linux

/dev/sdc2          0       -       0          0    0  Empty

/dev/sdc3          0       -       0          0    0  Empty

/dev/sdc4          0       -       0          0    0  Empty

Disk /dev/sdd: 264216 cylinders, 255 heads, 63 sectors/track

Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdd1          0+ 264215  264216- 2122314988+  83  Linux

/dev/sdd2          0       -       0          0    0  Empty

/dev/sdd3          0       -       0          0    0  Empty

/dev/sdd4          0       -       0          0    0  Empty

Then you can create the two file systems and mount them:

# mkfs -t ext3 /dev/sdc1

# mount -t ext3 /dev/sdc1 /VLS1

# mkfs -t ext3 /dev/sdd1

# mount -t ext3 /dev/sdd1 /VLS2

You need to add the file system in /etc/fstab in order to automatically mount them at startup.

However let’s first reboot the system to check if everything is alright.

At you surprise after reboot you can’t mount /dev/sdc1 and /dev/sdd1.

If you check with sfdisk –l  you can see that /dev/sdc1 and /dev/sdd1 still exist but /dev/sde1 and /dev/sdf1 appeared out of nowhere …

Actually, your two iSCSI disks are now /dev/sde1 et sdf1 (but sdc1 can also be sdd1 at the new reboot and so on …) thanks to the magic of Linux’s dynamic mapping of devices (udev).

That’s where UUID is your best friend !

UUID (Universal Unique Identifier) enable to uniquely identify you device. UUID is not changing at every reboot the way device files (/dev/sdX ou /dev/hdX …) are.

For instance you may know MAC addresses are unique identifier for network cards, well MAC addresses are UUID (version 1).

So if you can get the UUID for your two file systems your problem is solved.

On linux you can get a file system UUIDs with command “blkid” (“vol_id” command can sometimes be used for old Linux versions).

blkid can be found into util-linux package.

# blkid /dev/sdc1

/dev/sdc1: UUID="01066206-c67c-47d1-83a9-d61791fff943" SEC_TYPE="ext2" TYPE="ext3"

# blkid /dev/sdd1

/dev/sdd1 UUID="cea28516-ca98-4ac4-954f-6710b6ac36c7" SEC_TYPE="ext2" TYPE="ext3"

 

Then add the following lines to /etc/fstab :

UUID=4a496f92-6840-4736-a0d5-5b9916113835       /VLS1   ext3 _netdev    0 0
UUID=cab5e3ec-4797-4227-98e8-e9bca3c3f766       /VLS2   ext3 _netdev    0 0

“_netdev” substitute to the usual “default” because network needs to be up before iSCSI filesystems mount.

Now your iSCSI file systems will be automatically mounted after every reboot (well, as long as your iSCSI server is up ?).