Speed up WAN file transfer with compression

If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency.

To speed up you transfers you need to compress the data so you will have less to transfer.

You can chose to enable ssh compression but if you also have servers on LAN, you will probably slow down those transfer rates.

So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer.

This is how you can do it with only one command line.

myserver1 has a big file “/tmp/myfile.log” that you want on myservers2.

From myserver2  :

[myserver2]/# ssh myserver1 "gzip -c /tmp/myfile.log" |gunzip > myfile.log

Here is a comparison between a transfer using scp only and a transfer using ssh and gzip.

Transfer with compression is 6 times faster in this case :

# time ssh myserver1 "gzip -c /ORACLE/ora92/network/log/listener.log.myserver1" |gunzip > listener.log.myserver1
real    0m40.040s
user    0m2.159s
sys     0m1.665s
 
# time scp myserver1:/ORACLE/ora92/network/log/listener.log.myserver1 .
real    4m1.95s
user    0m6.15s
sys     0m5.80s

 

VI : how to delete lines with- Comment supprimer des lignes avec vi

Go to command mode with <ESC> and type :
Allez dans le mode commande avec <ECH> et taper :

dd

to delete one line.
pour supprimer une ligne.

12dd

to delete 12 lines from the current position.
pour supprimer 12 lignes de la position actuelle.

dG

to delete the lines from the current position to the end of the file.
pour supprimer les lignes de la position actuelle jusqu’à la fin du fichier.

:111,222d

to delete from line number 111 to 222 (included).
pour supprimer de la ligne 111 à la ligne 222 (inclus).

:1,$d

to delete all the lines.
pour supprimer toutes les lignes.

Last trick :
Dernière astuce :

:se nu

to print the line numbers on the screen.
pour afficher le numéro des lignes à l’écran.

How to make partitions larger than 2To with parted GPT support

On Unix or Linux I never had to manage LUNs larger than 2To because using LVM I can create very large filesystems with reasonably small LUNs (in general I use LUNs from 256Go to 2To depending on purpose of the filesystem). It’s more convenient to manage “small” LUNs as you arrange them the way you want.

Nevertheless, low cost storage DAS arrays (understand array without SAN like eSATA arrays) sometimes comes with very basic option RAID level, number of disks, period.

In this case I have a SAS array with two RAID 5 volumes, 23To each and I want to make 1 big filesystem with the 2 disks.

Usually I choose fdisk to create partitions but fdisk will fail to manage volumes greater than 2To (MBR limitations).

GPT/EFI partitions:

In order to create larger partitions you have to use GPT (GUID partition table) and EFI partitions.

http://en.wikipedia.org/wiki/GUID_Partition_Table

http://en.wikipedia.org/wiki/EFI_System_partition

GPT support must be included in the kernel in order to work with EFI partitions.

A lot of professional Linux distributions embed GPT support by default.

GNU Parted is GPT compatible, so this is how to proceed.

Parted:

Create the two 23To partitions with parted:

# parted /dev/sdc mklabel gpt
# parted /dev/sdc mkpart primary 1 -1
# parted /dev/sdd mklabel gpt
# parted /dev/sdd mkpart primary 1 -1

(-1 indicate the end of the disk).

 

# parted dev/sdc print
Model: DELL PERC 6/E Adapter (scsi)
Disk /dev/sdc: 24.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End     Size    File system  Name     Flags
1      1049kB  24.0TB  24.0TB               primary
 
# parted /dev/sdd print
Model: DELL PERC 6/E Adapter (scsi)
Disk /dev/sdd: 24.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End     Size    File system  Name     Flags
1      1049kB  24.0TB  24.0TB               primary

 

LVM and xfs :

Use LVM to aggregate the 2 partitions:

# pvcreate /dev/sdc1 /dev/sdd1
# vgcreate vg_data /dev/sdc1 /dev/sdd1
# lvcreate -l 100%FREE -n lvdata1 vg_data

And create the filesystem :

# mkfs -t xfs /dev/vg_data/lvdata1
# df -h /dev/mapper/vg_data-lvdata1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/ vg_data-lvdata1
44T  534G   44T   2% /backupst1

I had to use xfs (you will need a license on some professional Linux distributions) because ext4 is limited to 16To filesystems.

How to use UUID and blkid to manage devices

Having trouble with device mapping at reboot (iscsi device mapping can change at every reboot) ?

Use UUID (Universal Unique Identifier)!

 

Say you have two iSCSI targets on your Linux. They show, for example, as disks /dev/sdc and /dev/sdd the first time you discover them (with iscsiadm) :

# sfdisk -s

/dev/sda:  20971520
/dev/sdb: 104857600
/dev/sdc: 2122317824
/dev/sdd: 2122317824

Using fdisk or parted you will create the devices /dev/sdc1 and /dev/sdd1 :

# sfdisk -l /dev/sdc

Disk /dev/sdc: 264216 cylinders, 255 heads, 63 sectors/track

Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdc1          0+ 264215  264216- 2122314988+  83  Linux

/dev/sdc2          0       -       0          0    0  Empty

/dev/sdc3          0       -       0          0    0  Empty

/dev/sdc4          0       -       0          0    0  Empty

Disk /dev/sdd: 264216 cylinders, 255 heads, 63 sectors/track

Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

Device Boot Start     End   #cyls    #blocks   Id  System

/dev/sdd1          0+ 264215  264216- 2122314988+  83  Linux

/dev/sdd2          0       -       0          0    0  Empty

/dev/sdd3          0       -       0          0    0  Empty

/dev/sdd4          0       -       0          0    0  Empty

Then you can create the two file systems and mount them:

# mkfs -t ext3 /dev/sdc1

# mount -t ext3 /dev/sdc1 /VLS1

# mkfs -t ext3 /dev/sdd1

# mount -t ext3 /dev/sdd1 /VLS2

You need to add the file system in /etc/fstab in order to automatically mount them at startup.

However let’s first reboot the system to check if everything is alright.

At you surprise after reboot you can’t mount /dev/sdc1 and /dev/sdd1.

If you check with sfdisk –l  you can see that /dev/sdc1 and /dev/sdd1 still exist but /dev/sde1 and /dev/sdf1 appeared out of nowhere …

Actually, your two iSCSI disks are now /dev/sde1 et sdf1 (but sdc1 can also be sdd1 at the new reboot and so on …) thanks to the magic of Linux’s dynamic mapping of devices (udev).

That’s where UUID is your best friend !

UUID (Universal Unique Identifier) enable to uniquely identify you device. UUID is not changing at every reboot the way device files (/dev/sdX ou /dev/hdX …) are.

For instance you may know MAC addresses are unique identifier for network cards, well MAC addresses are UUID (version 1).

So if you can get the UUID for your two file systems your problem is solved.

On linux you can get a file system UUIDs with command “blkid” (“vol_id” command can sometimes be used for old Linux versions).

blkid can be found into util-linux package.

# blkid /dev/sdc1

/dev/sdc1: UUID="01066206-c67c-47d1-83a9-d61791fff943" SEC_TYPE="ext2" TYPE="ext3"

# blkid /dev/sdd1

/dev/sdd1 UUID="cea28516-ca98-4ac4-954f-6710b6ac36c7" SEC_TYPE="ext2" TYPE="ext3"

 

Then add the following lines to /etc/fstab :

UUID=4a496f92-6840-4736-a0d5-5b9916113835       /VLS1   ext3 _netdev    0 0
UUID=cab5e3ec-4797-4227-98e8-e9bca3c3f766       /VLS2   ext3 _netdev    0 0

“_netdev” substitute to the usual “default” because network needs to be up before iSCSI filesystems mount.

Now your iSCSI file systems will be automatically mounted after every reboot (well, as long as your iSCSI server is up ?).