Speed up WAN file transfer with compression

Unix, Unix - Linux ..., , , , , ,

If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency.

To speed up you transfers you need to compress the data so you will have less to transfer.

You can chose to enable ssh compression but if you also have servers on LAN, you will probably slow down those transfer rates.

So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer.

This is how you can do it with only one command line.

myserver1 has a big file “/tmp/myfile.log” that you want on myservers2.

From myserver2  :

[myserver2]/# ssh myserver1 "gzip -c /tmp/myfile.log" |gunzip > myfile.log

Here is a comparison between a transfer using scp only and a transfer using ssh and gzip.

Transfer with compression is 6 times faster in this case :

# time ssh myserver1 "gzip -c /ORACLE/ora92/network/log/listener.log.myserver1" |gunzip > listener.log.myserver1
real    0m40.040s
user    0m2.159s
sys     0m1.665s
 
# time scp myserver1:/ORACLE/ora92/network/log/listener.log.myserver1 .
real    4m1.95s
user    0m6.15s
sys     0m5.80s

 

AIX (5.3) NFS tuning

AIX, Unix - Linux ..., , , , , , , , , , ,

For an AIX migration I needed to maximize my transfer rate on a NFS share (AIX iSCSI is not working with QNAP Turbo NAS).

An AIX expert (one of the name you can read on IBM Redbooks covers) gave a tuning advice:

Pour un migration AIX j’avais besoin de maximiser les taux de transfert vers un partage NFS (comme le iSCSI AIX ne fonctionne pas avec les Turbo NAS QNAP).

Un export AIX (un de ceux qu’on peut lire sur la couverture des Redbook IBM) m’a donné un conseil de tuning assez simple à mettre en place:

To display the current network and nfs settings :

Pour afficher les valeurs actuelles des settings réseau et nfs :

nfso -a
no -a

Set the new parameters:

Mettre en place les paramètres suivants :

nfso -o nfs_use_reserved_ports=1
no -o rfc1323=1
no -o udp_recvspace=262144
nfso -o nfs_v3_pdts=8

(this tuning will be effective only on the next nfs mount)

(ce tuning  ne sera effectif qu’au prochain montage nfs)

then mount your share  :

ensuite, montez vos partages :

mount -o vers=3 qnap1:/Public /mnt

Be sure to run enable rfc1323 before incresing the tcp and udp receiving buffers (tcp_recvspace and udp_recvspace).

Il faut impérativement activer rfc1323 avant d’augmenter les buffers de reception tcp et udp.