Speed up WAN file transfer with compression

Unix, Unix - Linux ..., , , , , ,

If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency.

To speed up you transfers you need to compress the data so you will have less to transfer.

You can chose to enable ssh compression but if you also have servers on LAN, you will probably slow down those transfer rates.

So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer.

This is how you can do it with only one command line.

myserver1 has a big file “/tmp/myfile.log” that you want on myservers2.

From myserver2  :

[myserver2]/# ssh myserver1 "gzip -c /tmp/myfile.log" |gunzip > myfile.log

Here is a comparison between a transfer using scp only and a transfer using ssh and gzip.

Transfer with compression is 6 times faster in this case :

# time ssh myserver1 "gzip -c /ORACLE/ora92/network/log/listener.log.myserver1" |gunzip > listener.log.myserver1
real    0m40.040s
user    0m2.159s
sys     0m1.665s
 
# time scp myserver1:/ORACLE/ora92/network/log/listener.log.myserver1 .
real    4m1.95s
user    0m6.15s
sys     0m5.80s

 

Send directories with tar

Linux, Unix - Linux ..., , , ,

Here is a nice set of commands to send a directory from one server to another, with tar, so you can preserve permission (-p option) and use compression (network optimization).

Voici une petite combinaison de commandes sympathique pour envoyer un répertoire d’un serveur à un autre avec tar, pour conserver les priorités (-p) et utiliser la compression (optimisation du flux réseau).

His commands archive, compress, send and unachive, uncompress the directory on the fly.

La commande archive, compresse, envoie et désarchive, décompresse à la volée le répertoire.

# gtar czvfp - wordpress | ssh myhost1.mydomain "cd /tmp; gtar xzvfp -"

The same thing can be done the other way around (from server target):

Cela fonctionne aussi dans l’autre sens (depuis le serveur cible de la copie) :

# ssh myhost2.mydomain "cd /tmp; gtar czvfp - wordpress " |gtar xzvfp -

ssh can ben replaced with remsh if you have to.

Vous pouvez remplacer ssh par remsh au besoin.