Speed up WAN file transfer with compression

Unix, Unix - Linux ..., , , , , ,

If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency.

To speed up you transfers you need to compress the data so you will have less to transfer.

You can chose to enable ssh compression but if you also have servers on LAN, you will probably slow down those transfer rates.

So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer.

This is how you can do it with only one command line.

myserver1 has a big file “/tmp/myfile.log” that you want on myservers2.

From myserver2  :

[myserver2]/# ssh myserver1 "gzip -c /tmp/myfile.log" |gunzip > myfile.log

Here is a comparison between a transfer using scp only and a transfer using ssh and gzip.

Transfer with compression is 6 times faster in this case :

# time ssh myserver1 "gzip -c /ORACLE/ora92/network/log/listener.log.myserver1" |gunzip > listener.log.myserver1
real    0m40.040s
user    0m2.159s
sys     0m1.665s
 
# time scp myserver1:/ORACLE/ora92/network/log/listener.log.myserver1 .
real    4m1.95s
user    0m6.15s
sys     0m5.80s

 

Tar Archive and compression

Unix, Unix - Linux ..., , , ,

Sometimes when you are working on UNIX you only have the good old tar, compress, gzip and bzip2 commands (no gtar or zip).

If you need (for space issues for instance) to create a compressed archive of a directory, this is how you do it:

Lorsque vous travaillez sur UNIX, il arrive que vous n’ailliez que tar, compress, gzip et bzip2 (pas de gtar ou zip).

Si vous avez besoin (pour des problèmes de place par exemple) de créer une archive compressée d’un répertoire, voici comment faire :

tar cvfp - test_dir | gzip - > test_dir.tar.gz
tar cvfp - test_dir | compress - > test_dir.tar.Z
tar cvfp - test_dir | bzip2 - > test_dir.tar.bz2

With the basic options “compress” is the quickest way to do it but you’ll have a weak poor compression ratio.

Le plus rapide avec les options de base étant « compress » mais c’est aussi  celui qui donne le moins bon ratio de compression.