Speed up WAN file transfer with compression

If you have servers on Wide Area Network (WAN), you may experience very long transfer rates due to limited bandwidth and latency.

To speed up you transfers you need to compress the data so you will have less to transfer.

You can chose to enable ssh compression but if you also have servers on LAN, you will probably slow down those transfer rates.

So the solution is to use a compression tools like gzip or bzip or compress before and after the data transfer.

This is how you can do it with only one command line.

myserver1 has a big file “/tmp/myfile.log” that you want on myservers2.

From myserver2  :

[myserver2]/# ssh myserver1 "gzip -c /tmp/myfile.log" |gunzip > myfile.log

Here is a comparison between a transfer using scp only and a transfer using ssh and gzip.

Transfer with compression is 6 times faster in this case :

# time ssh myserver1 "gzip -c /ORACLE/ora92/network/log/listener.log.myserver1" |gunzip > listener.log.myserver1
real    0m40.040s
user    0m2.159s
sys     0m1.665s
 
# time scp myserver1:/ORACLE/ora92/network/log/listener.log.myserver1 .
real    4m1.95s
user    0m6.15s
sys     0m5.80s