rsync delta-transfer algorithm

rsync delta-transfer algorithm is pretty cool, reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

export alias scpr='rsync -av --partial --progress --rsh="ssh -c arcfour -o compression=no -o StrictHostKeyChecking=no"'

myhost:/tmp# ls -ltrh somefile
-rw-r----- 1 root root 2.3G Jan 24 11:53 somefile

myhost:/tmp# time scpr somefile somehost:/tmp/
somefile
  2452194584 100%   67.76MB/s    0:00:34 (xfer#1, to-check=0/1)

sent 2452494012 bytes  received 31 bytes  65399841.15 bytes/sec
total size is 2452194584  speedup is 1.00

real    0m37.408s
user    0m22.341s
sys     0m5.624s

append some data to somefile:

myhost:/tmp# dmesg >> somefile


myhost:/tmp# time scpr somefile somehost:/tmp/

somefile
  2452249209 100%  137.71MB/s    0:00:16 (xfer#1, to-check=0/1)

sent 266589 bytes  received 396255 bytes  20395.20 bytes/sec
total size is 2452249209  speedup is 3699.59

real    0m32.525s
user    0m16.521s
sys     0m0.484s
myhost:/tmp#

rsync just sends out the delta(!), 2452494012 bytes the very first time vs. 266589 to accomodate delta.

scp: transfer speed & compression

With compression:

$ time scp -c arcfour -C somefile somehost:/somewhere 

real    47m30.974s
user    46m18.974s
sys     1m4.040s

Without compression:

$ time scp -c arcfour  somefile somehost:/somewhere 
real    10m41.195s
user    2m41.142s
sys     0m51.611s

details of somefile:

$ file somefile 
somefile: MySQL MISAM compressed data file Version 1

Gist: if a file is already compressed, scp-ing with -C (compression) rightly slows down the transfer.