A few things, mostly technical notes...

Monday, January 24, 2011

rsync delta-transfer algorithm

rsync delta-transfer algorithm is pretty cool, reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

export alias scpr='rsync -av --partial --progress --rsh="ssh -c arcfour -o compression=no -o StrictHostKeyChecking=no"'

myhost:/tmp# ls -ltrh somefile
-rw-r----- 1 root root 2.3G Jan 24 11:53 somefile

myhost:/tmp# time scpr somefile somehost:/tmp/
2452194584 100% 67.76MB/s 0:00:34 (xfer#1, to-check=0/1)

sent 2452494012 bytes received 31 bytes 65399841.15 bytes/sec
total size is 2452194584 speedup is 1.00

real 0m37.408s
user 0m22.341s
sys 0m5.624s

append some data to somefile:

myhost:/tmp# dmesg >> somefile

myhost:/tmp# time scpr somefile somehost:/tmp/

2452249209 100% 137.71MB/s 0:00:16 (xfer#1, to-check=0/1)

sent 266589 bytes received 396255 bytes 20395.20 bytes/sec
total size is 2452249209 speedup is 3699.59

real 0m32.525s
user 0m16.521s
sys 0m0.484s

rsync just sends out the delta(!), 2452494012 bytes the very first time vs. 266589 to accomodate delta.

No comments:


Creative Commons License
This work is licensed under a Creative Commons License.