rsync/incementral

[rogerhoward at rogerroger.org]

On Aug 1, 2007, at 9:45 PM, Ashley Aitken wrote:

>
> On 02/08/2007, at 11:58 AM, rogerhoward at rogerroger.org wrote:
>
>>
>> On Aug 1, 2007, at 6:55 PM, steve harley wrote:
>>
>>> they whom i call Roger Howard wrote:
>>>> Am going to begin testing on a much larger filesystem soon.
>>>
>>> it would be most excellent if you'd write us an update when the  
>>> test results are in
>
> I used rsync (via RsyncX) for backing up 40GB with incremental  
> changes of a few GBs. It worked fine.  I had 50 rotating  
> incremental backups on a networked machine.

I use it for nightlies of my ~800GB of photos and panoramas and other  
home projects, works like a charm at that scale.
> I note that setting up the hard links for a very large number of  
> file takes time (the backups weren't the fastest) and space  
> (particularly because they are simulated on HFS+).

Still much faster in every case I've seen than copying the actual  
data, and of course much lighter on your storage budget (I'm not sure  
how much space a hardlink takes on HFS+, but it's certainly much  
smaller than my average file size)...
>
> You may be better off with just standard incremental backups.  I've  
> since moved to using just Backup for full and incremental backups  
> to an external Firewire drive (and that has its problems too).

We use BakBone for the tape-based backups, but if rsync+hardlinks  
works at this scale - 28TB or so, ~20TB of large-ish files and the  
rest tons and tons of smaller files (<1MB) - it'll be very convenient  
as a mechanism for continuity in our production departments if the  
SAN goes down, while we rebuild and restore from tape. If rsync on  
OSX doesn't work well, we may still run it, just on a platform/FS  
which it supports better.

Backup definitely won't cut it :)