rsync/incementral

[Izidor Jerebic]

On 2.8.2007, at 6:54, rogerhoward at rogerroger.org wrote:

>
> We use BakBone for the tape-based backups, but if rsync+hardlinks  
> works at this scale - 28TB or so, ~20TB of large-ish files and the  
> rest tons and tons of smaller files (<1MB) - it'll be very  
> convenient as a mechanism for continuity in our production  
> departments if the SAN goes down, while we rebuild and restore from  
> tape. If rsync on OSX doesn't work well, we may still run it, just  
> on a platform/FS which it supports better.
>
> Backup definitely won't cut it :)

When using rsync version below 3.0.0 (which is not yet released,  
latest is 2.6.9), there may be problems with a large file count (not  
size - just the number of files), because rsync builds complete file  
list in memory before doing any transfer. This takes about 100 bytes  
per filename, so if you have enourmous number of files, it may be a  
problem in time and space (memory). But you can probably always  
separate one large backup into several smaller backups.

New version of rsync (3.0), which is in the testing but not yet  
released, has incremental file list building and does not have this  
problem.

There is an active mailing list dedicated to rsync (google rsync  
mailing list) and people there (including rsync's maintainer) are  
friendly and happily answer questions, so if you intend to seriously  
abuse rsync I suggest joining and asking there if there are problems.  
Maybe just describing your setup and asking for potential problems  
will yield useful information.


izidor