Backup Software
Not recently I have looked through the backup solution for my home zoo. After looking through a number of alternatives (afbackup, backup-manager, backup2l, backintime, backupninja, boxbackup, cdbackup, cedar-backup2, chiark-backup, dvbackup, dvdbackup, faubackup, flexbackup, jpilot-backup, rdiff-backup, slbackup, storebackup, vbackup) from wikipedia I have stopped at BackupPC. There are the following reasons, why I have chosen this software:
- Web-interface. I know, that is not important for backup software, which can be a CLI-utility, but from the other side if you have apache server running, PHP-based interface is a plus. You can configure BackupPC from browser, force a backup, view the logs and browse currently available file backupped versions in a tree viewer and retrieve the necessary version just by clicking on it.
- It supports full and incremental backups of different levels.
- It performs per-file data compression before storing it to the pool.
- Daemon periodically checks for host availability and performs backup when host becomes online (useful feature for notebooks and other mobile devices).
The main idea of BackupPC is the following: it goes through the data tree to backup and retrieves only new/modified files since the last backup. The files are fetched to local filesystem and are organized in a pool, so if several clients have the same file, it will be cached only once. For home users this feature is not beneficial, as usually all boxes have different non-relative information to backup. This a behaviour of most backup software (backintime, rdiff-backup, etc).
Another modern approach is one provided by CrashPlan, which offers peer-to-peer backup with your friends (who should also have this software installed). The drawbacks of this approach might be:
- The intersection of the time when you and your peer are online is small and you risk to miss daily backup for example. So you should have several peers to minimise the risk, but paying back with increased outbound traffic (multiplied by number of peers).
- If your peer is resided physically long distance away from you, it might be not easy to get a full backup (ask your friend to burn and send you a DVD?)
- Also you “pay” for the traffic, both in- and outbound.
Further reading:
- Remastersys – makes a full system backup including personal data to a live CD or DVD that you can install anywhere.
-
- Syncbox turns an existing PC immediately into a dedicated server and one can start file synchronization and sharing services right away.
- https://www.greyhole.net/ – Samba-based replication solution
- Aulix Backup2DVD utility does very very super reliable backups to optical disks like Bluray and DVD
Universal utility to access cloud
Provided that Mail.Ru has abandoned their Linux client and Dropbox has limited the amount of clients to 3 and restricted the local filesystem to be ext4 there is a need to search for universal alternative:
| Project | Supports MailRu? | Supports DropBox? | Notes |
|---|---|---|---|
| rclone |  here, but does not work with 2FA enabled | here | :MINUS: No two-way synchronization, see https://forum.rclone.org/t/upback-two-way-synchronization-utility-based-on-rclone/4692/5 |
| CloudCross |  affected by 2FA? | | :MINUS: Cannot synchronize part of the tree |
Utility to optimally distribute files into multiple fixed-sized volumes
Implements First Fit Decreasing (FFD):
Implements First Fit:
dirsplitfrom cdrkit – makes given number of random shuffles (500 by default) and chooses the best one (source).
Takes into account that data in ISO image allocates more space because of additional structures.
Closed source (hence algorithm is unknown):
-
Has problems with Cyrillic when creating ISO. Cannot create UDF ISO. So the only option to overcome these is to create .irpfilelist for InfraRecoder.- When using two other modes different from “In order” (“Re-order …”), the result is every time different (looks like Random First Fit?).
My comparison:
DVD Span: 16176MB left on last volumedirsplit(-a 5000iterations): 18084MB left on last volume
See also:
dirsplit for 50GB3) BD DL disks:- Create filelists:
dirsplit -s47730M -a50000 -e1 -p /mnt/iso/video_ /mnt/video
Note that split size is found out in empirical way.
- Verify resulting ISO size. The maximum ISO size should be no more than the size of the target media (otherwise make a correction / rerun previous step):
for file in /mnt/iso/video_*.list do echo -n "[$file]: " extents=`genisoimage -no-rr -allow-limited-size -graft-points -q -print-size -path-list $file` echo "$(($extents * 2048)) B = $(($extents / 512)) MB = $(($extents / 512 / 1024)) GB = $extents extents" done | sort -r -n -k 2,2
which prints something like this:
[/mnt/iso/video_2.list]: 50048901120 B = 47730 MB = 46 GB = 24437940 extents [/mnt/iso/video_3.list]: 50048890880 B = 47730 MB = 46 GB = 24437935 extents [/mnt/iso/video_1.list]: 50048743424 B = 47730 MB = 46 GB = 24437863 extents [/mnt/iso/video_4.list]: 50048741376 B = 47730 MB = 46 GB = 24437862 extents [/mnt/iso/video_5.list]: 47073021952 B = 44892 MB = 43 GB = 22984874 extents
- Generate ISO:
for file in /mnt/iso/video_*.list do echo "[$file]" genisoimage -no-rr -allow-limited-size -graft-points -path-list $file -V "${file%.*}_`date +%F`" -o ${file%.*}.iso done
- Burn ISO starting from most biggest (one that comes first in step 2) to be more secure that you will safely burn others.
Recovery
- Тест бесплатных программ для восстановления данных (и далее по ссылкам в разделе Похожие публикации)
BackupPC questions answered
- How to maximize BackupPC performance (nice hard drive optimisation tips)
- How to workaround rsync stalls when backing up a Windows 2003 server (actually, comes up to installing
rsyncd) -
for host in host1 host2; do su backuppc -c "/usr/share/backuppc/bin/BackupPC_tarCreate -h $host -t -n -1 -s \* ." | 7z a -mx=9 -ms=on -mqs=on -si /mnt/mybook/backup.$host.`date +"%F"`.tar.7z; done
- The information about how to hide the user ID, which you use for backup to access a shared resources, see windows page.
Additional methods for resolving host IP except nmblookup or ping
wins support = yes and domain master = yes configuration options has not given the desired result. I have also checked registry setting for LanManager (see 1 and 2 and 3 and 4), thinking that it is running in “hidden” mode. Finally I've traced the network traffic and found out, that broadcast UDP packets do not pass through WiFi router in direction to Windows box, what I found strange. So in this configuration nmblookup does not reliably resolve Windows host IP address and ping does not work either, because of domain search problem.
- Try to
ping hostfirst, thenping host.and get the results from one that succeeded - Implement one more check,
nslookup host.
Also look for latest updates on .
How to test the network share is accessible by given user?
smbclient -U user%pass //host/share$ -c dir
The problem when trying to connect to the share: “Connection to [host] failed (Error NT_STATUS_UNSUCCESSFUL)”
When restoring files using web interface the downloaded ZIP archive is always broken
libarchive-zip-perl v1.30.
The solution is either to use compression level 0, or to . Also reported to issue #54827.
How to uncompress zlib data?
zlib but they are not true .gz or .zip files.
printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - RestoreLOG.z | gunzip
For example:
printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | cat - /var/lib/backuppc/pc/centurion/XferLOG.1.z | gunzip | egrep -v '^ (pool|create)' | less
