File Backup
Posted by: Mr Underhill on 13 May 2012
Since I bought the Drobo, having decided to rip my DVDs, 352 down so far, I have been deciding HOW to backup all my audio & video files.
The NAS gives me resilience - although I have read some disturbing reports on the Drobo nothing wrong so far, but I would HATE to EVER have to do this again.
In days gone by this was easy for me, two external HDDs hitched to an old Linux server with Rsync copying everything on Disk 1 to Disk 2 and I was sorted, but how do I work this now?
I cannot simply take a mirror copy of my NAS, not without buying a second NAS with 6TB of resilient storage, and growing!
I am sure I cannot be the only person in this position, and you may have come up with a better solution. Certainly I fervently wish the Drobo FS had one or more USB ports, it would make the whole thing a lot more straight forward, but it doesn't.
So ......
I decided to make the best use of the equipment I have to hand:
Two (2) 2TB HDDs,
One (1) 1TB HDD,
One (1) 500GB HDD,
One (1) old Dell laptop, running XP sp3.
The Dell laptop I set up a couple of years ago to rip my DVD-As, which I still occasionally buy when I see them at the right price. This now has Serviio DLNA Server installed - for streaming content from the internet & the Drobo.
The Drobo is built on Linux, and I have installed Rsync & NFS.
Traditionally I have hosted my files on two external HDDs, and kept another copy at a friend's as my off-site backup (OSB).
Step 1: Secure What I've Got.
I copied all my music files, WAV & flac, and photos onto the 1TB HDD and gave this to my friend = OSB.
I recovered my 500GB HDD, my old OSB HDD.
Step 2: Set up HDDs
Reformatted 500GB HDD & one 2TB HDD.
Both HDDs attached to Dell Laptop, kept in loft and running VNC.
Step 3: Incremental Backup ....er..
OK, in the good old days all my files would have EASILY fitted onto a 2TB disk, no longer. The video files are already 1.5TB, and growing daily - actually feels more like grinding daily.
My 2TB HDD will no longer cope on its own.
I used to be able to simply maintain a copy of what I had in two places, but now I need to split it.
Having secured my older files in my OSB I do not want to keep copying these.
What I want to do is simply copy my NEWER files to two different locations:
1. My video files to the 2TB disk; and
2. My audio files to the 500GB disk.
As the laptop is running XP my original thought was xcopy. This was scuppered as the command cannot work with shares onto the Linux based Drobo, can't cope with the file datetime stamps ...bugger.
My next though was to load up cygwin, this is a framework for adding unix commands onto a windows box. My hope was to then use Rsync to copy the newer files from the Drobo to the USB HDDs, there is a handy switch to allow this:
rsync --files-from=<(find /../src -atime +6 -print) src/ dest/
I read this, thought GREAT, and failed to get it working as described. Having played for a while I decided to dodge the issue by using a chain of commands, nice having unix and a windows box! In this way I could simply troubleshoot any issues on a command by command basis.
What I came up with was this:
find /cygdrive/w -type f -mtime -1 -print | while read file; do mkdir -p /cygdrive/f/BU/"`dirname "$file"`" && cp -an "$file" /cygdrive/f/BU"$file"; done
For those of you who are not familiar with BASH I'll break this down briefly. In essence this finds a list of files based on age (-mtime), and feeds this into a loop (using the pipe '|').
The loop first looks to see if the target directory exists (dirname), if it doesn't it creates it. It then copies the file into that directory - UNLESS in already exists, in which case it simply moves to the next file in the list (cp -an).
In cygwin all the local drives are mapped via the cygdrive directory:
/cygdrive/w = my video drive (W:\)
/cygdrive/f/BU/ = backup directory on my 2TB USB HDD (F:\).
find
-mtime = file modification time & -1 means within the last day.
This list of files is printed and fed one at a time into a loop as the variable 'file'.
mkdir
If the directory doesn't exist it is made, the directory path is extracted from $file - the earlier created 'file' variable.
The directory path is appended onto /cygdrive/f/BU/.
The path has to be surrounded by speech marks to cope with spaces. As Naim picked windows as its OS for its servers spaces abound, hence: "`dirname "$file"`".
This then leads into the copy command, which will now succeed as the target directory exists.
I have set up three such commands: 1. For my NS01 share; 2. One for my NS01 store, and; 3. One for my video files, all hosted on my NAS.
These are Scheduled Tasks within Win XP running in the early hours.
Once the HDDs are full they will go to my offsite BU location, and new ones purchased.
Hope this may help, or give you ideas.
M