Safeguarding my digital files

Backup to good quality external usb 3.0 hard drives.  Keep one at home.  Keep another off-site (at your office, mum's house, etc.).  This is all that's necessary and plenty robust imho. And cost-effective.

50gb per blu-ray would be a non-starter for me; I have almost 2tb of data.

A very good question my last one lasted  3.5 years I had one cube mtr of CDs on it chap at PC world said that was about average, 

That stinks, sounds like you've been unlucky. 

Online backup is getting very good and cheap. Carbonite (and many others) have good solutions with easy tools, then there's Amazon Clacier storage, but its not point and click.

How much disk space are you using?  If you want to have something local, get a NAS bay with RAID for redundant drives. This is definitely a "you get what you pay for" scenario, with a lot of low quality, low price hardware out there. Spending $1000 gets you something very high quality. But, probably cheaper in the long run to have online backup + a cheaper, consumer NAS.

Having said that, I personally use Bart's method, but plan on moving online soon.

DVD, blue ray and usb flash drives are all far less reliable than quality hard drives, and don't forget about the huge amount of time it'll take you to transfer all the data and baby sit the process.

I keep all my music files on a NAS. I have two USB external HD's and plug these in each month running a backup job. One stays at home. The other at work. I keep a third drive with all the files copied across from the NAS using Carbon Copy Cloner (incremental backup) so I have one drive which isn't in a proprietary format, just incase I need to rebuild my library to a different NAS. If all really goes wrong, I still have all my CDs. I don't trust the online solutions - i prefer to be in control of where i keep my files.

I have a separate NAS for backup. It switches on automatically once a week and does an incremental backup of my shares on my main NASs, sends me an email to confirm its done it successfully and then shutsdown for another week. 

Works for me. I use various Netgear NASs - and some are a few years old now and have been rock solid reliable.

Simon

There are a number of options open to you but it depends on how much money you are willing to spend and time to do backup activities.

Personally I have a regime that is a combination of Simon and Bart. At home I use a 4 drive Synology NAS that uses Raid 5 giving me the ability to cope with 2 simultaneous hard drive failures. I also use another external hard drive (like Bart) and keep that away from home. If you use two hard drives at differing locations remember to rotate them to ensure that they are all as up to date as possible (look up grandfather, father, son rotation on the internet to get the idea). 

Also note that, as announced yesterday, if you happen to use the cloud services of one of the large IT providers linked to their productivity suite you can gain access to 1TB cloud backup within the monthly cost. Remember that you will need to factor in broadband usage in this option - some ISP products have data transfer caps.  

I have a Unitiserve, a NAS and a USB hard drive.

Wherever I store my files, I ensure to have 2 copies, just in case.

My Unitiserve is the most straightforward to back up, doing an Auotomatic Scheduled Back Up each week on a Thursday.

Everything else is manual, but I just make sure to copy across at regular intervals, logging on a spreadsheet when I last backed up on all devices.

A bit sad I know, but its just about having more than one copy and being organised.

I don't trust the cloud yet, even though I know this is probably irrational. I just prefer to have control of my own destiny.

Does sound like you have been particularly unlucky though.

Good luck!

2 back ups of the NAS is a good rule of thumb - One onsite and one off.  The price of hard drives is certainly cheap enough.

If you have a PC or Linux ( I know the minority here - their next version is due to support Os X ) a good tool for just transferring only the 'updated/new' files is Beyond Compare as it does a comparison on files and folders http://www.scootersoftware.com/  - it can also work for Pictures / Work files etc.  It makes the incremental backups easy and quick.

BTW Timbo, nice pic of Morants Curve :-)

Carbon Copy Cloner is a good OS X app for incremental backups.

I too back up to the nas in the first instance, and then backup the nas to usb hard drives -- one stays home and one goes to work with me.

"The cloud" is fine, but in mid-2014 it is still a bit expensive, and very slow.  A full backup TO the cloud would take me days.  At this time, a decent external usb 3.0 drive is much more time and cost-effective.

This for me. Semi-continuous back-ups at home, then about every two months, I bring a couple of USB drives home from the office, back-up everything and take them back to the office.

I tried Backblaze ( http://www.backblaze.com )as a trial, but it was far too slow. The initial back-up of just one computer didn't complete in the 15-day trial period.

When considering the role of RAID in data security, don't lose sight of the fact that RAID keeps copies of your current data, whereas backups are copies of your previous data.

RAID (in ltself) doesn't allow you to roll back to previous versions. If a file becomes corrupt or a whole lot of data is accidentally deleted, RAID will simply save a copy of the corrupt file, and dutifully delete the copies of the accidentally deleted data. RAID is security against hardware failure, but does little to safeguard against software failure and human error.

Thanks for the info everyone. It does look that I have been unlucky with my NAS boxes. As for the music I'm sitting at about 2TB almost filled my QNap NAS which is two 2TB drives in a mirror array and then backed up to a separate esata 2TB drive which is attached to the QNap box.

Photographs are another consideration, I'm currently running at about 17TB of files spread over roughly 12 NAS boxes. I've been scanning and cleaning up a load of old family photo's for the last 6-7 years. I think there must be about 65,000 old photo's (big family & prolific photographers). Plus all the digital files I keep taking. By the way I do get rid of the crap ones, so shoot about 200 and lucky if I keep 10-12, but I do shoot RAW which takes up the space.

The cloud looks like a good idea especially as I might be moving to a different country in the medium term.

Tim

Amazon Web Services Glacier lets you store 1 GB/m for ~ .01$.

That would be almost $200/month for Timbo, if I did the math correctly.

Not to mention that he probably could NEVER get it all up to the cloud before he was ready to do an incremental backup.

Well he should compress everything.  Glacier provides multiple redundancies and the price will only go down over time.

You did your maths wrong 

It would be about $10/month for each TeraByte, excluding sales tax, vat, and whatever.

Upload speeds are an issue though at 46 days to upload 1 TB on a 2 Mb/s upload line.

you need a extremely good upload speed to get that done in acceptable time.

You can mail the a hard drive to do the initial load.

Cloud storage is going to be difficult unless you have a connection with a very fast upload speed.  So if you're not going for cloud storage...

Are any of your individual repositories going to (or already) exceed 4TB of unstructured storage?

By this I mean that although the total is 17TB, can this be logically viewed in identifiable chunks that are each <4TB?

IF you can arrange this then the last (off-site) backup can be done to external hard disk.  If not the you'll need ALL your backup systems to handle ALL your data (all 17TB of it) therefore they'll ALL have to be RAID systems.

The point made by winkyincanada is important, RAID redundancy isn't the same as backup.  For 17TB I'd use both.  My recommendation...

Tier1   24TB+ RAID 5   - Working storage,  Always powered, always on-line

(striped storage, can handle disk failure without loss)

N.B. These should be 'Enterprise Level' NAS disks that are designed to run 24x7, with TLER enabled.

Tier2   24TB+ RAID 0   - First level backup,  Powered up only when needed

(redundancy not required as Tier1 still has the data)

These should be desktop disks that are designed to be powered up and powered down on demand, with TLER NOT enabled.

Tier 3   - 2nd level backup (preferably off-site)

If you can logically split your data into 4Tb chunks then you can use multiple external hard disks, otherwise you'll need another 24TB+ RAID 0 (same disk config as tier 2) that can be moved off-site.

As you have so many files another option is to backup to another NAS drive that you own which is sited at a different location, i.e. over the internet.

My Netgear ReadyNas has the capability of using something called ReadyNas Replicate whereby you setup backup jobs between 2 Nas drives and the backup jobs work regardless of where they are located.

Maybe you have a friend with a similar requirement and you could site a second NAS drive at their house and have a reciprocal arrangement.

I'm sure this capability isn't just limited to the Netgear models.

Crashplan allows you do something similar, but is dependent on a PC running at both ends so isn't quite such an elegant solution.

Richard

I do sechedule local backups with my NAS on a daily basis. 

In addition, I also purchase good drive storage for some of my backup needs. 1T for 9.99USD per month.

Exactly what I do Simon, except with Synology NAS

The problem with any Internet based transport (Cloud Storage or VPN to a remote NAS drive) is easy to show:

Sustained data upload rate (ADSL 2) = 500kbps (peak 1Mbps) = 50k bytes / s.

Note: These figures are rather optimistic, and also assume that your ISP doesn’t throttle the stream.

So, for 17TB

17TB / 50k = 340,000seconds ~ 4 days solid.

The full 17TB backups (done every 6 months as a minimum) will tie up your entire available internet bandwidth for four days!  The next alternative is to use a background transfer at say 10% occupancy; then it’ll take 40 days, but the impact on your internet connection will be relatively minor.

Don’t forget that each time an incremental backup is to be done, it has to load the directory information from the far end to check the properties of the files and compare them so that it knows which files have changed and need to be backed up.  For slowly changing data, this download of the directory information can take longer than the time to update the changed files.

Time vs. convenience vs. cost - your call.

For smaller collections this is viable and a good solution, but, for 17Tb you'll need 340 Bluray BD-DL disks.

If you do want to use Bluray, I suggest having a separate sub-folder in the DLNA music share folder for all you new acquisitions (whether downloaded or ripped).  This is then the only folder you need to back up regularly (the other music files will already be backed up on previous disks), after backing up this should then be moved into the main on-line repository.  New music will be served from this just like any other location - it's normally selected using the tags, not the location on the disk.

Don't forget that all backups need to be tested for readability occasionally (recommended every 2 years min).  I'd hate to have to test all 340 disks!  NAS drives or external hard disks may be a more practical solution for backup of large data collections.

This has gotten interesting - its a problem we'll all need to deal with eventually, and I think we'll all be headed above 10TB in the not too distant future.

Makes me wonder why hi fi shops haven't branched out a bit to boutique home IT specialists. Given the number of questions here about networking and streaming, it seems obvious, and also disappointing how many people bring home a new Naim system without adequate dealer support. Maybe some of them are already there - my dealer says he pays the bills doing home theatre installs and fancy audio gear is icing on the cake and it isn't viable as a stand alone business.