File Deterioration
Posted by: endlessnessism on 07 November 2013
Somebody once told me, a propos digital photo files, that every time you opened them on screen they deteriorated ever so slightly. I don't remember who told me, or when, and it may be complete rubbish. If, however, there's truth in it, are our favourite music files also in danger of disappearing through over-enjoyment?
Somebody once told me, a propos digital photo files, that every time you opened them on screen they deteriorated ever so slightly. I don't remember who told me, or when, and it may be complete rubbish. If, however, there's truth in it, are our favourite music files also in danger of disappearing through over-enjoyment?
Not quite true. Lossy compressed files like jpegs will deteriorate each time a new generation is saved. That's why, even if the orginal is compressed you should convert a version to TIFF (or other lossless format) before multi generational editing. Convert back to jpeg once you're done, if necessary. Newer photo editing sfotware like Lightroom doesn't change the underlying file at all, but simply saves the editing instructions as meta-data, recreating the edited image each time the file is opened. Images edited with this style of software doesn't suffer from generational degradation due to compression on file saving.
Simply opening to view a photo and closing again does not affect the underlying file. Similarly, opening an music file to listen to it doesn't affect it. In the case of lossy files, the damage has already been done with the initial compression.
begs the question whether over the long term a gradual albeit minimal physical deterioration of NAS disk drives reduces sound quality if bits can be lost this way?
begs the question whether over the long term a gradual albeit minimal physical deterioration of NAS disk drives reduces sound quality if bits can be lost this way?
Professional operating systems for storage networks (like ZFS) prevent this by storing redundant data and have checksums on the actual data that are being checked on reads. Normal OS don't have checksums on actual data, but only on TOC.
you could build one yourself because the OS is free.
cheers
aleg
...... if bits can be lost this way?
What way?
Will a playing file deteriorate if there in nobody to listen to it????
I think it may be a law of physics that a music file can have no greater number of bits than the number of the listener's brain cells. Which must explain why the young enjoy MP3 but doesn't quite explain why the old (whose hearing is shot) still believe that hi def is so much better.
I've sealed my NAS drive with ducktape to make sure no bits fall out.
This may the secret with the Unitiserve, the case design stops the bits from escaping.
Richard
There are several levels of protection... firstly disc drives at their physical layer have redundancy and recovery using checksums to identify errors. Mirroring can detect bad data and go for correct data, and finally the file can be marked bad - and in which case the file is lost and is not recoverable. Discs drives are designed should data become corrupted through being unable to read/write then that part of the disc is made non usable.
However data loss or corruption if a single word in a stream might be a single extremely brief click or possibly gap of silence. Listening to CD masters you do hear a few clicks - and a good system will show those up in the down steam conversion.
If you were getting random 'bit' errors then you would getting ticks and clicks all the time and it would be irritating and sound faulty.
A loss of digital data through data corruption usually results in a significant impact to the encoded signal - its not subtle and at best is a gap of silence.
Simon
I don't know about anyone else but I think that it a good idea to rip from CD at least three times so that the data is nice and thick.
I've sealed my NAS drive with ducktape to make sure no bits fall out.
begs the question whether over the long term a gradual albeit minimal physical deterioration of NAS disk drives reduces sound quality if bits can be lost this way?
No. We will not go down this path. Not everything matters... only some things matter. Please...
Please... for my sanity... can we all just stop enumerating all the unlikely possibilities. I honestly think I might go crazy if I start to suspect that a digital file can deteriorate. Please.... lets just focus on ethernet cables.... yeah thats a great topic.
My god this is the most terrifying thread I've ever read....
Yes it is quite disturbing.. Literally.
Actually, I'm serious -
Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. - from Wikipedia
I think our hearing will deteriorate first.
Tog
A CD will not get scratched if it stays in its case. Won't the hard drive still develop bad sectors whether being used or not?
Which is why all data should be backed up, the data is not just dependant on the integrity of the surface of the drive but on the firmware which controls the data location, swapping of faulty sectors etc etc. The firmware is operated from chips and things on pcbs in the drive housing, these can be corrupted, fail etc etc.
In the old days of huge disk drives that were so large that they would not be able to hold a track of music, the read write head would read the data and then rewrite the data back to the magnetic material - I do not know if the same process happens to day.
You missed the thread on CDs left in their cases developing holes due to the ink used?
Actually, I'm serious -
Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. - from Wikipedia
One not to worry about then.
- Choose a file format that is an open published standard. This way, you are not at the tender mercy of a proprietary or closed format if its producer moves the goalposts or goes out of business. I believe FLAC is an open standard, and I have adopted this for music.
- Physical disks do fail so you need multiple copies on different physical disks (they call it "redundancy" I believe). From what I read, it seems that you should have at least 3 copies of all data you care about, and one of these should be offsite. The offsite part is important because if anything happens to the place (e.g. building) where the main data is stored, you still have a copy. I have a NAS at home, and my strategy is to have 3 disks:
- Disk 1 is the master disk (e.g. the one I use day to day - for docs, pics and to play music from)
- Disks 2 and 3 are rotating offsite backups (they are encrypted also). At all times, one is in a drawer at work and the other is at home in the NAS. Evey week, I make sure the disk at home is up to date by synchronising with the master disk, and then I take it to work and swap it over.
- Long term data archiving generally means migrating the data onto new physical media from time to time, as the media is likely to fail eventually, e.g. hard disk crashes, etc. I have found that this kind of comes naturally as my storage requirements increase and I need to buy new disks. I have found that my requirements kind of keep pace with the price of hard disks - by the time I have filled up e.g. a 1TB disk, a 2TB disk is the same price as what I paid for the 1TB disk. There is just the simple but time consuming matter of copying all the data on the old disk onto the new disk, and the result is the new disk is about half full. I then don't use the old disk. This could be seen as a bit wasteful, but this happens about every 2-3 years, so its not that often, and I the benefit is that I am using newish disks all the time. This also gives the opportunity to change to new disk formats or file system types as they become available.
Actually, I'm serious -
Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month. - from Wikipedia
One not to worry about then.
You could. The chances are higher you'll win a lottery without buying a ticket.
Don't consider a NAS as a backup. My NAS (with redundant drives) fell off the shelf, and both drives broke.
Just some context.
Jude
Ok everybody you can put your minds at rest. I am writing some software that'll ensure you do not listen to files that have deteriorated. 
More information as it gets closer to delivery.