Funny story on drive failures
I have had the external HD from Panoche Hills Ecological Reserve with records of background animal cam monitoring on my desk for about a month. About to back up, had call with Ally, and she told me her nightmare experiences with external drive failures this season. Righteously, I held up this particular ‘x’ brand hard drive and said ‘ohya these are way better, never fail’.
After call, I am thinking, well time to use that unlimited google drive folder and back up these animal cam photos. Plug in drive, opens up, spins for less than a minute, then promptly fails. We do not have backup so this was only copy….
I am like are you kidding, here I am about to finally back up to cloud, and it crashes!!
Spoke with NCEAS informatics team. Apparently, there are just two types of drives – those about to fail, those that have failed. Also, the failure rate for mechanical drives can be up to 45% when there many, many small files versus lower volume, larger files. Of course, we have exactly the worst case scenario (a million small file-sized animal pics from the Carrizo this season). Mechanical drives, in trying to read many little files, jump around a lot.
So, again, we need to shoot for this bare minimum rule.
Three backups of all video/animal cam raw imagery.
TWO external HDs in different places.
ONE cloud back up.
Workflow for archiving
1. Copy from external HD to HD of a local, dedicated lab machine (doing directly from external introduces too much lag).
2. Use ethernet not wifi to move to cloud drive.
3. Put in appropriate folder on drive with subfolders.
4. Google drive does not support folder dropping with Safari so use Chrome or Opera.
5. PUT a raw.imagery.data.csv or googlesheet file in same master folder as your raw imagery explaining folder and file names and replication so that ANYONE could go to that master folder and understand the raw imagery. Meta-data make the world go round and science reproducible, verifiable, and open.
Final note, publish imagery data to data journal before you write paper. Taylor did this via GigaScience, but we could also try for Zooniverse or Nature Scientific Data.
After all, these data are precious.