File Archiving Strategy – the Bucket System

0Comments

A Guest Post by Nick Rains.

Using metadata correctly means that images can be fully catalogued and images can be retrieved at any time without necessarily knowing which folder the file is in. King Penguins, Macquarie Island – Canon 5D MkII, 300f2.8L 1/500 second @ f5.6.

Filing

Where do you actually put your image files?

Do you file images in folders with meaningful names like “Sydney 01-01-2010” and “Perth 11-03-2010”, or “Flowers” or some such. What if it’s a picture of a flower in Perth; which folder would you put it in? Or would you put it in both? This is a physical filing system, not unlike literal files in a literal filing cabinet. It can serve as a filing system but it neglects the single most useful aspect of digital imagery – metadata.

Correct use of metadata means that computers can do what they are best at, remembering large amounts of data and making connections between lists and records. An image with metadata such as City = ‘Perth’ and Caption = ‘flower’ can easily be referenced by a database which merely matches an image file’s metadata with the words Flower and Perth and displays its location on you hard drive. If you search for ‘Flower’ and ‘Perth’ the database will list all files which are tagged with those words – and here’s the trick, the file does not have to be in any particular place on your hard drive as long as the database has previously recorded its position. In other words it has already catalogued all the file locations. You could have a dozen images of flowers in Perth in a dozen different locations and the database can effortlessly list those files when you search for those terms. This is what computers do best, and they are very good at it.

Using good catalogue software means you simply do not have to arrange your files in any sort of logical folder structure as long as the software has catalogued all the locations of all the images. If all your images live on one hard drive, and that hard drive has been fully catalogued, then the folder structure of that hard drive can be anything you want.

OK, so we have established that using something like Idimager, Lightroom or Expressions Media 2 is a good idea. That’s one part of the problem. The other part is how do we store those files off-site as a back-up on DVDs (given that we can only fit 4.5GB of images onto one DVD) in such as manner that we can easily retrieve a file if it’s lost or corrupted somehow. How do we know where it is? We could catalogue each DVDs as as well I suppose but that would be very time consuming and fortunately it’s not necessary.

The trick is to mirror the contents of your DVDs on your hard drive by using folders with the exact same names as the DVDs, containing the exact same images. If we catalogue a hard drive laid out like this then the catalogue will, at the same time, be a catalogue of the DVDs.

This panel from Expressions Media 2 shows the folders on the networked PC. D Drive \ DVDs 151-200 \ DVD161. The green dot means that is what is currently being displayed in a browser window ( see screenshot below).

The Bucket System

I follow the 3-2-1 approach to archiving and back up. That’s at least three copies of each image, on two different media types (HD and DVD) and at least one copy stored fully off-site.

I also follow the ‘Bucket’ approach to archiving popularised by Peter Krogh in his book The DAM Book. I had been using a similar system for a few years when I came across Peter’s book. It was great to have my own methods re-affirmed and developed, so I bought the book, made some changes to my workflow and have been following this path ever since.

The Bucket system is based around optical media of finite size, like DVDs. Please feel free to substitute ‘BlueRay’ for ‘DVD’ in this article as technology has moved on. Regardless of the actual media that you use, the point remains the same – you put your files into folders called buckets and when a bucket is ‘full’ you burn it to a DVD, start a new bucket with a new folder name and start filling that up, and so on. A bucket is considered ‘full’ when it approaches the size of the optical media to which it will be burned.

So ‘Buckets’ are simply folders that are created to be filled with images until they reach, for single sided DVDs, the 4.5GB mark at which point they are burned to DVD and filed off-site. I name these folders on my hard drive DVD001, DVD002, DVD003 and so on. When I burn the DVD its title in the DVD burning software will be the exact same, DVD001 etc and I will write DVD001 on the case (not on the DVD).

Now, here is the crucial point. If you keep your images in these bucket folders on your main hard-drive, or wherever you habitually store your images, you can import them into a cataloguer still within these folders leaving the folders as they are with the same folder name, DVD001 etc. The cataloguer will reference these folders by their folder name and so you have an exact mirror of what is on your hard drive burned to a set of DVDs. For each physical DVD there will be a corresponding folder on your hard-drive with the exact same name containing the exact same files. This makes it amazingly easy to track down a file if for some reason you cannot access it on the hard drive. The cataloguer will tell you which folder it should be in, and all you need to do is find the DVD of the same name.

The browser window in Expressions Media 2 shows the thumbnails of the images in the folder and identifies the individual file at the top, in the status bar. In this case this file is clearly in a folder named DVD161. It's of course also on a DVD called DVD161 which is stored off-site but which could be retrieved and the whole catalogue rebuilt in the event of a disaster.

So, I edit my images and add metadata , make adjustments in Lightroom (and then convert them all to DNG format) and then take the whole collection and split it into new folders each containing about 4.5GB of images. These folders are sequentially named, i.e. DVD001, DVD001 etc and then the whole set of folders is imported into the cataloguer which in my case is Expressions Media 2. Each folder is also copied onto a second computer which acts as a local backup and burned to a DVD of the same name.

The net result is that I have three sets of identical folders containing identical images. Two sets on two different hard-drives (on two different PCs in my case) and the third set on DVDs stored off-site. The cataloguer has imported this exact folder structure as well, so any file references will match the physical DVD names as well as the identically named folders on the two hard-drives. It even works over a network.

There is one slight fly in the ointment here – adding files to folders so that they total 4.5GB is tedious because you can normally only do it manually. You have to select groups of files, copy them into the ‘bucket’ and keep checking the total folder size. It’s a bottleneck. I despaired of finding a way to automate this after searching the net and only finding one application for the Mac that could do this (Big Mean Folder Machine) and none whatsoever for the PC. So, to cut a long story short, I made my own application for PC, called Bucketeer and by the time you read this it should be available from my website (look under Products / Software).

Bucketeer simply takes a large folder of images and copies the contents to new folders of a specified size. Each folder is named sequentially so I end up with one big folder and the same images in a set of smaller folders named in a sequence like DVD001, DVD002 – exactly the method outlined above. All I then have to do is burn each folder to a DVD (manually for now) and it’s all done – no more trial-and-error multiple selections to fill a 4.5GB DVD.

The bucket system is a boon for small collections of a few tens of thousands of images. Bigger collections might benefit from an enterprise level system with servers and dedicated RAID drive arrays but, for most of us, DVDs and BlueRay discs will do just fine as long as you have a good cataloguing system and a methodical apprioach to filing, archiving and backup.

Nick Rains has been a professional photographer for 28 years and his work has been published in books, calendars and magazines all over the world. He currently specialises in feature work around Australia and is a regular contributor to Australian Geographic magazine. Nick is also the Editor of Better Digital Camera magazine and regularly conducts advanced photographic workshops around the country.

Read more from our Post Production category

Guest Contributor This post was written by a guest contributor to dPS.
Please see their details in the post above.

Become a Contributor: Check out Write for DPS page for details about how YOU can share your photography tips with the DPS community.

  • 321 is the way forward. All my images are accesible to me on my RAID0 for speed, with CDs, then DVDs, now DL DVDs of images as well as a hard drive copy off site (my office).

    321 doesn’t work for me in the office though, I tried it for the adidas Football campaign in 2008, I had 800GB of images (after I deleted the JPGs and converted to DNG without embedding original). 26 BluRay disks of images, that even though I had in Lightroom for sorting, I use Expression to catalogue the disk to verify it. Weeks of work archiving!

    I’m trying to use LTO tape to archive images, but I chose to let Lightroom write XMP data to the file, and Retrospect/Tape sees the file has changed and backs it up again. Not useful.

    But, 321 nd buckets are the way to go.

  • Sorry to disagree with you but the bucket system has too many holes if you are trying to build a complete backup/restore process. (DVDs get scratched and can become worthless)

    As someone who works in the computer industry I learned a valuable lesson in 2002. Your backup is only good when you need to recover data. My backup was saying 100% successful and my bi-weekly test restores were 100% successful. The problem was not all of the data was on the tape when I went to restore my files. We used the top of the line software and hardware for backing up. 25% of our data ended up missing.

    On a personal note. I had a dell with a 250 GB primary drive, a 250 gb secondary drive, and a 250n gb external drive. I had about 25,000 pictures on the primary with a backup copy on the other 2 drives. My external drive crashed (stopped spinning). I was in the process of moving so I did not have the time to get a replacement. The night before we moved in I brought over a few things including my PC. That night the house was broken into and my PC was gone. I had a few pictures on DVD I was able to recover but lost over 50% of the pictures.

    The best solution I have found if you are serious about protecting your files.
    1) Sign up for an photo sharing site that allows unlimited space.
    I have a http://smugmug.com http://peanutsgallery.com account and a http://flickr.com/faf1967 account
    2) I have a 500 gb external HD that I use as a primary backup
    3) I have a 1 tb external HD that I use to backup my primary
    4) I also use Microsoft Windows Live to backup my important pictures. Microsoft gives you 25 GB of free space using windows live.

    Remember to use a good site that will be around for a long time when looking at photo sharing site. Also, read the fine print.

    http://smugmug.com offers the best security and gives you the ability to sell your photos.

  • Tom

    Don’t forget to backup your metadata database – and make sure you know how to restore it.

  • Kevin Z

    Great article and thanks for writing it! I have read Peter’s book and have a couple of questions….

    1. Won’t LR loose track of the location of your files after using Bucketeer to sort them into “buckets”?
    2. Say you have an image in a “bucket” and have burned a DVD, then several months later work on the image in LR roundtripping to photoshop. This will create a psd in that bucket thus causing your hard drive bucket and burned DVD to be different. How do you handle this?

    Thanks,
    Kevin

  • Portable External Hard Disks are now very cheap and save a lot of time if you are shooting several GB worth of images a day. Besides I found the wrong way that DVDs are not error proof!!

    But Be sure to follow the 3-2-1 system with the hard disks as well

  • Kylie Jensen

    This post came at a good time. Im still struggling with how best to archive my family photos. I have Nikon transfer which adds the name and date to photos but they are in folders and more folders. Im trying to get them all loose in one big folder and sort out duplicates, maybe rename them better. Still a bit lost what to do.

  • Kevin

    1. Yes, LR will loose synch if you do any file moving operations outside LR. You need to organise your files before you catalogue them – that’s why I don’t use LR as a DAM but prefer Expressions Media 2 or IdImager.
    2. The offline DVDs are all about an offsite ‘last resort’ archive of original raw images. Anything you do with the online images will not be reflected in the DVDs. If you lose current work you will have to go back to the original and re-work the file. Not to open another discussion but I use DNG as my preferred format since the xmp sidecar files are effectively included in one file so it’s not possible to loose them.

    Pete, no system is ‘perfect’, except maybe very expensive server based ones. You suggestion of storing your most valuable images in an online storage server makes sense. In the meantime my suggested process has served me well so far. As they say, YMMV 🙂

  • Tariag

    The problem of using metadata vs. physical filing is that metadata is VERY dependent to a precise software. What will happen to your metadata in 10, 20, 30 years?
    I tried this system years ago. The software I used isn’t published anymore and old version doesn’t work with newer windows. So I had to start everything again…
    Now I stick to physical filing, and with a well organised system I (almost) never lose time finding my pictures.

    I have 5 copies of my files :
    -1 on a very fast NAS for direct work in my office.
    -1 on another, slow NAS, in another room (auto sync every night)
    -1 on a NAS at home (auto sync through internet every week)
    -2 on external hard drives (manual sync every time an important job done)

  • I agree with your general plan, have at least three copies and at least one of them off site. I see no reason for storing them in different media types.

    Unless you generate a small number of files, DVDs are not the best solution because of their small capacity.

    After some trial and error I use the following

    All program files on my Internal Hard Drive
    3- One Terabyte External HDs (One for Photo, 1 for Video, 1 Misc.)
    2- Two Terabyte External HDs with complete backups( Rotated with one always off site)
    Additionally I keep large resolution files on a web server (About 15,000 files that I want to have should all else fail)

  • Tariag

    By using DNG files there is no problem. The metadata lives within the file as XMP which is basically text. It’s very easy to access this data. The trick here is not to store the metadata only in the database, but to synch metadata edits back to the original files so that it can be accessed by any program that can read XMP metadata. If you lose your catalogue, or want to change applications, you can re-catalogue the entire collection without too much trouble. It took Expressions Media 2 about 5 hours to re-catalogue my entire collection of 70000+ images. No biggie in the grand scheme of things.

  • Rex Maximilian

    I couldn’t begin to backup 1TB of photos on DVDs… I use an external HD as the primary, another as the in-house backup, and a third off-site at a friend’s place.

    Also, when putting dates on files names, YYYY-MM-DD (with or without the hyphens) work best.

  • Tom

    for those looking for a free alternative to the bucketing software, theres Sizeme http://lars.werner.no/?page_id=2 . It may not be as slick, but it does the job.

  • I havent found a peice of software that is x-platform (Win / Linux) that suits. I’ve settled on a pretty simple directory structure system that lives on my media servers 1.5TB RAID mirror.

    / share / pics / {year} / {location}

    in each {location} folder I have three sub-folders

    raw – for RAW files
    mod – for processed jpeg files
    rsz – for resized images for the web

    sometimes theres a couple of other folders, for special purpose stuff. I use “Bulk Rename Utility” to rename my raw files to something location relevant (usually the same name as the location), keeping the index number of each pic for a uniqe name.

  • Jan Prawdzik

    Thanks for the article Nick. Tips about backing up your photos are always welcome, especially for people that don’t believe in Murphy’s Law, such as myself.

  • zOth

    Such a physical filing strategy can work fine as long as you only ever want to search for the metadata that you have based it on – in your case, locations and dates. What if you want to look for a date range, or shots with a certain lens, or non-location specific subjects like beach, flower, animal etc.

    Using a decent cataloguer like Expressions Media allows you to keep your chosen folder structure intact (it doesn’t care where files are) but also lets you search all your files for ANY metadata. That’s the power of proper catalogues vs folder-based directory structures.

  • Rex

    Agreed that DVDs are limited – but if you use them as you progressively shoot new stuff they are fine. A 40GB shoot still takes only about 8 discs but you need to do it each time you shoot new stuff – keeping on top of your cataloguing tasks is critical. Going back and writing 1TB to DVDs would be a chore all right.

    I’m looking at moving to BlueRay soon myself.

  • Jane

    Oh, to auto-mate 🙂 Great article but I am far from ever using this type of system; call me lazy daisy, that’s me.

    I hope they come out with a larger capacity usb key; presently at 64 gigs. that’s what I’ve been using to store my images. I’m finding it way cheaper then DVD, cd, ehd, etc. & less time consuming. I stored 12 months on 24 keys so far. I’m not planned to move the disc images over until they come out with a larger capacity key, which, croossing my toes, should be out by next summer as I hope they got my message to increase them, That’s the way to go for pretty much every thing in the future right now. Image your hard drive in the future being a usb flash drive with a 1 terabyte capacity, oh, Nelly, watch them doggies roll.
    Have a great one!

  • @jane 24 USB keys! I’m just trying to imagine the size of your USB hub! 😉

    BluRays is pretty good, 40GB, but my LTO4 at 800GB/1/6TB compressed archiving a 4TB Drobo is solid.

    That the achive stragtegy, for accessing images I have the loosest RAID0 3 x 2TB. But it works at 300MB/sec which is wonderful for catalouge access, then a slow overnight sync to the Drobo.

  • Margaret

    This is just a forewarning to those who are not aware of the downside of USB thumb/flash drives.

    I work in technology and higher education and I have seen far too many flash drives fail–I would not rely on them as my only means of backup or storage. I have seen a lot of people lose important documents such as their thesis and other important files due to hardware failures. Those flash drives have limitations.

    I have found flash memory cards to be more reliable. They are not as convenient due to their size and the necessity of having a card reader, but far more reliable.

    Whatever method you choose, diligence and consistency is preferred. Even servers fail and if you use their services just hope they have an active backup and recovery system of their own.

    I am going to try Nick’s program and keep 3 copies–computer, DVD, and DVD off-site.

  • Hi Margaret

    Might I suggest offsite DVD and on-site HD x2. Burning two sets of DVD s is tedious for one thing, but using two HDs means you can automate the process by backing up from one HD to another. There are plenty of backup apps out there which will effectively mirror two drives. This is a big topic in itself but it’s a very effective strategy. I use SynchBack myself for HD mirroring and Acronis True Image for system backup.

  • I don’t think the point of the article was whether to use physical storage or metadata…since I believe that any photographer (whether professional or amateur) should use both, or at least should strive to use both.

    Physical storage with a coherent naming convention that takes advantage of the operating system sorting abilities, combined with a simplified and consistent approach to applying metadata should work well for anyone. Both aspects are very important and should not be ignored.

    As far as backing up…hmmm….all the approaches suggested have strong points but they also have weak points since there is no backup strategy that is perfect. I do agree however with Pete about having an account and uploading most pictures to online sharing services like Fotki or SmugMug.

  • “So, I edit my images and add metadata , make adjustments in Lightroom (and then convert them all to DNG format) and then take the whole collection and split it into new folders each containing about 4.5GB of images. ”

    This is the most complicated part for me. Managing and moving files and/or folder in LR2. I end moving them outside LR and then mess up everything grrrrrrrr

  • Catalogue your images in LR after splitting them into buckets. This is a crucial aspect of the process, a cataloguer cannot really do its job if you move the files it has catalogued.

    Or if you really want to do your edits and adjustment first, use two catalogues, a temporary one for edits and a master one for final archived files in buckets.

Some Older Comments

  • Nick Rains August 21, 2010 08:06 am

    Catalogue your images in LR after splitting them into buckets. This is a crucial aspect of the process, a cataloguer cannot really do its job if you move the files it has catalogued.

    Or if you really want to do your edits and adjustment first, use two catalogues, a temporary one for edits and a master one for final archived files in buckets.

  • db Fotografy August 21, 2010 05:27 am

    "So, I edit my images and add metadata , make adjustments in Lightroom (and then convert them all to DNG format) and then take the whole collection and split it into new folders each containing about 4.5GB of images. "

    This is the most complicated part for me. Managing and moving files and/or folder in LR2. I end moving them outside LR and then mess up everything grrrrrrrr

  • Vlad Georgescu August 11, 2010 07:52 am

    I don't think the point of the article was whether to use physical storage or metadata...since I believe that any photographer (whether professional or amateur) should use both, or at least should strive to use both.

    Physical storage with a coherent naming convention that takes advantage of the operating system sorting abilities, combined with a simplified and consistent approach to applying metadata should work well for anyone. Both aspects are very important and should not be ignored.

    As far as backing up...hmmm....all the approaches suggested have strong points but they also have weak points since there is no backup strategy that is perfect. I do agree however with Pete about having an account and uploading most pictures to online sharing services like Fotki or SmugMug.

  • Nick Rains August 3, 2010 12:56 pm

    Hi Margaret

    Might I suggest offsite DVD and on-site HD x2. Burning two sets of DVD s is tedious for one thing, but using two HDs means you can automate the process by backing up from one HD to another. There are plenty of backup apps out there which will effectively mirror two drives. This is a big topic in itself but it's a very effective strategy. I use SynchBack myself for HD mirroring and Acronis True Image for system backup.

  • Margaret August 3, 2010 12:31 am

    This is just a forewarning to those who are not aware of the downside of USB thumb/flash drives.

    I work in technology and higher education and I have seen far too many flash drives fail--I would not rely on them as my only means of backup or storage. I have seen a lot of people lose important documents such as their thesis and other important files due to hardware failures. Those flash drives have limitations.

    I have found flash memory cards to be more reliable. They are not as convenient due to their size and the necessity of having a card reader, but far more reliable.

    Whatever method you choose, diligence and consistency is preferred. Even servers fail and if you use their services just hope they have an active backup and recovery system of their own.

    I am going to try Nick's program and keep 3 copies--computer, DVD, and DVD off-site.

  • Mark July 28, 2010 07:16 am

    @jane 24 USB keys! I'm just trying to imagine the size of your USB hub! ;-)

    BluRays is pretty good, 40GB, but my LTO4 at 800GB/1/6TB compressed archiving a 4TB Drobo is solid.

    That the achive stragtegy, for accessing images I have the loosest RAID0 3 x 2TB. But it works at 300MB/sec which is wonderful for catalouge access, then a slow overnight sync to the Drobo.

  • Jane July 28, 2010 04:26 am

    Oh, to auto-mate :) Great article but I am far from ever using this type of system; call me lazy daisy, that's me.

    I hope they come out with a larger capacity usb key; presently at 64 gigs. that's what I've been using to store my images. I'm finding it way cheaper then DVD, cd, ehd, etc. & less time consuming. I stored 12 months on 24 keys so far. I'm not planned to move the disc images over until they come out with a larger capacity key, which, croossing my toes, should be out by next summer as I hope they got my message to increase them, That's the way to go for pretty much every thing in the future right now. Image your hard drive in the future being a usb flash drive with a 1 terabyte capacity, oh, Nelly, watch them doggies roll.
    Have a great one!

  • Nick Rains July 26, 2010 08:04 am

    Rex

    Agreed that DVDs are limited - but if you use them as you progressively shoot new stuff they are fine. A 40GB shoot still takes only about 8 discs but you need to do it each time you shoot new stuff - keeping on top of your cataloguing tasks is critical. Going back and writing 1TB to DVDs would be a chore all right.

    I'm looking at moving to BlueRay soon myself.

  • Nick Rains July 26, 2010 08:00 am

    zOth

    Such a physical filing strategy can work fine as long as you only ever want to search for the metadata that you have based it on - in your case, locations and dates. What if you want to look for a date range, or shots with a certain lens, or non-location specific subjects like beach, flower, animal etc.

    Using a decent cataloguer like Expressions Media allows you to keep your chosen folder structure intact (it doesn't care where files are) but also lets you search all your files for ANY metadata. That's the power of proper catalogues vs folder-based directory structures.

  • Jan Prawdzik July 23, 2010 02:47 pm

    Thanks for the article Nick. Tips about backing up your photos are always welcome, especially for people that don't believe in Murphy's Law, such as myself.

  • z0th July 23, 2010 02:32 am

    I havent found a peice of software that is x-platform (Win / Linux) that suits. I've settled on a pretty simple directory structure system that lives on my media servers 1.5TB RAID mirror.

    / share / pics / {year} / {location}

    in each {location} folder I have three sub-folders

    raw - for RAW files
    mod - for processed jpeg files
    rsz - for resized images for the web

    sometimes theres a couple of other folders, for special purpose stuff. I use "Bulk Rename Utility" to rename my raw files to something location relevant (usually the same name as the location), keeping the index number of each pic for a uniqe name.

  • Tom July 21, 2010 05:59 pm

    for those looking for a free alternative to the bucketing software, theres Sizeme http://lars.werner.no/?page_id=2 . It may not be as slick, but it does the job.

  • Rex Maximilian July 20, 2010 04:56 pm

    I couldn't begin to backup 1TB of photos on DVDs... I use an external HD as the primary, another as the in-house backup, and a third off-site at a friend's place.

    Also, when putting dates on files names, YYYY-MM-DD (with or without the hyphens) work best.

  • Nick Rains July 19, 2010 05:52 pm

    Tariag

    By using DNG files there is no problem. The metadata lives within the file as XMP which is basically text. It's very easy to access this data. The trick here is not to store the metadata only in the database, but to synch metadata edits back to the original files so that it can be accessed by any program that can read XMP metadata. If you lose your catalogue, or want to change applications, you can re-catalogue the entire collection without too much trouble. It took Expressions Media 2 about 5 hours to re-catalogue my entire collection of 70000+ images. No biggie in the grand scheme of things.

  • Harold July 19, 2010 04:19 am

    I agree with your general plan, have at least three copies and at least one of them off site. I see no reason for storing them in different media types.

    Unless you generate a small number of files, DVDs are not the best solution because of their small capacity.

    After some trial and error I use the following

    All program files on my Internal Hard Drive
    3- One Terabyte External HDs (One for Photo, 1 for Video, 1 Misc.)
    2- Two Terabyte External HDs with complete backups( Rotated with one always off site)
    Additionally I keep large resolution files on a web server (About 15,000 files that I want to have should all else fail)

  • Tariag July 19, 2010 12:38 am

    The problem of using metadata vs. physical filing is that metadata is VERY dependent to a precise software. What will happen to your metadata in 10, 20, 30 years?
    I tried this system years ago. The software I used isn't published anymore and old version doesn't work with newer windows. So I had to start everything again...
    Now I stick to physical filing, and with a well organised system I (almost) never lose time finding my pictures.

    I have 5 copies of my files :
    -1 on a very fast NAS for direct work in my office.
    -1 on another, slow NAS, in another room (auto sync every night)
    -1 on a NAS at home (auto sync through internet every week)
    -2 on external hard drives (manual sync every time an important job done)

  • Nick Rains July 18, 2010 09:47 pm

    Kevin

    1. Yes, LR will loose synch if you do any file moving operations outside LR. You need to organise your files before you catalogue them - that's why I don't use LR as a DAM but prefer Expressions Media 2 or IdImager.
    2. The offline DVDs are all about an offsite 'last resort' archive of original raw images. Anything you do with the online images will not be reflected in the DVDs. If you lose current work you will have to go back to the original and re-work the file. Not to open another discussion but I use DNG as my preferred format since the xmp sidecar files are effectively included in one file so it's not possible to loose them.

    Pete, no system is 'perfect', except maybe very expensive server based ones. You suggestion of storing your most valuable images in an online storage server makes sense. In the meantime my suggested process has served me well so far. As they say, YMMV :-)

  • Kylie Jensen July 18, 2010 03:50 pm

    This post came at a good time. Im still struggling with how best to archive my family photos. I have Nikon transfer which adds the name and date to photos but they are in folders and more folders. Im trying to get them all loose in one big folder and sort out duplicates, maybe rename them better. Still a bit lost what to do.

  • Tarique Sani July 18, 2010 01:17 pm

    Portable External Hard Disks are now very cheap and save a lot of time if you are shooting several GB worth of images a day. Besides I found the wrong way that DVDs are not error proof!!

    But Be sure to follow the 3-2-1 system with the hard disks as well

  • Kevin Z July 18, 2010 10:07 am

    Great article and thanks for writing it! I have read Peter's book and have a couple of questions....

    1. Won't LR loose track of the location of your files after using Bucketeer to sort them into "buckets"?
    2. Say you have an image in a "bucket" and have burned a DVD, then several months later work on the image in LR roundtripping to photoshop. This will create a psd in that bucket thus causing your hard drive bucket and burned DVD to be different. How do you handle this?

    Thanks,
    Kevin

  • Tom July 18, 2010 09:06 am

    Don't forget to backup your metadata database - and make sure you know how to restore it.

  • Pete July 18, 2010 07:04 am

    Sorry to disagree with you but the bucket system has too many holes if you are trying to build a complete backup/restore process. (DVDs get scratched and can become worthless)

    As someone who works in the computer industry I learned a valuable lesson in 2002. Your backup is only good when you need to recover data. My backup was saying 100% successful and my bi-weekly test restores were 100% successful. The problem was not all of the data was on the tape when I went to restore my files. We used the top of the line software and hardware for backing up. 25% of our data ended up missing.

    On a personal note. I had a dell with a 250 GB primary drive, a 250 gb secondary drive, and a 250n gb external drive. I had about 25,000 pictures on the primary with a backup copy on the other 2 drives. My external drive crashed (stopped spinning). I was in the process of moving so I did not have the time to get a replacement. The night before we moved in I brought over a few things including my PC. That night the house was broken into and my PC was gone. I had a few pictures on DVD I was able to recover but lost over 50% of the pictures.

    The best solution I have found if you are serious about protecting your files.
    1) Sign up for an photo sharing site that allows unlimited space.
    I have a http://smugmug.com http://peanutsgallery.com account and a http://flickr.com/faf1967 account
    2) I have a 500 gb external HD that I use as a primary backup
    3) I have a 1 tb external HD that I use to backup my primary
    4) I also use Microsoft Windows Live to backup my important pictures. Microsoft gives you 25 GB of free space using windows live.

    Remember to use a good site that will be around for a long time when looking at photo sharing site. Also, read the fine print.

    http://smugmug.com offers the best security and gives you the ability to sell your photos.

  • Mark July 18, 2010 07:01 am

    321 is the way forward. All my images are accesible to me on my RAID0 for speed, with CDs, then DVDs, now DL DVDs of images as well as a hard drive copy off site (my office).

    321 doesn't work for me in the office though, I tried it for the adidas Football campaign in 2008, I had 800GB of images (after I deleted the JPGs and converted to DNG without embedding original). 26 BluRay disks of images, that even though I had in Lightroom for sorting, I use Expression to catalogue the disk to verify it. Weeks of work archiving!

    I'm trying to use LTO tape to archive images, but I chose to let Lightroom write XMP data to the file, and Retrospect/Tape sees the file has changed and backs it up again. Not useful.

    But, 321 nd buckets are the way to go.

Join Our Email Newsletter

Thanks for subscribing!


DPS offers a free weekly newsletter with: 
1. new photography tutorials and tips
2. latest photography assignments
3. photo competitions and prizes

Enter your email below to subscribe.
Email:
 
 
Get DAILY free tips, news and reviews via our RSS feed